Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space
Authors: Jiehong Lin
Hongyang Li
Ke Chen
Jiangbo Lu
Kui Jia
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D convolution
SE(3)-equivariant feature learning
pose estimation

Abstract: As a basic component of SE(3)-equivariant deep feature learning, steerable convolution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3)-equivariance. Based on SS-Conv, we propose a general pipeline for precise estimation of object poses, wherein a key design is a Feature-Steering module that  takes the full advantage of SE(3)-equivariance and is able to conduct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Ablation studies also show the superiority of our SS-Conv over alternative convolutions in terms of both accuracy and efficiency. Our code is released publicly at https://github.com/Gorilla-Lab-SCUT/SS-Conv.

Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Eqiivarian Features for Estimation and Tracking of Object Poses in 3D Space
Jiehong ! Hongyang ; Chen 1.3 Jiangbo Lu Kui Jia 13,51 South China University of Technology. "DexForce Co. Ltd. Peng Cheng Laboratory. "SmartMore Technology Co. Ltd. 'Pazhou Laboratory. {lin. j jiehong, eeli Bonngyang/mmail.comm scuat. edu. cn jiangbo lu@gmail... com, {chenk, kuijia))scut.. edw.cuuteeecccm
Abstract
As basic component of SE(3)-cquivarian deep feature learning, steerable con- volution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3) equivarqui Based on SS-Conv, we propose general pipeline for precise estimation of object poses, wherein key design is a Feature-Steerin module that takes the full advantage of SE(3)-equivarianc and is able to con- duct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, level 6D pose tracking. Our proposed pipeline based SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Abla- tion studies also show the superiority of our SS-Com over alternative convolu- tions terms both accuracy and efficiency. Our code is released publicly at https:// tuu..u..... Com//orinn -aarrllllaabbbba -aa--aa--S-SUUUSCComm
category-
Introduction
-Euuiiaaaarrr deep networks 20 25 7 have shown the promise recently in some tasks of 3D semantic analysis, among which 3D Steerable CNN (25) is representative one. 3D Steerable CNNs employ steerable convolutions (termed as ST-Conv) to learn pose-equivariant features in layer-wise manner, thus preserving the pose information of the 3D input. Intuitively speaking, for a layer of ST-Conv, any SE(() transformation (r, t) applied to its 3D input would induce a pose-synchronized transformation to its output features, where
SO(3) stands for rotation and t E 1 for translation. Fig. (a) gives an illustration where given an SE(3) transformation of the input, the locations which feature vectors and the feature vectors themselves are also rotated by p(r) (p(r) is representation of rotation r). This property of SE(3)-equivariance enables the steerability of feature space. For example, without transforming the input, SE(3) transformation can be directly realized by steering in the feature space. To produce steerable features, ST-Conv confines its feature domain to regular grids of 3D volumetric
defined are rigidly transformed with respect to (r, t).
"Equal contribution CCoresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

ST-Com
Co-Comv
(64
(r.4)
ST-Com
fer
SS-Comv
pM
for (x (x-00
[ntre/]] = p1r
[cort/]]on ppr) 60)
(a)
(b)
Figure 1: An illustration of 3E(3)-equivarian based on dense tensors, and (b) our Sparse Steerable Convolution (SS-Conv) based on sparse tensors, where arrows defined the 3D fields denote vector-formed, oriented features. Best view in the electronic version.
achieved by (a) STeerable Convolution (ST-Conv)
data; it can thus be conveniently supported by 3D convolution routines. This compatibility with 3D convolutions eases the implementation of ST-Conv, but at the sacrifice of efficiently processing 3D> data (e.g., point clouds) that are typically irregular and sparse; consequently, ST-Conv is still less widely used in broader areas of 3D semantic analysis.
In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the aforementioned shortcoming faced by ST-Conv. SS-Conv can greatly accelerate steerable con- volutions with sparse tensors, while strictly preserving the SE(3)--quivariance in feature learning; Fig. [ (b) gives the illustration. To implement SS-Conv, we construct convolutional kernels as linear combinations of basis ones based on spherical harmonics, which satisfy the rotation-steerable constraint -equiaaaav convolutions
[25], and implement the convolution as matrix-matrix multiply-add operations on GPUs only at active sites, which are recorded along with their features as sparse tensors.
Although SEu3i-eqqivar feature learning is widely used in 3D object recognition, its potentials for other tasks of 3D semantic analysis have not been well explored yet. In this work, we make the attempt to apply our proposed SS-Conv to object pose estimation in 3D space. To this end, we propose general pipeline based on SS-Conv, which stacks layers of SS-Conv as the backbone, and decodes object poses directly from the learned 3E(3))equivarian features. A novel Senttur-Stteringg module also designed into the pipeline to support iterative pose refinement, by taking advantage of the steerability of the learned features. We conduct thorough experiments on three tasks of pose-related, 3D object semantic analysis, including instance-level 6D pose estimation, category-level 61) pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks; the gaps are clearer in the regimes of high-precision pose estimation. Ablation studies also show the superiority of our SS-Conv over alternative convolutions terms of both accuracy and efficiency.
2 Related Work
Equiv-aaquv Representation Learning SE(()-equivariance is an important property in 3D computer vision. In earlier works, researchers ease the problem by focusing SO(3)-equivariaace, and design Spherical CNNs 66, [5) by stacking -quui--auvva spherical convolutions which are implemented in the spherical harmonic domain. Recently, series of works 20 25 build networks based on steerable kernels, which are parameterized as linear
deep SE(3)) combinations of basis kernels. Thomas et al. firstly propose Tensor Field Network (TFN) (0 to features on irregular point clouds, and later, Fuchs et al. present SE(3)- Transformer, which extends TFN with attention mechanism. However, those networks working on point clouds are required to compute kernels with respect to different input points inefficiently. To tackle this problem, 3D steerable convolution (ST-Conv) (25 proposed to work on regular volumetric data, so that basis kernels with respect to regular grids could be pre-computed; however, it still encounters challenging computational demands due to the ignorance of data sparsity. Compared to the above methods, proposed sparse steerable convolution aims at efficient -qqivvraan
learn

---

representation learning for volumetric data, which is realized with sparse tensors to accelerate the computation.
Estimation and Tracking of Object Poses in 3D Space In the context of pose estimation, instance- level 6D pose estimation is classical and well-developed task, for which a body of works proposed. These works can be broadly categorized into three types: i) template matching [12] by constructing templates search for the best matched poses; ii) 2D-3D correspondence methods which establish 2D-3D correspondence via 2D keypoint detection [19 I7 or
114
16 followed by PnP algorithm to obtain the target pose; via deep networks. Recently, more challenging task of category-level 6D pose and size estimation formally introduced [24), aiming estimating poses of 3D unknown objects with respect to a categorical normalized object coordinate space (NOCS).
dense 3D coordinate predictions I
14
iii) direct pose regression [26
13 23
The early works [24 21 focus on regression of NOCS maps, and the poses can be obtained by aligning NOCS maps with the observed depth maps. Later, methods of direct pose regression are proposed thanks to the special designs of fusion of pose-dependent and cose-independent features (2). decoupled rotation mechanism [4, dual pose decoders [1S]. Motivated by [24], Wang et. al propose the task of category-level 6D pose tracking, aiming for the small change of object poses between two adjacent frames in sequence; they also present 6-PACK, pose tracker estimating the change of poses by matching keypoints of two frames.
Sparse Steerable Convolutional Neural Network
3.1 Background
3D Convolution A conventional 3D convolution can be formulated as follows:
fx= [1s fn((x)) / x(E (y)y.
JR3
where (x) RK,, fa+1(x) E RK*+1, and IR3 + R K XK,, is continuous learnable kernel.
Squii-Equv Given transformation (g) : KR + IR. K. for 3D rigid motion g ESE(3),
a 3D convolution in Eq. [ is SE(3) if there exists transformation Tn+1(g) Ka+1 +
R K such that
[T+ (g)f = [n [T (g)
(2)
Such an
convolution is steerable, since the feature fn+1 (x) can be steered by
Tn (g) in the feature space [25
In general, the transformation n (g) is group representation of SE(3), which satisfies nn (g1gg) Tn (g1 JT
T (g2). If g is decomposed into a 3D rotation E SO(3) and a 3D translation t E R. written
as g = tr, (g) (g) can be defined as follows:
(n (g))) [n, (tr)ff()))))) = (x)
t)),
where (r) K., + K. is an SO(3) representation. The illustration is given in Fig.
Constraint To guarantee SE(()--quivariance in Eq. it can be derived that the kernel x of 3D convolution must be rotation-steerate 25), which satisfies the following constraint:
<(rw) = Px+1 (F))()))
(4)
Irreducible Feature (r) is SO(3) representation, which can be decomposed into F, irreducible representations as follows:
F. pn (r(r) = @ D' 1=0
(5)
where Q is a K,, x Kn, change-of-basis matrix, D is the (24, + 1) x (21; + 1) irreducible Wigner-D matrix [8] of order 4 (4, 0, 2..... and + represents block -iikoodaaaonal construction of (D)(r))) so that K, = F. 24 + 1. Based on Eq. Fn irreducible features {fr E 2R. each f% (x) is associated with a Dr. When 4 = 0.
(x) can be constructed by stacking
(r) = so that f (x:) scalar invariant to any rotation; when 4 > 0, fí (x)) is vector which can be rotated by D (r).
3

---

3.2 Sparse Steerable Convolution
3D STeerable Convolution (ST-Conv) enjoys the property of -quivvaairree,nne;; however, as dis- cussed in Sec. [ it suffers from heavy computations as conventional 3D convolution does. Motivated by recent success of SParse Convolation (SP-Conv) [9, we propose novel design of Sparse Steer- able Convolution (SS-Conv) with sparse tensors, which takes the natural sparsity of 3D data into account, while strictly keeping the steerability of features.
Specifically, assuming K is a discretized, s X x s, cubic kernel with grid sites S (s an odd), our proposed SS-Conv can be formulated follows:
1,,,,, .,., }d
EE "mm-vES.a, (E).,a.y)-11 k( (y)) if o (n++(())
if on+ (x)
Sati (x) = fn)()) (0,
(6)
s.L. Vr E SO(3), <(r=) = Pn+ (r)
where on(2)) (ar) represents the state of site in the feature space RK,, on (a) denotes an inactive state at æz where La (x)) is in its ground state; when Jn is beyond the ground state, this site would be activated as o, (2))
In SS-Conv, we set the ground state as zero vector.
Compared with ST-Conv, our sparse version is accelerated in two ways: i) convolutions are conducted at activated output sites, not on the whole 3D volume. where the number of active sites only takes small proportion; ii) in the receptive field of each activated output site, only active input features are convolved. For these purposes,
represent the input and output features sparse tensors (H,, n, F,)) and (H Fn+1). respectively. H, and Hn+1 are hash tables recording the coordinates of active sites only, while F, and Fn+1 are feature matrices. For sparse tensor, its hash table and feature matrix correspond to each other row-by-row; that is, In+ 1, is the row number of æ in Hn+1, then Fo+ [,, f (ar)
In this respect, the goal of SS-Conv is to convolve (H,, with k to obtain (H Fn41). which can be implemented in three steps: i) Rotation-Steerabee Kernel Construction (cf. 3.2.1p for generation of k, ii) Site State Definition (cf. 3.2.2) for the output hash table Hn+1, and Sparse Convolutional Operation (cf. (.2.3) for the output feature matrix Fut1. We will introduce the detailed implementations shortly.
3.2.1 Rotation-Steerable Kernel Construction
The key to satisfy the rotation-steerable constraint (4) is to control the angular directions of feature vectors, and a recent research shows that spherical harmonics Y {У give the unique and complete solution (25). Linear combination of the basis kernels based on spherical harmonics produces the intation-steerabe convolutional kernel K.
For simplicity, firstly consider both input and output features individual irreducible ones of
orders / and k, respectively; the kernel k R3 combination of basis kernels x Jm,
+ 1(2++1) is parameterized as linear
3 + IR 1)
A+1
M (x)
Σ
Σ
w Jm, g Jm
- | 1- !
where
M.
Σ TT ((<))))
| 14
(8)
Q is a (2k + 1) x (2l + 1) change-of-basis matrix, also known as Clebsch-Gordan coefficients,
and tmth a continuous Gaussian radial function: ([==() = e - =|===l In the basis kernel Jm y controls the angular direction, while 60th controls the radial one; then Jmm linearly combined by learnable coefficients (mmm as in Eq. (7) further adjust the radial
We only discuss convolutions (stride = 1) in our paper, since as pointed out in 25), convolutions (stride
1) damage the smoothness of features and break the properties of equivariance. For feature downsampling, we follow [25] and use combination of convolution (stride 1) with an average pooling.

---

direction, which is the only degree of freedom in the process of optimization. Accordingly, the angular direction is totally controlled by Y such that the inaatio--steerbbee constraint is strictly followed. In addition, the total number of learnable parameters in Eq. (7) is M [2minn(k: (k: () 11 (M the number of selected {m))), which is, practice, marginally less than that of conventional 31> convolution, which has (2k + 1) + parameters.
Finally, assuming that the input and output features stacked irredueible features, whose orders are (4,-... (F., and (Ki,-. ,KF,1) respectively, the rotation-steerable kernel of SS-Conv can be constructed follows:
khil (x)
belen (x))
x(x)=
(9)
k F (2²)
* lon (x))
with the size of K x Kn, where Kn+1 = Ei=t" 2kF,11 + and K2 -E- 21v,
3.2.2 Site State Definition
The key to enable the efficiency of SS-Conv lies in the definition of site state. In general, for an output grid site æ, any of input sites in its receptive field are active, this site will be activated, and convolution at this site will be conducted; otherwise, this site will keep inactive, meaning that its feature will be directly set zero vector (representing the ground state) without convolutional operation. We formulate the above definition of state at site æ as follows:
( (..
y H, and y S
Ou+1 (a))
(10)
others
The output hash table Hn+1 is then generated as fx =n+1(()) = 1}.
The number of active sites defined in (TO) will increase fnyer-by-layer enabling long-range message transfer. However, if dozens or even hundreds of convolutions are stacked, the rapid growth rate of active sites would result in heavy computational burden and the so-called "submanifold dilation problem" (9. To alleviate this problem, we follow (9] and consider another choice of state definition in our SS-Conv, which keeps the output state consistent with the input one at same grid site, i.e., on+1 (xr = o (æ), such that H Hn. This kind SS-Conv without dilation makes it possible to construct a deep but efficient network for sparse volumetric data, and we term it as "Submanifold SS- SS-Com""
In practice, we mix general SS- SS-Convs and Submanifold SS-Convs in an alternating manner to achieve high accuracy and efficiency.
3.2.3 Sparse Convolutional Operation
After obtaining Hn+1, the next target is to compute the values of Fa+1- Specifically, we firstly initialize Fa+1 zeros; then the feature vectors in Fa+1 updated via the following algorithm:
ALGORITHM I: Sparse Steerable Convolution
Input: (H (n). Fn).
Fa+1), (K(*): 8)
Output: (H Fat1)
1: R R, 8:885
// Initialize the rule book R. // Construct the rule book R.
2: for in H do
3: for in H., do
4:
if
S: Append (F (nyy) ) to R..
5:
11 M1r41 is the row number of xt in H
6:
end for 7: end for
11 /Tray is the row number of y in H...
8:
for R., in R do
11 Update FASr-
9: for (Fot12, Frag) ) in Rs do F Foti/rutial + Fat1 [o+11a] k(8) x Falragl
10:
11: end for
12:
end for
This process can divided into two substeps. The first one to construct a rule book = RR,
E S), where active output site is paired with active input in each Rg, if x y The
5

---

Teaser-to Posiet
f
1
Stage
$ poyd
SS-Cons
Potro--bbe Vooduree
(r.
0r.4g)
(rutt)
Feature Steering Module
Pesture-Sivering Modille
Feature Steering Modali:
Feature Steerinng Medale
Stage 1wo
22
Palnt-wia Fentures
Tensor do-P Poine
Figure 2: An illustration of network architecture for instance-level 6D object pose estimation.
second one update Fn according to the paired relationships recorded R: for example, if the paired relationship of output æ and input y is recorded in R., the current f=+1(x) will be updated by adding the multiplication of fo (y) and (s). this process, the construction of R is very critical, which helps to implement the second substep by matrix-matrix multiply-add operations on GPUs efficiently.
3.3 Normalization and Activation
As conventional CNNs do, SS-Convs are also followed by normalization and activation, i.e., Ac- tivation(Normn(Ie f=))))+++)))))))) (x))). Those operations of normalization and activation are required to be specially designed, not to break the SE(3) (e(iiv-auuva of features. Since each feature is formed by stacking irreducible ones, without loss of generality, take an example an irreducible feature ((x with order so that the normalization can be formulated follows:
Norm(f(x)) { (f(x)) / f(x)/ B[I (1) +
+ 1=
(1I)
1> 0
where B[|] and Var[-] are population
and variance, respectively. is very small constant. For the activation of f(x). if 0, ReLU can be chosen to increase non-lincarity; if > 0, we follow (25) and multiply to f(x) a scalar, which is learned by SS-Conv and applied the Sigmoid function:
ReLU(f(x)), (igmoid([:: f())))((x).
t= 0
Activation(f(a))
(12)
1>0
The above normalization and activation operations are both -Euuierqui since a feature vector multiplying any scalar keeps its equivariance; when applying them features formed by numerous irreducible ones, we treat each irreducible member individually to ensure the equivariance.
4 Applications for Estimation and Tracking of Object Poses in 3D Space
4.1 Instance-level 6D Object Pose Estimation
Given an RGB-D D image of a cluttered scene, instance-level 6D pose estimation is to estimate the 6D poses of known 3D objects with respect the camera coordinate system. As introduced Sec. 3.1 a 6D pose g E SE(3) can be decomposed into 3D rotation SO(3) and 3D translation t E g which makes sparse steerable convolutional network well suited for this task, due to: SS-Convs extract strong SE(()--quivvriant features to decode a precise 6D pose; ii) the steerability of feature maps helps to enable a second stage of pose refinement. Therefore, we propose an efficient general pipeline based on SS-Convs for 6D pose estimation, depicted in Fig.

---

Specifically, we firstly segment out the objects of interest via an off-the- model of instance segmentation, assigning each object with an RGB segment and a cropped point cloud: then each 3D object is voxelized and represented by a sparse tensor (Ho: Fo), where each feature in Fo is 4 4-dimensional vector, containing RGB values and constant "1". For the input tensor, we set the site active if the quantified grid centered at this site encloses any points, and average point features of those enclosed by a same grid. (Ho, Fo) is then fed into our pipeline in Fig. where the pose estimation could be achieved in the following two stages.
In the first stage, we construct an efficient SS JS-Conv-based backbone, which extracts hierarchical SE(3)--quivariant feature maps, represented in the form of sparse tensors { }. Those feature tensors are used for interpolation of multi-level point-wise features by using a Tensor-to-Poine module, proposed in [TO], transforming features of discretized grid sites to those of real-world point coordinates. Each point feature fed into two separate MIPss, regressing a point offset and a rotation, respectively; the addition of the point coordinate and its offset generates translation. The initially predicted pose (r1, f1) of this stage is obtained by averaging point-wise predictions.
In the second stage, we refine the pose (r1, t1) by learning a residual pose (r2, t2), wherein a
Feature-Steering module is designed, generating transformed features { {HH,, F2)] } by efficiently
steering hierarchical backbone features {(H,, } individually with (r1, (2). Again we interpolate
point-wise features from {HH,, Fx) }. and average point-wise predictions obtain (r2, ty). Finally,
the predicted 6D pose is updated as (2172, t1 trtz)) ). In addition, owing to the novel reature-Steering modules, this stage can be iteratively repeated, generating finer and finer poses.
4.1.1 The Feature-Steering Module
Feature- Steuuringg module in the pipeline is to transform (Hn, F..) of the backbone to (H', Fz), where rigid transformation of H,, with (r. r) and a rotation of F, with p(r) are included. Specifically, for Fn. we compute p(r) as defined in (5) and rotate F, by matrix multiplication; for Hn, we convert the sites in it to the real-world point coordinates, which are then applied of (r, t) and re- voxelized as grid sites. The same new sites are merged to a unique one, while their features averaged. We also and activation, to enrich the new features and generate the final steered HH,, F.).
rigid transformation
another SS-Convs, each followed by steerable normalization
4.2 Category-Ievel 6D Object Pose and Size Estimation
Category-Ievel 6D pose and size estimation is formally introduced in 24 This is a more challenging task, which aims to estimate categorical 6D poses of unknown objects, and also the 3D object sizes. To tackle this problem, we use a similar network as that in Fig. 2 and make some adaptive modifications: for each stage in Fig. 2 we add another two separate MLPs for point-wise predictions of 3D sizes and point coordinates in the canonical space, respectively; ii) in each Feature-Steering module, the real-world coordinates of all 3D objects are also scaled by their predicted 3D sizes to be enclosed within a unit cube. for estimating more precise poses.
4.3 Category-level 6D Object Pose Tracking
Motivated by the above task of categorical pose estimation, category-level 6D pose tracking is also proposed to estimate the small change of 6D poses in two adjacent RGB-D frames of an image sequence [22]. Due to the available pose of the previous frame, the target object can be roughly located in the current frame, avoiding the procedures of object detection or instance segmentation in images. However, without a precise mask, the estimation of small pose change from noisy 3D) data is a big challenge for deep networks. Our sparse stecrable convolutional network also surprisingly performs well in such noisy data, even though we only conduct one-stage pose estimation that achieves real-time tracking. For more details, one may refer to the supplementary material.
5 Experiments
Datasets We conduct experiments on the benchmark LineMOD dataset 11 for instance-level 6D pose estimation, which consists 13 different objects. For both category-level 6D pose estimation and tracking, we experiment on REAL275 dataset [24). which is a more challenging real-world
7

---

Table 1: Quantitative comparisons of Plain12 based on different convolutions on the LineMOD
dataset [11]
Conv
ADD(S) t
FPS
# Param
Dense-Conv
46.5
224
26.2 M
SP-Conv
62.8
486
26.2 M
ST-Conv
92.8
148
3.6 M
SS-Conv
93.5
404
3.6 M
0.06
Evaluation Loss
0.05
Derce
Cony
0.25
0.05
Training
0.04
0.62
0.1
0.01
0.00
0.0
0.4
1.2
1.6
2.4
0.00
0.0
0.8
1.2
2.0
Iteration|x
10f)
Iteration(x 10")
(a)
(b)
Figure 3: Training of Plain12 based on different convolutions on the LineMOD dataset
dataset with 4, 300 training images and 2, 750 testing ones, containing object instances of 6 categories.
Following 24 22 we augment the training data of REAL275 with synthetic RGB-D images.
Evaluation Metries For instance-level task, we follow (23) and evaluate the results of LineMOD dataset on ADD(S) metrie. For the category-level tasks, we report the mean Average Precision (mAP) of intersection over union (IoU) and n'm cm, following [24]; mean rotation error (Terr) in degrees and mean translation eтrоr (terr) in centimeters are also reported for pose tracking. Additionally, we compare the numbers of parameters (#Para)) and the running speeds (FPS) for different models. Testing conducted on a server with GeForce RTX 2080ti GPU for batch size of 32, and FPS is computed by averaging the time cost of forward propagation on the whole dataset.
5.1 Comparisons with Different 3D Convolutions
We firstly conduct experiments to compare proposed SS-Conv with other kinds of 3D convolutions, including conventional 3D convolution (Dense-Conv) sparse convolution (SP-Conv) (9, and steerable convolution (ST-Conv) (25). on the LineMOD dataset for instance-leve 6D pose estimation. Among those convolutions, SP-Conv improves the speed of Dense-Comy by considering data sparsity and turns out be efficient in some tasks of 3D semantic analysis(e.g.. 3D object detection), while ST-Conv constructs rotation-steerable kernels and then realizes the convolution based on Dense-Conv.
To meet various computational demands of different convolutions, those experiments are conducted on a light plain architecture, termed as Plain12, in the same experimental settings, for fair comparison. The architecture consists of 12 convolutional layers, of which the kernel sizes are all set as 3 x 3 x 3; for SS-Convs and ST-Convs, we set the superparameters of radial function upth Eq. (8) as {m} = {0, 1} and 0.6. We use ADAM to train the networks for total of 30, 000 iterations, with an initial learning rate of 0.01, which halved every 1, 500 iterations. We voxelize the input segmented objects into 64 x 64 x 64 dense/sparse grids, and set the training batch size 16.
Quantitative results of different convolutions are listed in Table which confirms the advantages of our SS-Conv in both accuracy and efficiency. In terms of accuracy. SS-Conv achieves comparable results on ADD(S) metric as ST-Conv does, which significantly outperforms those of Dense-Cony and SP-Conv, indicating the importance of preservation of relative poses of features layer-by-layer, the property of SE(3)-equivariance makes feature learning capture more information of object poses. We also visualize the behaviors of the four convolutions in the process of training in Fig. 3 where the learning based on SS-Conv/ST-Conw converges better and faster than that of Saans--Conv/S-CCCC
feature learning on pose estimation

---

Plain12
Plain24
ResNet50
Bottch sines 512
PPS:2165 2063
Ratck sient 512
Bitth
387.6
120
Batch
119% SUAL
Rutch
100
145.4
Botch sime 515 -4.7
20 28 14r 26 2011
Usatch staz: Mensary: 18:G
Momary: 18A6
Butch seec Menuury 9.70
Turch slre $11 Mennoyy RUDG
Memory(C)
Memory/a)
Memoryy
Butch size 512 Memoryy 334G
Butcc
Menory
Batch das 15
Mensoyy: LG2
UB 52 3024
Batch size
Batch size
Batch size
Figure 4: Plottings of FPS and memory consumption versus different batch sizes for different networks based ST-Conv/SS- Experiments conducted on LineMOD dataset
Table 2: Quantitative comparisons of different methods the LineMOD dataset for instance- level 6D object pose estimation. The evaluation metric is ADD(S).
Implicit +ICP
[13 +ICP
PointFusion
DenseFusion
DenseFusion (Iteaative) 25
G2L
Ours w/o second stage
Ours
27
23
ape
20.6
65
70.4
79.5
92.3
96.8
92.9
97.4 4
bench.
64.3
80
80.7
84.2
93.2
96.1
97.4
99.3
camera
63.2
78
60.8
76.5
94.4
98.2
97.7
99.5
can
86.6
93.1
98.0
96.1
99.6
cal
72.0
88.8
96.5
98.8
99.8
driller
41.6
73
47.3
77.7
87.0
99.8
98.7
99.6
duck
32.4
66
63.0
76.3
92.3
97.7
91.1
97.8
98.6
100
99,9
99.9
99.8
100.0
100.0
99.9
glue
96.4
100
99.3
99.4
100.0
100.0
98.6
99.6
hole.
49.9
49
71.8
79.0
92.1
99.0
96.3
99.4
iron
83.2
97.0
98.7
99.2
lamp
91.7
73
62.3
92.3
95.3
99.5
99.5
99.7
phone 71.0
79
78.8
88.0
92.8
98.9
97.5
98.2
MEAN
64.7
79
73.7
86.2
94.3
98.7
97.2
99.2
In of efficiency, our sparse steerable convolutional networks are more efficient and flexible for complex systems, e.g., for Plain 12, SS- SS-Co brings about 2.7x speedup w.r.t. ST-Conv (404 FPS versus 148 FPS) with batch size of 32, as listed Table I More results of FPS with improved sizes of data batches are given in Fig. 4 where ST-Conv can be only run at the extreme batch size of on the GPU with 12G memory. while SS-Conv batch size as large as 512; running with larger batch sizes further improves the efficiency of our proposed SS-Conv (FPS goes to 725 when running with the batch size of 512 on Plain 12). We also compare the efficiency of ST-Conv and SS-Con S-CCov two other deeper networks (dubbed Plain24 and ResNet50, respectively): as shown in Fig. 4 SS-Conv consistently improves FPS over ST-Conv on these two architectures, with less GPU memory consumption.
much less memory and can thus support
5.2 Comparisons with Existing Methods
Instance-level 6D Object Pose Estimation For the instance-level task, we compare the results of our SS-Conv-based pipeline with existing methods on LineMOD dataset [Л]. Quantitative results

---

Table 3: Quantitative comparisons of different methods on REAL275 dataset 24 for category-level 6D object pose and size estimation.
mAP
Method
IoU
IoU7s
5
5cm
10º2cm
10m5
NOCS
24
78.0
30.1
7.2
10.0
13.8
25.2
SPD
21
77.3
53.2
19.3
21.4
43.2
54.1
CASS
77.7
23.5
58.0
FS-Net
92.2
63.5
28.2
60.8
DualPoseNet
[5
79.8
62.2
29.3
35.9
50.0
66.8
Ours w/o second stage Ours
79.5
58.7
19.2
25.2
35.1
49.9
79.8
65.6
36.6
43.4
52.6
63.5
Table 4: Quantitative comparisons of different methods on REAL275 dataset 24 for category-level 6D object pose tracking.
Method
Metric
bottle
bow
camera
can
laptop
mug
MEAN
5 5cm 1oU2s* t
24.5
55.0
10.1
22.6
63.5
24.1
33.3
91.1
100.0
87.6
92.6
98.1
95.2
94.1
6-PACK 22
Terr +
15.6
5.2
35.7
13.9
4.7
21.3
16.1
terr +
4.0
1.7
5.6
4.8
2.5
2.3
3.5
5 5cmm
70.3
60.6
10.6
49.9
87.7
47.9
54.5
IoU2s f
93.5
99.9
99.9
99.8
99.8
99.9
98.8
Ours
Terr +
3.7
4.6
9.8
4.6
3.0
5.6
5.2
terr 1.9
1.2
2.0
2.7
2.4
1.1
1.9
shown in Table where two-stage pipeline outperforms all the existing methods and achieves a new state-of-the-arr result of 99.2% on mean ADD(S) metric. We can also observe that the second stage of pose refinement with Feature-Steering modules in pipeline indeed improves the predictions in the first stage, benefitting from the steerability the feature spaces SS-Convs.
lete 6D Object Pose and Size Estimation We conduct experiments on REAL275 [24] for the more challenging category- task. Quantitative results in Tableecoonfin the advantage of our pipeline in the high-precision regime, especially on the precise metric of 5 where improve the state-of-the-art result in [15] from 35.9% to 43.4%. The second stage of pose refinement also plays important role in this task, achieving remarkable improvements over the first stage.
Category-level 6D Object Pose Tracking We compare the results of our one-stage tracking pipeline with the baseline of 6-PACK (22) on REAL275 244. In 6-PACK, the relative pose between two frames computed based on predicted keypoint pairs inefficiently, while our pipeline regresses the pose in direct way. The results in Table 4 show that our pipeline outperforms 6-PACK on all the evaluation metrics, demonstrating the ability of Conv--based network for fine- aneeggained pose estimation in noisy input data.
More implementation details and qualitative results shown in the supplementary material.
Broader Impact
The studied problems of object pose estimation and tracking in 3D space are very important many real-world applications, including augmented reality, robotic grasping, and autonomous driving. By precisely predicting object poses
3D space, virtual contents could be seamlessly embedded in real environments. creating fascinating personal experience; the contrary, less precise predictions may cause property loss and even life threat, especially autonomous driving. The contributed solution based on SS-Cony would improve the overall level of safety.
10

---

Acknowledgments and Funding Disclosure
This work was partially supported by the Guangdong R&D key project of China (No.: 20198010155001,, the National Natural Science Foundation of China (No.; 61771201), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No.: 20172T07X183).
References
[1] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In European conference on computer vision, pages 336-551. Springer. 2014.
[2] Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11973-1 11982, 2020.
[3] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, and Ales Leonardis. G2l-net: global to local network for real-time 6d pose estimation with embedding vector features. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4233-4242, 2020.
[4] Wei Chen, Xi Jia, Hyung Jin Chang, Jimming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. arXiv preprint vxXiv2203.00054. 2021.
[5] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In International Conference on Learning Representations, 2018
[6] Carlos Esteves, Christine Allen-Blanchee Blanchette, Ameesh Makadia, and Kostas Daniilidss Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68, 2018
[7] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3>-transformers: 3d roto-tramslation cquivariant attention networks. Advances in Neural Information Processing Systems, 33, 2020.
[8] Robert Gilmore. Lie groups. physics, and geometry: introduction for physicists, engineers and chemists. Cambridge University Press, 2008.
[9] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and puttern recugnition, pages 9224-9232, 2018.
[10] Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11873 11882, 2020.
[11] Stefan Hinterstoisser, Stefan Holzer, Cedric Cagniart, Slobodan Ilie, Kurt Konolige, Nassir Navab, and Vincent I Lepetit. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In / international conference on computer vision, pages 858-865. IEEE, 2011.
[12] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilie, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference computer vision, pages 548-562. Springer, 2012.
[13] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilie, and Nassir Navab. Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again. Proceedings of the IEEE international conference on computer vision, pages 1521-1529, 2017.
[14] Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Learning snalysis-by-yytteeiis for 6d pose estimation in rgb-d images. In Proceedings of the IEEE international conference computer vision, pages 954-962, 2015.
[15] Jiehong Lin, Zewei Wei, Zhihao Li. Songeen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-Ieve 6d object pose and size estimation using dual pose network with retined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3560-3569 October 2021.
11

---

[16] Frank Michel, Alexander Kirillov, Eric Brachmann. Alexander Krull, Stefan Gumhold, Bogdan Savchyn- skyy, and Carsten Rother. Global hypothesis generation for 6d object pose estimation. In Proceedings the IEEE Conference on Computer Vision and Puttern Recoognition, pages 462-471, 2017.
[17] Sida Peng. Yuan Liu, Qixing Huang. Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network 6dof pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561-4570, 2019.
[18] Martin Sundermeyer, Zoltan- Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 1999-115,7 2018.
[19] Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6d object pose prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pages 292-301, 2018.
[20] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-cquuiariann neural networks for 3d point clouds. arXiv preprint arXiv: /8Xx::8002.080 08219, 2018.
[21] Meng Tian, Marcelo H Ang. and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In European Conference on Computer Vision, pages 530 546. Springer, 2020.
[22] Chen Wang, Roberto Martin,Martii, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category- -eeeel 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066) IEEE, 2020.
[23] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin. Cewu Lu, Fei-Fei, and Silvio Savarese. Densefusion: ohject pose estimation by iterative dense fusion. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3343-3352, 2019.
[24] He Wang. Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2642-2651, 2019.
[25] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, arXiv preprint arXiv: 1007005447 2018.
[26] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv: 171 1,00 2017.
[27] Danfci Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 244-253, 2018
12


Reviews:
[{"review_id": "ytnkAD6DEE", "reviewer": "Reviewer_dFJh", "summary": "a) This paper aims to learn SE(3)-equivariant deep features;\n\nb) The major contribution is propose a Sparse Steerable Convolution (SS-Conv) to address the shortcoming of steerable convolution;\n\nc) Experiments on three tasks of 3D object semantic analysis show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points.\n\na) The pipeline of Sparse Steerable Convolution is given in detail;\n\nb) Experimental results are promising.\n\n\nWeak points.\n\na) I cannot find significant weaknesses of this paper, only minor comments.\n\nThe main contribution of this paper is changing the Steerable Convolution to sparse. As such, I would see comparisons/discussions of the proposed method with respect to some off-the-shelf sparse 3D convolutions other than [9].\n\nFor example, the Minkowski Engine. https://github.com/NVIDIA/MinkowskiEngine\n\nWhen checking equation 6, I found it is similar to the Minkowski Engine, other than the SE(3)-equivariant constraint.\n\nb) It is not clear how the rotation is parametrized throughout the paper.\nSpecifically, in Figure 2, how many parameters are used to denote r_1 or r_2?\n\nc) If I understand correctly, the proposed method can handle instance-level or category-level point clouds. Can this paper handle more general cases, e.g., 3D scans of a room? \n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "uKEwDVxlU01", "reviewer": "Reviewer_gTpb", "summary": "The paper presents a Sparse Steerable Convolution to accelerate steerable convolution (ST-Conv) via sparse tensors, while still preserving the SE(3)-equivariance. \n\nThe authors also show its application to 3d semantic analysis including the tasks of instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking and they achieve superior performance on these datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The authors present a new Sparse Steerable Convolution (SS-Conv) that speeds up the existing steerable convolution (ST-Conv), by borrowing benefits from recently introduced sparse-tensor-based computation frameworks. It is clear that this paper presents a non-trivial attempt in making the steerable convolution more practical to handle sparse data, e.g., irregular and sparse 3D point clouds.\n\n**Quality**: The paper is technically sound. The motivation of using sparse tensor computation is intuitive. However, the authors are suggested to conduct experiments to directly compare steerable convolution and the proposed sparse steerable convolution along several dimensions, including memory consumption, training/inference times under a similar resolution. Also, it could be more intuitive to actually quantify the sparsity of the tackled datasets. For example, to count the number of occupied grids/voxels with a fixed resolution.  \n\n**Clarity**: The paper is well-written. Most of the experimental results supported the claims. \n\nIt is not clear how the authors conduct the experiment of Dense-Conv. From reading the paper, the reviewer didn’t find any detail of Dense-Conv including architecture, input resolutions, input types (image-like dense tensors???). \n\n**Significance**: The reviewer believes that Sparse Steerable Convolution could be a useful operator for tasks such as pose estimation and tracking in real-world applications, which could potentially make a larger impact once it is published.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Main Review for making potential improvements to the paper.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QQ89YtK9j-p", "reviewer": "Reviewer_BhGd", "summary": "This paper proposes a sparse SE(3)-equivariant convolution for voxel-based deep networks.  To avoid high computational costs of dense voxel processing, only active voxels are processed.  Every other layer increases the number of active voxels through dilation to account for the filters' range of influence.  The proposed sparse convolution is embedded into a network that first performs an approximative matching, then a refinement, and is applied to 6 DoF object pose estimation, 6 DoF category pose estimation, and 3D object tracking.  All networks are evaluated on standard datasets, showing very good results and improved computational performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overview\n\nThe paper introduces a well thought-off and generic method can can be used as building block for many 3D applications. The method is well described, practical and shows very good results compared to state of the art, including an impressive 99.2% detection rate on LineMOD. The theoretical contribution is overall incremental over [24] and [9]. The exact contribution of the introduced method are not quite clear from the experiments.\n\n\n# Details\n\n* The evaluation is reasonably convincing regarding the overall performance of the proposed networks. It is performed on large, standard datasets, and the proposed method is compared to state of the art.  The results are very good, especially the refinement part (\"Stage Two\") of the network appears to have a significant effect on improving the predictions.\n\n* The writing is good, the method is well motivated and explained and can likely be reproduced based on the paper.\n\n* The introduced convolutions heavily relies on [24], especially in Sec. 3.1, 3.2.1, 3.3, and [9] in 3.2.2 (though, arguably, [9] has a more interesting approach to the sparsity problem). Overall, it appears to me that the SE(3)-equivariance is borrowed from [24], and the main contribution for the convolutions is the usage of a sparse instead of a dense voxel grid, coupled with dilating the region of active voxels over time.\n\n  Regarding the overall network design, the main contribution is using a two-stage approach where the second network further refines the output of the first. This approach leads to significantly more accurate results, as is visible in the experiments.\n  \n* While the impact of the introduced sparse voxel grid is analyzed in ablation studies, those are not quite clear to me and several open questions are below.\n\n\n# Open Questions\n\nFrom most to least important.\n\n* Table 1: Are the FPS computed for complete images, or for single patches (found by the preceeding object detection / instance segmentation network)? Overall, giving concrete timings for inference would be a plus (i.e. how much runtime does the network use for processing one LineMOD image? How much runtime does the instance segmentation network take?). Does the FPS include the instance segmentation, or is it only the time spent in the proposed network, processing the detected instances? I am especially curious about this because while sparse voxels often have a theoretical advantage, that is often mitigated in practice because of the required out-of-order memory access (here lines 2--7 in Algorithm 1), which is not well suited for modern GPU architectures.\n\n  Also in Table 1, are the ADD(S) and FPS for ST-Conv also measured using the proposed two-stage network?\n\n* Sec. 3.1 / Figure 2: I am unsure what the input to the MLPs is, since the number of active sites can change, but MLPs need a constant number of inputs. Maybe the effect of the Tensor-to-Point-Block should be described in 1-2 sentences.\n\n* Sec. 3.3: What is the motivation for using a different normalization and activation for l=0?\n\n* Why is there a difference in the ADD(S)-metric of the proposed method between Table 1 and Table 2 (SS-Conv vs. Ours)?\n\n* Supplementary Material: When using a \"ResNet50\" like backbone (Appendix A), how are skip connections implemented if the dimensions of the output of a layer are larger than that of the input (as done according to Eq.(10) of main paper)?\n\n* l.260--261: What does \"feature channels are kept consistent\" mean exactly?\n\n* l.193: Which instance segmentation network was used, exactly? I believe this should be mentioned in the paper.\n\n\n# Typos etc., no need to respond:\n\n* l.267: I believe \"marginally\" is the wrong word here, since the gap is quite large. Similar in l.138.\n\n* For consistency, think about making the best values bold in Tables 1 and 4.\n\n\n# Potentially interesting citiations\n\nThe following citiations might be interesting for the paper, though there is no requirement to include them:\n\n* Zhao, Yongheng, et al. \"Quaternion equivariant capsule networks for 3d point clouds.\" European Conference on Computer Vision. Springer, Cham, 2020.\n    \n* Shen, Wen, et al. \"3d-rotation-equivariant quaternion neural networks.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. Springer International Publishing, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact was not discussed and was not expected by the reviewer for topic at hand. Limitations of the method were not discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Pp1lwO6VFNl", "reviewer": "Reviewer_zYbA", "summary": "The paper proposed to equip the 3D steerable convolution with the sparsity handling capability for more effective use of the computing resources. In particular, only the activated grid locations among the feature map (sigma_{n+1}(x)) are computed for feature values. To achieve this goal, a rule book R is constructed before conducting the sparse convolution propagation. This results in an iterative forwarding procedure among the network forwarding flows. The authors cultivated several useful applications of the proposed framework and conducted several experiments to indicate that the proposed approach seems to be able to outperform existing works.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to handle sparsity for steerable convolution for better practical use. I believe the motivation is good and the problem addressed in this paper provides a nice example to better deploying theoretical contribution.\n\nMy concerns toward the weaknesses of the paper are as follows and I wish the authors address them in the rebuttal.\n\nThis paper seems not particularly exciting with respect to its novelty, and particularly, to me this paper seems to be a quick combination of [24] and [9]. My major concern is whether using the techniques of the spherical harmonics in Sec. 3.2.1 are particularly important under the sparsity settings. If so, I hope the authors can provide a clear explanation in both the rebuttal and the final version, on why this part (spherical harmonics in Sec. 3.2.1) is essential to the sparsity setting, e.g. without applying spherical harmonics, why existing art cannot be well suited for the sparsity setting. I understand that the spherical harmonics satisfy the rotation steering constraints, which is an important property to be used, but its connection to the sparse convolution has not been thoroughly discussed. Quick issues for Sec. 3.2.1: m seems not defined in L137 (Is it an iterator or the number of bases in a row of the spherical harmonics?). It would also be helpful to state that each element in the matrix (Eq 9) is in the size of s by s by s.\n\nThere are also concerns from the experiments. First, I am concerned that the speed acceleration seems not particularly exciting (one time faster than the non-sparse counterpart). Second, I did not find comparisons to the non-sparse counterpart, e.g. [24]. It is OK to be slightly worse than the non-sparse counterpart, but it would be essential to see its trade-off between the performance drop and the speed acceleration. Third, I appreciate that the authors cultivated several important applications as a result of the proposed framework, but I also wish to see its comparison on the original task (e.g. 3D object classification or recognition) to the existing art.\n\nPlease address my concerns in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential social impact in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a Sparse Steerable Convolution to accelerate steerable convolution (ST-Conv) via sparse tensors, while still preserving the SE(3)-equivariance.
The authors also show application of the proposed method to 3D semantic analysis including the tasks of instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking and they achieve superior performance on these datasets.
All reviewers agree on the soundness of the proposed method and its efficiency over steerable convolutions or dense convolutions. 

Decision: Accept (Poster)