Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space
Authors: Jiehong Lin
Hongyang Li
Ke Chen
Jiangbo Lu
Kui Jia
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D convolution
SE(3)-equivariant feature learning
pose estimation

Abstract: As a basic component of SE(3)-equivariant deep feature learning, steerable convolution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3)-equivariance. Based on SS-Conv, we propose a general pipeline for precise estimation of object poses, wherein a key design is a Feature-Steering module that  takes the full advantage of SE(3)-equivariance and is able to conduct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Ablation studies also show the superiority of our SS-Conv over alternative convolutions in terms of both accuracy and efficiency. Our code is released publicly at https://github.com/Gorilla-Lab-SCUT/SS-Conv.

Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Eqiivarian Features for Estimation and Tracking of Object Poses in 3D Space
Jiehong ! Hongyang ; Chen 1.3 Jiangbo Lu Kui Jia 13,51 South China University of Technology. "DexForce Co. Ltd. Peng Cheng Laboratory. "SmartMore Technology Co. Ltd. 'Pazhou Laboratory. {lin. j jiehong, eeli Bonngyang/mmail.comm scuat. edu. cn jiangbo lu@gmail... com, {chenk, kuijia))scut.. edw.cuuteeecccm
Abstract
As basic component of SE(3)-cquivarian deep feature learning, steerable con- volution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3) equivarqui Based on SS-Conv, we propose general pipeline for precise estimation of object poses, wherein key design is a Feature-Steerin module that takes the full advantage of SE(3)-equivarianc and is able to con- duct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, level 6D pose tracking. Our proposed pipeline based SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Abla- tion studies also show the superiority of our SS-Com over alternative convolu- tions terms both accuracy and efficiency. Our code is released publicly at https:// tuu..u..... Com//orinn -aarrllllaabbbba -aa--aa--S-SUUUSCComm
category-
Introduction
-Euuiiaaaarrr deep networks 20 25 7 have shown the promise recently in some tasks of 3D semantic analysis, among which 3D Steerable CNN (25) is representative one. 3D Steerable CNNs employ steerable convolutions (termed as ST-Conv) to learn pose-equivariant features in layer-wise manner, thus preserving the pose information of the 3D input. Intuitively speaking, for a layer of ST-Conv, any SE(() transformation (r, t) applied to its 3D input would induce a pose-synchronized transformation to its output features, where
SO(3) stands for rotation and t E 1 for translation. Fig. (a) gives an illustration where given an SE(3) transformation of the input, the locations which feature vectors and the feature vectors themselves are also rotated by p(r) (p(r) is representation of rotation r). This property of SE(3)-equivariance enables the steerability of feature space. For example, without transforming the input, SE(3) transformation can be directly realized by steering in the feature space. To produce steerable features, ST-Conv confines its feature domain to regular grids of 3D volumetric
defined are rigidly transformed with respect to (r, t).
"Equal contribution CCoresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

ST-Com
Co-Comv
(64
(r.4)
ST-Com
fer
SS-Comv
pM
for (x (x-00
[ntre/]] = p1r
[cort/]]on ppr) 60)
(a)
(b)
Figure 1: An illustration of 3E(3)-equivarian based on dense tensors, and (b) our Sparse Steerable Convolution (SS-Conv) based on sparse tensors, where arrows defined the 3D fields denote vector-formed, oriented features. Best view in the electronic version.
achieved by (a) STeerable Convolution (ST-Conv)
data; it can thus be conveniently supported by 3D convolution routines. This compatibility with 3D convolutions eases the implementation of ST-Conv, but at the sacrifice of efficiently processing 3D> data (e.g., point clouds) that are typically irregular and sparse; consequently, ST-Conv is still less widely used in broader areas of 3D semantic analysis.
In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the aforementioned shortcoming faced by ST-Conv. SS-Conv can greatly accelerate steerable con- volutions with sparse tensors, while strictly preserving the SE(3)--quivariance in feature learning; Fig. [ (b) gives the illustration. To implement SS-Conv, we construct convolutional kernels as linear combinations of basis ones based on spherical harmonics, which satisfy the rotation-steerable constraint -equiaaaav convolutions
[25], and implement the convolution as matrix-matrix multiply-add operations on GPUs only at active sites, which are recorded along with their features as sparse tensors.
Although SEu3i-eqqivar feature learning is widely used in 3D object recognition, its potentials for other tasks of 3D semantic analysis have not been well explored yet. In this work, we make the attempt to apply our proposed SS-Conv to object pose estimation in 3D space. To this end, we propose general pipeline based on SS-Conv, which stacks layers of SS-Conv as the backbone, and decodes object poses directly from the learned 3E(3))equivarian features. A novel Senttur-Stteringg module also designed into the pipeline to support iterative pose refinement, by taking advantage of the steerability of the learned features. We conduct thorough experiments on three tasks of pose-related, 3D object semantic analysis, including instance-level 6D pose estimation, category-level 61) pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks; the gaps are clearer in the regimes of high-precision pose estimation. Ablation studies also show the superiority of our SS-Conv over alternative convolutions terms of both accuracy and efficiency.
2 Related Work
Equiv-aaquv Representation Learning SE(()-equivariance is an important property in 3D computer vision. In earlier works, researchers ease the problem by focusing SO(3)-equivariaace, and design Spherical CNNs 66, [5) by stacking -quui--auvva spherical convolutions which are implemented in the spherical harmonic domain. Recently, series of works 20 25 build networks based on steerable kernels, which are parameterized as linear
deep SE(3)) combinations of basis kernels. Thomas et al. firstly propose Tensor Field Network (TFN) (0 to features on irregular point clouds, and later, Fuchs et al. present SE(3)- Transformer, which extends TFN with attention mechanism. However, those networks working on point clouds are required to compute kernels with respect to different input points inefficiently. To tackle this problem, 3D steerable convolution (ST-Conv) (25 proposed to work on regular volumetric data, so that basis kernels with respect to regular grids could be pre-computed; however, it still encounters challenging computational demands due to the ignorance of data sparsity. Compared to the above methods, proposed sparse steerable convolution aims at efficient -qqivvraan
learn

---

representation learning for volumetric data, which is realized with sparse tensors to accelerate the computation.
Estimation and Tracking of Object Poses in 3D Space In the context of pose estimation, instance- level 6D pose estimation is classical and well-developed task, for which a body of works proposed. These works can be broadly categorized into three types: i) template matching [12] by constructing templates search for the best matched poses; ii) 2D-3D correspondence methods which establish 2D-3D correspondence via 2D keypoint detection [19 I7 or
114
16 followed by PnP algorithm to obtain the target pose; via deep networks. Recently, more challenging task of category-level 6D pose and size estimation formally introduced [24), aiming estimating poses of 3D unknown objects with respect to a categorical normalized object coordinate space (NOCS).
dense 3D coordinate predictions I
14
iii) direct pose regression [26
13 23
The early works [24 21 focus on regression of NOCS maps, and the poses can be obtained by aligning NOCS maps with the observed depth maps. Later, methods of direct pose regression are proposed thanks to the special designs of fusion of pose-dependent and cose-independent features (2). decoupled rotation mechanism [4, dual pose decoders [1S]. Motivated by [24], Wang et. al propose the task of category-level 6D pose tracking, aiming for the small change of object poses between two adjacent frames in sequence; they also present 6-PACK, pose tracker estimating the change of poses by matching keypoints of two frames.
Sparse Steerable Convolutional Neural Network
3.1 Background
3D Convolution A conventional 3D convolution can be formulated as follows:
fx= [1s fn((x)) / x(E (y)y.
JR3
where (x) RK,, fa+1(x) E RK*+1, and IR3 + R K XK,, is continuous learnable kernel.
Squii-Equv Given transformation (g) : KR + IR. K. for 3D rigid motion g ESE(3),
a 3D convolution in Eq. [ is SE(3) if there exists transformation Tn+1(g) Ka+1 +
R K such that
[T+ (g)f = [n [T (g)
(2)
Such an
convolution is steerable, since the feature fn+1 (x) can be steered by
Tn (g) in the feature space [25
In general, the transformation n (g) is group representation of SE(3), which satisfies nn (g1gg) Tn (g1 JT
T (g2). If g is decomposed into a 3D rotation E SO(3) and a 3D translation t E R. written
as g = tr, (g) (g) can be defined as follows:
(n (g))) [n, (tr)ff()))))) = (x)
t)),
where (r) K., + K. is an SO(3) representation. The illustration is given in Fig.
Constraint To guarantee SE(()--quivariance in Eq. it can be derived that the kernel x of 3D convolution must be rotation-steerate 25), which satisfies the following constraint:
<(rw) = Px+1 (F))()))
(4)
Irreducible Feature (r) is SO(3) representation, which can be decomposed into F, irreducible representations as follows:
F. pn (r(r) = @ D' 1=0
(5)
where Q is a K,, x Kn, change-of-basis matrix, D is the (24, + 1) x (21; + 1) irreducible Wigner-D matrix [8] of order 4 (4, 0, 2..... and + represents block -iikoodaaaonal construction of (D)(r))) so that K, = F. 24 + 1. Based on Eq. Fn irreducible features {fr E 2R. each f% (x) is associated with a Dr. When 4 = 0.
(x) can be constructed by stacking
(r) = so that f (x:) scalar invariant to any rotation; when 4 > 0, fí (x)) is vector which can be rotated by D (r).
3

---

3.2 Sparse Steerable Convolution
3D STeerable Convolution (ST-Conv) enjoys the property of -quivvaairree,nne;; however, as dis- cussed in Sec. [ it suffers from heavy computations as conventional 3D convolution does. Motivated by recent success of SParse Convolation (SP-Conv) [9, we propose novel design of Sparse Steer- able Convolution (SS-Conv) with sparse tensors, which takes the natural sparsity of 3D data into account, while strictly keeping the steerability of features.
Specifically, assuming K is a discretized, s X x s, cubic kernel with grid sites S (s an odd), our proposed SS-Conv can be formulated follows:
1,,,,, .,., }d
EE "mm-vES.a, (E).,a.y)-11 k( (y)) if o (n++(())
if on+ (x)
Sati (x) = fn)()) (0,
(6)
s.L. Vr E SO(3), <(r=) = Pn+ (r)
where on(2)) (ar) represents the state of site in the feature space RK,, on (a) denotes an inactive state at æz where La (x)) is in its ground state; when Jn is beyond the ground state, this site would be activated as o, (2))
In SS-Conv, we set the ground state as zero vector.
Compared with ST-Conv, our sparse version is accelerated in two ways: i) convolutions are conducted at activated output sites, not on the whole 3D volume. where the number of active sites only takes small proportion; ii) in the receptive field of each activated output site, only active input features are convolved. For these purposes,
represent the input and output features sparse tensors (H,, n, F,)) and (H Fn+1). respectively. H, and Hn+1 are hash tables recording the coordinates of active sites only, while F, and Fn+1 are feature matrices. For sparse tensor, its hash table and feature matrix correspond to each other row-by-row; that is, In+ 1, is the row number of æ in Hn+1, then Fo+ [,, f (ar)
In this respect, the goal of SS-Conv is to convolve (H,, with k to obtain (H Fn41). which can be implemented in three steps: i) Rotation-Steerabee Kernel Construction (cf. 3.2.1p for generation of k, ii) Site State Definition (cf. 3.2.2) for the output hash table Hn+1, and Sparse Convolutional Operation (cf. (.2.3) for the output feature matrix Fut1. We will introduce the detailed implementations shortly.
3.2.1 Rotation-Steerable Kernel Construction
The key to satisfy the rotation-steerable constraint (4) is to control the angular directions of feature vectors, and a recent research shows that spherical harmonics Y {У give the unique and complete solution (25). Linear combination of the basis kernels based on spherical harmonics produces the intation-steerabe convolutional kernel K.
For simplicity, firstly consider both input and output features individual irreducible ones of
orders / and k, respectively; the kernel k R3 combination of basis kernels x Jm,
+ 1(2++1) is parameterized as linear
3 + IR 1)
A+1
M (x)
Σ
Σ
w Jm, g Jm
- | 1- !
where
M.
Σ TT ((<))))
| 14
(8)
Q is a (2k + 1) x (2l + 1) change-of-basis matrix, also known as Clebsch-Gordan coefficients,
and tmth a continuous Gaussian radial function: ([==() = e - =|===l In the basis kernel Jm y controls the angular direction, while 60th controls the radial one; then Jmm linearly combined by learnable coefficients (mmm as in Eq. (7) further adjust the radial
We only discuss convolutions (stride = 1) in our paper, since as pointed out in 25), convolutions (stride
1) damage the smoothness of features and break the properties of equivariance. For feature downsampling, we follow [25] and use combination of convolution (stride 1) with an average pooling.

---

direction, which is the only degree of freedom in the process of optimization. Accordingly, the angular direction is totally controlled by Y such that the inaatio--steerbbee constraint is strictly followed. In addition, the total number of learnable parameters in Eq. (7) is M [2minn(k: (k: () 11 (M the number of selected {m))), which is, practice, marginally less than that of conventional 31> convolution, which has (2k + 1) + parameters.
Finally, assuming that the input and output features stacked irredueible features, whose orders are (4,-... (F., and (Ki,-. ,KF,1) respectively, the rotation-steerable kernel of SS-Conv can be constructed follows:
khil (x)
belen (x))
x(x)=
(9)
k F (2²)
* lon (x))
with the size of K x Kn, where Kn+1 = Ei=t" 2kF,11 + and K2 -E- 21v,
3.2.2 Site State Definition
The key to enable the efficiency of SS-Conv lies in the definition of site state. In general, for an output grid site æ, any of input sites in its receptive field are active, this site will be activated, and convolution at this site will be conducted; otherwise, this site will keep inactive, meaning that its feature will be directly set zero vector (representing the ground state) without convolutional operation. We formulate the above definition of state at site æ as follows:
( (..
y H, and y S
Ou+1 (a))
(10)
others
The output hash table Hn+1 is then generated as fx =n+1(()) = 1}.
The number of active sites defined in (TO) will increase fnyer-by-layer enabling long-range message transfer. However, if dozens or even hundreds of convolutions are stacked, the rapid growth rate of active sites would result in heavy computational burden and the so-called "submanifold dilation problem" (9. To alleviate this problem, we follow (9] and consider another choice of state definition in our SS-Conv, which keeps the output state consistent with the input one at same grid site, i.e., on+1 (xr = o (æ), such that H Hn. This kind SS-Conv without dilation makes it possible to construct a deep but efficient network for sparse volumetric data, and we term it as "Submanifold SS- SS-Com""
In practice, we mix general SS- SS-Convs and Submanifold SS-Convs in an alternating manner to achieve high accuracy and efficiency.
3.2.3 Sparse Convolutional Operation
After obtaining Hn+1, the next target is to compute the values of Fa+1- Specifically, we firstly initialize Fa+1 zeros; then the feature vectors in Fa+1 updated via the following algorithm:
ALGORITHM I: Sparse Steerable Convolution
Input: (H (n). Fn).
Fa+1), (K(*): 8)
Output: (H Fat1)
1: R R, 8:885
// Initialize the rule book R. // Construct the rule book R.
2: for in H do
3: for in H., do
4:
if
S: Append (F (nyy) ) to R..
5:
11 M1r41 is the row number of xt in H
6:
end for 7: end for
11 /Tray is the row number of y in H...
8:
for R., in R do
11 Update FASr-
9: for (Fot12, Frag) ) in Rs do F Foti/rutial + Fat1 [o+11a] k(8) x Falragl
10:
11: end for
12:
end for
This process can divided into two substeps. The first one to construct a rule book = RR,
E S), where active output site is paired with active input in each Rg, if x y The
5

---

Teaser-to Posiet
f
1
Stage
$ poyd
SS-Cons
Potro--bbe Vooduree
(r.
0r.4g)
(rutt)
Feature Steering Module
Pesture-Sivering Modille
Feature Steering Modali:
Feature Steerinng Medale
Stage 1wo
22
Palnt-wia Fentures
Tensor do-P Poine
Figure 2: An illustration of network architecture for instance-level 6D object pose estimation.
second one update Fn according to the paired relationships recorded R: for example, if the paired relationship of output æ and input y is recorded in R., the current f=+1(x) will be updated by adding the multiplication of fo (y) and (s). this process, the construction of R is very critical, which helps to implement the second substep by matrix-matrix multiply-add operations on GPUs efficiently.
3.3 Normalization and Activation
As conventional CNNs do, SS-Convs are also followed by normalization and activation, i.e., Ac- tivation(Normn(Ie f=))))+++)))))))) (x))). Those operations of normalization and activation are required to be specially designed, not to break the SE(3) (e(iiv-auuva of features. Since each feature is formed by stacking irreducible ones, without loss of generality, take an example an irreducible feature ((x with order so that the normalization can be formulated follows:
Norm(f(x)) { (f(x)) / f(x)/ B[I (1) +
+ 1=
(1I)
1> 0
where B[|] and Var[-] are population
and variance, respectively. is very small constant. For the activation of f(x). if 0, ReLU can be chosen to increase non-lincarity; if > 0, we follow (25) and multiply to f(x) a scalar, which is learned by SS-Conv and applied the Sigmoid function:
ReLU(f(x)), (igmoid([:: f())))((x).
t= 0
Activation(f(a))
(12)
1>0
The above normalization and activation operations are both -Euuierqui since a feature vector multiplying any scalar keeps its equivariance; when applying them features formed by numerous irreducible ones, we treat each irreducible member individually to ensure the equivariance.
4 Applications for Estimation and Tracking of Object Poses in 3D Space
4.1 Instance-level 6D Object Pose Estimation
Given an RGB-D D image of a cluttered scene, instance-level 6D pose estimation is to estimate the 6D poses of known 3D objects with respect the camera coordinate system. As introduced Sec. 3.1 a 6D pose g E SE(3) can be decomposed into 3D rotation SO(3) and 3D translation t E g which makes sparse steerable convolutional network well suited for this task, due to: SS-Convs extract strong SE(()--quivvriant features to decode a precise 6D pose; ii) the steerability of feature maps helps to enable a second stage of pose refinement. Therefore, we propose an efficient general pipeline based on SS-Convs for 6D pose estimation, depicted in Fig.

---

Specifically, we firstly segment out the objects of interest via an off-the- model of instance segmentation, assigning each object with an RGB segment and a cropped point cloud: then each 3D object is voxelized and represented by a sparse tensor (Ho: Fo), where each feature in Fo is 4 4-dimensional vector, containing RGB values and constant "1". For the input tensor, we set the site active if the quantified grid centered at this site encloses any points, and average point features of those enclosed by a same grid. (Ho, Fo) is then fed into our pipeline in Fig. where the pose estimation could be achieved in the following two stages.
In the first stage, we construct an efficient SS JS-Conv-based backbone, which extracts hierarchical SE(3)--quivariant feature maps, represented in the form of sparse tensors { }. Those feature tensors are used for interpolation of multi-level point-wise features by using a Tensor-to-Poine module, proposed in [TO], transforming features of discretized grid sites to those of real-world point coordinates. Each point feature fed into two separate MIPss, regressing a point offset and a rotation, respectively; the addition of the point coordinate and its offset generates translation. The initially predicted pose (r1, f1) of this stage is obtained by averaging point-wise predictions.
In the second stage, we refine the pose (r1, t1) by learning a residual pose (r2, t2), wherein a
Feature-Steering module is designed, generating transformed features { {HH,, F2)] } by efficiently
steering hierarchical backbone features {(H,, } individually with (r1, (2). Again we interpolate
point-wise features from {HH,, Fx) }. and average point-wise predictions obtain (r2, ty). Finally,
the predicted 6D pose is updated as (2172, t1 trtz)) ). In addition, owing to the novel reature-Steering modules, this stage can be iteratively repeated, generating finer and finer poses.
4.1.1 The Feature-Steering Module
Feature- Steuuringg module in the pipeline is to transform (Hn, F..) of the backbone to (H', Fz), where rigid transformation of H,, with (r. r) and a rotation of F, with p(r) are included. Specifically, for Fn. we compute p(r) as defined in (5) and rotate F, by matrix multiplication; for Hn, we convert the sites in it to the real-world point coordinates, which are then applied of (r, t) and re- voxelized as grid sites. The same new sites are merged to a unique one, while their features averaged. We also and activation, to enrich the new features and generate the final steered HH,, F.).
rigid transformation
another SS-Convs, each followed by steerable normalization
4.2 Category-Ievel 6D Object Pose and Size Estimation
Category-Ievel 6D pose and size estimation is formally introduced in 24 This is a more challenging task, which aims to estimate categorical 6D poses of unknown objects, and also the 3D object sizes. To tackle this problem, we use a similar network as that in Fig. 2 and make some adaptive modifications: for each stage in Fig. 2 we add another two separate MLPs for point-wise predictions of 3D sizes and point coordinates in the canonical space, respectively; ii) in each Feature-Steering module, the real-world coordinates of all 3D objects are also scaled by their predicted 3D sizes to be enclosed within a unit cube. for estimating more precise poses.
4.3 Category-level 6D Object Pose Tracking
Motivated by the above task of categorical pose estimation, category-level 6D pose tracking is also proposed to estimate the small change of 6D poses in two adjacent RGB-D frames of an image sequence [22]. Due to the available pose of the previous frame, the target object can be roughly located in the current frame, avoiding the procedures of object detection or instance segmentation in images. However, without a precise mask, the estimation of small pose change from noisy 3D) data is a big challenge for deep networks. Our sparse stecrable convolutional network also surprisingly performs well in such noisy data, even though we only conduct one-stage pose estimation that achieves real-time tracking. For more details, one may refer to the supplementary material.
5 Experiments
Datasets We conduct experiments on the benchmark LineMOD dataset 11 for instance-level 6D pose estimation, which consists 13 different objects. For both category-level 6D pose estimation and tracking, we experiment on REAL275 dataset [24). which is a more challenging real-world
7

---

Table 1: Quantitative comparisons of Plain12 based on different convolutions on the LineMOD
dataset [11]
Conv
ADD(S) t
FPS
# Param
Dense-Conv
46.5
224
26.2 M
SP-Conv
62.8
486
26.2 M
ST-Conv
92.8
148
3.6 M
SS-Conv
93.5
404
3.6 M
0.06
Evaluation Loss
0.05
Derce
Cony
0.25
0.05
Training
0.04
0.62
0.1
0.01
0.00
0.0
0.4
1.2
1.6
2.4
0.00
0.0
0.8
1.2
2.0
Iteration|x
10f)
Iteration(x 10")
(a)
(b)
Figure 3: Training of Plain12 based on different convolutions on the LineMOD dataset
dataset with 4, 300 training images and 2, 750 testing ones, containing object instances of 6 categories.
Following 24 22 we augment the training data of REAL275 with synthetic RGB-D images.
Evaluation Metries For instance-level task, we follow (23) and evaluate the results of LineMOD dataset on ADD(S) metrie. For the category-level tasks, we report the mean Average Precision (mAP) of intersection over union (IoU) and n'm cm, following [24]; mean rotation error (Terr) in degrees and mean translation eтrоr (terr) in centimeters are also reported for pose tracking. Additionally, we compare the numbers of parameters (#Para)) and the running speeds (FPS) for different models. Testing conducted on a server with GeForce RTX 2080ti GPU for batch size of 32, and FPS is computed by averaging the time cost of forward propagation on the whole dataset.
5.1 Comparisons with Different 3D Convolutions
We firstly conduct experiments to compare proposed SS-Conv with other kinds of 3D convolutions, including conventional 3D convolution (Dense-Conv) sparse convolution (SP-Conv) (9, and steerable convolution (ST-Conv) (25). on the LineMOD dataset for instance-leve 6D pose estimation. Among those convolutions, SP-Conv improves the speed of Dense-Comy by considering data sparsity and turns out be efficient in some tasks of 3D semantic analysis(e.g.. 3D object detection), while ST-Conv constructs rotation-steerable kernels and then realizes the convolution based on Dense-Conv.
To meet various computational demands of different convolutions, those experiments are conducted on a light plain architecture, termed as Plain12, in the same experimental settings, for fair comparison. The architecture consists of 12 convolutional layers, of which the kernel sizes are all set as 3 x 3 x 3; for SS-Convs and ST-Convs, we set the superparameters of radial function upth Eq. (8) as {m} = {0, 1} and 0.6. We use ADAM to train the networks for total of 30, 000 iterations, with an initial learning rate of 0.01, which halved every 1, 500 iterations. We voxelize the input segmented objects into 64 x 64 x 64 dense/sparse grids, and set the training batch size 16.
Quantitative results of different convolutions are listed in Table which confirms the advantages of our SS-Conv in both accuracy and efficiency. In terms of accuracy. SS-Conv achieves comparable results on ADD(S) metric as ST-Conv does, which significantly outperforms those of Dense-Cony and SP-Conv, indicating the importance of preservation of relative poses of features layer-by-layer, the property of SE(3)-equivariance makes feature learning capture more information of object poses. We also visualize the behaviors of the four convolutions in the process of training in Fig. 3 where the learning based on SS-Conv/ST-Conw converges better and faster than that of Saans--Conv/S-CCCC
feature learning on pose estimation

---

Plain12
Plain24
ResNet50
Bottch sines 512
PPS:2165 2063
Ratck sient 512
Bitth
387.6
120
Batch
119% SUAL
Rutch
100
145.4
Botch sime 515 -4.7
20 28 14r 26 2011
Usatch staz: Mensary: 18:G
Momary: 18A6
Butch seec Menuury 9.70
Turch slre $11 Mennoyy RUDG
Memory(C)
Memory/a)
Memoryy
Butch size 512 Memoryy 334G
Butcc
Menory
Batch das 15
Mensoyy: LG2
UB 52 3024
Batch size
Batch size
Batch size
Figure 4: Plottings of FPS and memory consumption versus different batch sizes for different networks based ST-Conv/SS- Experiments conducted on LineMOD dataset
Table 2: Quantitative comparisons of different methods the LineMOD dataset for instance- level 6D object pose estimation. The evaluation metric is ADD(S).
Implicit +ICP
[13 +ICP
PointFusion
DenseFusion
DenseFusion (Iteaative) 25
G2L
Ours w/o second stage
Ours
27
23
ape
20.6
65
70.4
79.5
92.3
96.8
92.9
97.4 4
bench.
64.3
80
80.7
84.2
93.2
96.1
97.4
99.3
camera
63.2
78
60.8
76.5
94.4
98.2
97.7
99.5
can
86.6
93.1
98.0
96.1
99.6
cal
72.0
88.8
96.5
98.8
99.8
driller
41.6
73
47.3
77.7
87.0
99.8
98.7
99.6
duck
32.4
66
63.0
76.3
92.3
97.7
91.1
97.8
98.6
100
99,9
99.9
99.8
100.0
100.0
99.9
glue
96.4
100
99.3
99.4
100.0
100.0
98.6
99.6
hole.
49.9
49
71.8
79.0
92.1
99.0
96.3
99.4
iron
83.2
97.0
98.7
99.2
lamp
91.7
73
62.3
92.3
95.3
99.5
99.5
99.7
phone 71.0
79
78.8
88.0
92.8
98.9
97.5
98.2
MEAN
64.7
79
73.7
86.2
94.3
98.7
97.2
99.2
In of efficiency, our sparse steerable convolutional networks are more efficient and flexible for complex systems, e.g., for Plain 12, SS- SS-Co brings about 2.7x speedup w.r.t. ST-Conv (404 FPS versus 148 FPS) with batch size of 32, as listed Table I More results of FPS with improved sizes of data batches are given in Fig. 4 where ST-Conv can be only run at the extreme batch size of on the GPU with 12G memory. while SS-Conv batch size as large as 512; running with larger batch sizes further improves the efficiency of our proposed SS-Conv (FPS goes to 725 when running with the batch size of 512 on Plain 12). We also compare the efficiency of ST-Conv and SS-Con S-CCov two other deeper networks (dubbed Plain24 and ResNet50, respectively): as shown in Fig. 4 SS-Conv consistently improves FPS over ST-Conv on these two architectures, with less GPU memory consumption.
much less memory and can thus support
5.2 Comparisons with Existing Methods
Instance-level 6D Object Pose Estimation For the instance-level task, we compare the results of our SS-Conv-based pipeline with existing methods on LineMOD dataset [Л]. Quantitative results

---

Table 3: Quantitative comparisons of different methods on REAL275 dataset 24 for category-level 6D object pose and size estimation.
mAP
Method
IoU
IoU7s
5
5cm
10º2cm
10m5
NOCS
24
78.0
30.1
7.2
10.0
13.8
25.2
SPD
21
77.3
53.2
19.3
21.4
43.2
54.1
CASS
77.7
23.5
58.0
FS-Net
92.2
63.5
28.2
60.8
DualPoseNet
[5
79.8
62.2
29.3
35.9
50.0
66.8
Ours w/o second stage Ours
79.5
58.7
19.2
25.2
35.1
49.9
79.8
65.6
36.6
43.4
52.6
63.5
Table 4: Quantitative comparisons of different methods on REAL275 dataset 24 for category-level 6D object pose tracking.
Method
Metric
bottle
bow
camera
can
laptop
mug
MEAN
5 5cm 1oU2s* t
24.5
55.0
10.1
22.6
63.5
24.1
33.3
91.1
100.0
87.6
92.6
98.1
95.2
94.1
6-PACK 22
Terr +
15.6
5.2
35.7
13.9
4.7
21.3
16.1
terr +
4.0
1.7
5.6
4.8
2.5
2.3
3.5
5 5cmm
70.3
60.6
10.6
49.9
87.7
47.9
54.5
IoU2s f
93.5
99.9
99.9
99.8
99.8
99.9
98.8
Ours
Terr +
3.7
4.6
9.8
4.6
3.0
5.6
5.2
terr 1.9
1.2
2.0
2.7
2.4
1.1
1.9
shown in Table where two-stage pipeline outperforms all the existing methods and achieves a new state-of-the-arr result of 99.2% on mean ADD(S) metric. We can also observe that the second stage of pose refinement with Feature-Steering modules in pipeline indeed improves the predictions in the first stage, benefitting from the steerability the feature spaces SS-Convs.
lete 6D Object Pose and Size Estimation We conduct experiments on REAL275 [24] for the more challenging category- task. Quantitative results in Tableecoonfin the advantage of our pipeline in the high-precision regime, especially on the precise metric of 5 where improve the state-of-the-art result in [15] from 35.9% to 43.4%. The second stage of pose refinement also plays important role in this task, achieving remarkable improvements over the first stage.
Category-level 6D Object Pose Tracking We compare the results of our one-stage tracking pipeline with the baseline of 6-PACK (22) on REAL275 244. In 6-PACK, the relative pose between two frames computed based on predicted keypoint pairs inefficiently, while our pipeline regresses the pose in direct way. The results in Table 4 show that our pipeline outperforms 6-PACK on all the evaluation metrics, demonstrating the ability of Conv--based network for fine- aneeggained pose estimation in noisy input data.
More implementation details and qualitative results shown in the supplementary material.
Broader Impact
The studied problems of object pose estimation and tracking in 3D space are very important many real-world applications, including augmented reality, robotic grasping, and autonomous driving. By precisely predicting object poses
3D space, virtual contents could be seamlessly embedded in real environments. creating fascinating personal experience; the contrary, less precise predictions may cause property loss and even life threat, especially autonomous driving. The contributed solution based on SS-Cony would improve the overall level of safety.
10

---

Acknowledgments and Funding Disclosure
This work was partially supported by the Guangdong R&D key project of China (No.: 20198010155001,, the National Natural Science Foundation of China (No.; 61771201), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No.: 20172T07X183).
References
[1] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In European conference on computer vision, pages 336-551. Springer. 2014.
[2] Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11973-1 11982, 2020.
[3] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, and Ales Leonardis. G2l-net: global to local network for real-time 6d pose estimation with embedding vector features. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4233-4242, 2020.
[4] Wei Chen, Xi Jia, Hyung Jin Chang, Jimming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. arXiv preprint vxXiv2203.00054. 2021.
[5] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In International Conference on Learning Representations, 2018
[6] Carlos Esteves, Christine Allen-Blanchee Blanchette, Ameesh Makadia, and Kostas Daniilidss Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68, 2018
[7] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3>-transformers: 3d roto-tramslation cquivariant attention networks. Advances in Neural Information Processing Systems, 33, 2020.
[8] Robert Gilmore. Lie groups. physics, and geometry: introduction for physicists, engineers and chemists. Cambridge University Press, 2008.
[9] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and puttern recugnition, pages 9224-9232, 2018.
[10] Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11873 11882, 2020.
[11] Stefan Hinterstoisser, Stefan Holzer, Cedric Cagniart, Slobodan Ilie, Kurt Konolige, Nassir Navab, and Vincent I Lepetit. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In / international conference on computer vision, pages 858-865. IEEE, 2011.
[12] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilie, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference computer vision, pages 548-562. Springer, 2012.
[13] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilie, and Nassir Navab. Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again. Proceedings of the IEEE international conference on computer vision, pages 1521-1529, 2017.
[14] Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Learning snalysis-by-yytteeiis for 6d pose estimation in rgb-d images. In Proceedings of the IEEE international conference computer vision, pages 954-962, 2015.
[15] Jiehong Lin, Zewei Wei, Zhihao Li. Songeen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-Ieve 6d object pose and size estimation using dual pose network with retined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3560-3569 October 2021.
11

---

[16] Frank Michel, Alexander Kirillov, Eric Brachmann. Alexander Krull, Stefan Gumhold, Bogdan Savchyn- skyy, and Carsten Rother. Global hypothesis generation for 6d object pose estimation. In Proceedings the IEEE Conference on Computer Vision and Puttern Recoognition, pages 462-471, 2017.
[17] Sida Peng. Yuan Liu, Qixing Huang. Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network 6dof pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561-4570, 2019.
[18] Martin Sundermeyer, Zoltan- Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 1999-115,7 2018.
[19] Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6d object pose prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pages 292-301, 2018.
[20] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-cquuiariann neural networks for 3d point clouds. arXiv preprint arXiv: /8Xx::8002.080 08219, 2018.
[21] Meng Tian, Marcelo H Ang. and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In European Conference on Computer Vision, pages 530 546. Springer, 2020.
[22] Chen Wang, Roberto Martin,Martii, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category- -eeeel 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066) IEEE, 2020.
[23] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin. Cewu Lu, Fei-Fei, and Silvio Savarese. Densefusion: ohject pose estimation by iterative dense fusion. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3343-3352, 2019.
[24] He Wang. Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2642-2651, 2019.
[25] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, arXiv preprint arXiv: 1007005447 2018.
[26] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv: 171 1,00 2017.
[27] Danfci Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 244-253, 2018
12


Reviews:
[{"review_id": "z8qOB0OUenM", "reviewer": "Reviewer_BHAN", "summary": "This paper extends steerable convolution by taking advantage of the sparsity of 3D data, such that the speed in both training and inference is significantly faster than the original one [24]. The method itself is motivated by SP-Conv [9], e.g. the site state definition and “submanifold dilation”. In evaluation, it is demonstrated that the method outperforms or is comparable to the state-of-the-arts in 3D object pose estimation, tracking and size estimation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, as it provides a gentle review of the background works which this one is built upon, and most of its discussion about related works and motivations is adequate and precise.  I’m also fully convinced by the practical impact of this work -- the noticeable speed up of steerable convolution is certainly something the community desires. \n\nHowever, as a reviewer, I do concern the “novelty” of this work, as it reads to me, is \"just\" applying the engineering details proposed by  [9], e.g. “submanifold conv” to steerable conv [24].  I fail to see the technical novelty which is actually owned by this paper. I’d like to hear from other reviewers regarding this, and the author is of course welcomed to point out their contribution if I unfortunately missed.\n\nOther minors regarding to quantitative comparison: (1) It would be nice if the authors also compare the running speed in table 2-4, or at least comment on their relative speeds. This potentially could make the readers appreciate more about the practical impact of this work.\n\n(2) In table3, the proposed method leads in almost every metric, except IoU50, which shows a big gap when compared to  FS-Net[4]’s 92.2% accuracy. It seems that the proposed method has more catastrophic failures. Could the author show / comment on the type / reason of failures. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "pXkHhNxHHl", "reviewer": "Reviewer_hf1K", "summary": "This paper extends the concepts of sparse manifold convolution to the steerable convolution. In particular, a sparse steerable convolution operation is proposed to reduce the redundant computation in volumetric data, while maintaining the SE(3) equivariance. Extensive experiments conducted on three tasks (i.e., instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking) demonstrated the efficiency and effectiveness of the proposed SS-Conv.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the idea of this paper is straightforward and feasible. It is desirable to extend the sparse convolution to steerable convolution, especially in light of the expensive computational cost.  The reported experimental results also demonstrate better efficiency of the proposed SSConv operation.\n\nHowever, the real technical contribution of this paper is somewhat limited, since the sparse convolution has been well-developed and achieving remarkable performance in other fields such as semantic segmentation. In addition, several well-optimized libraries such as SparseConvNet and Minkowski engine further simplified the implementation. \n\nExperiments:\n\n-Please increase the readability of Figure 3, the fonts are too small to be recognized.\n\n-Learning generalized local descriptors from 3D point clouds also requires rotation and translation invariance. I am wondering if the proposed SSConv can be integrated into the FCGF [1] framework to achieve better registration performance.\n\n[1] Fully convolutional geometric features. In ICCV, 2019.\n\n-Please also report the FLOPS of different implementations in Table 1.\n\n-In Table 2, the performance of the proposed approach is inferior to G2L [3] when the second stage is not used, what is the possible reason? what is the impact of the second stage on the proposed framework? What is the extra computational cost introduced by the second stage?\n\n\nMinor comments:\n\nLine 82, the reference is missing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "RBPt2ptOg9t", "reviewer": "Reviewer_viJM", "summary": "This paper proposes sparse steerable convolution that efficiently learns SE(3)-equivariant features and demonstrates its usefulness in object pose estimation tasks. Essentially, the proposed method is a combination of 3D steerable cnn [24] and submanifold sparse convolutional networks [9]. The proposed method, being able to extract SE(3)-equivariant features, is proven to be useful in 6D pose estimation, outperforming previous state-of-the-art methods", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work to my best understanding is a nice engineering effort to bring 3D steerable convolution to sparse voxels. The experiments show that the property of SE(3)-equivariance extracts better information of object poses while being more efficient than its dense counterpart.\n\nHowever, I am concerned about the novelty of this work. To my best understanding, almost all of the theoretical basis of 3D steerable convolution came from its previous work [24] and submanifold sparse convolutional networks [9], and this work mostly reiterates previous literature. I am not sure what makes this novel, especially considering the venue of the conference. Although I agree that in engineering perspective, the proposed sparse steerable convolutions is efficient and reasonable, considering the venue of the conference, I am not sure if this work satisfies the novelty claim that it makes. If there is any unique problem being sparse, I believe it should better be highlighted. The comparison in the supplementary material only discusses engineering perspectives such as speed or memory consumption, which is the novelty of sparse convolution, not this work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be helpful if the authors highlight failure cases of this work compared to a regular sparse convolution neural networks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3hrs"}, "extra_text": {}}, {"review_id": "P84Bf_qIzPv", "reviewer": "Reviewer_aue9", "summary": "This paper builds upon Weiler's [24] steerable filters on volumetric data\nand creates a new equivariance by design by sparsifying group convolutions.\nAuthors are inspired by Graham's [9] sparsification of convolution that takes into account the state of a voxel (active /inactive) without sacrificing the steerability. \nThe kernel construction follows the spherical harmonics weighted by the Clebsch-Gordan coefficients and radially weighted with a Gaussian.\nThe paper contains an extensive experimentation, comparing several 3D convolutions, and applying the method on 6dof instance pose, tracking and simultaneous class pose and size.\n\n\n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper develops a new efficient scheme for steerable convolution on volumetric data. Volumetric convolutions are very expensive and any way to accelerate them has tremendous practical impact.\n\nThe paper takes the two main ideas of 3D steerability [24] and sparsified convolutions [9] to design a tensor-based sparsified convolution that is steerable in SO(3). The authors provide all details of the novel design. The reader needs some background reading on steerability and to understand the general steerability constraint (4) that is applicable for any dimension of the input and the output (the formula is much simpler for scalar fields). \n\nThe active sites (voxels) are stored in a hash table $H$ updated by the active/inactive state. This is a very elegant design indeed and $H$ together with the feature vector $F$ constitute the vector that is updated from layer to layer.\n\nThe paper is one of the few, might be next to Esteves ICML 2019 (equivariant embeddings that are used for relative SO(3) computations)  that contains extensive experimentation on non-classification tasks (no MNIST, Cifar10 etc) but standard 6dof object pose benchmarks like LineMOD and REAL275. \n\nThese applications alone could be a paper in a mainstream vision conference.\n\nFor instance-based 6dof pose the authors segment the volume creating the initial hash table and start with the vector (R,G,B,1) as feature input. After several layers, features are converted into point coordinates that are regressed to compute the pose. \nAt this point, the reader has really to go to the appendix to understand the loss functions. While ground-truth poses are used the loss functions are ICP-like. Please move the losses to the main text. The Appendix is highly informative and like the paper itself a delight to read.\n\nThe method outperforms all methods on pose and size and comparable or better to DenseFusion of G2L on instance-based 6dof pose estimation.\n\nPOSTREBUTTAL: I read all reviews and the authors' responses and I stay with my positive mark: The combination of steerability and sparisty is not a simple integration, and the experiments on pose are novel in the context of steerability. The reader can learn a lot by reading the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations explained in Conclusion.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "M4hhOiX298W", "reviewer": "Reviewer_zbbB", "summary": "This work proposes a reformulation of Steerable Convolutions (ST-Conv) , dubbed Sparse Steerable Convolutions (SS-Conv),  that allows them to run on sparse voxel grid while still preserving SO3 equivariance.\n\nSpecifically, accelerations is achieved in two ways: 1) convolutions are run only on activated voxels, 2) in the receptive field of activated sites, only active input features are convolved.\nTo further increase efficiency and alleviate the \"dilation\" problem of sparse convolutions, authors also propose a modification to SS-Convs, dubbed Submanifold SS-Convs, where output voxel state is kept consistent with the input one, making it possible to construct a deep but efficient network. In practice Manifold SS-Convs are alternated to SS-Convs to guarantee high performance and efficeincy.\n\nThe proposed method is first compared to ST-Convs, Dense Convs, and Sparse convs on the LineMOD dataset, where it demonstrates similar performance to ST-Convs while being 2x faster. Furthermore, it is validated through the following 3D semantical analysis tasks: instance-level 6D pose estimation, category-level 6D pose and size estimation, and category level 6D pose tracking. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Sparse implementation of ST-Convs could be beneficial for community working with sparse 3D data, as allows for efficiency that is on par with standard 3D convolutions\n- Proposed application-specific architectures exploit Steerable convolutions to achieve state-of-the-art performance and demonstrate the effectiveness on steerable convolutions on tasks that had not been tackled by this architecture before\n\nWeaknesses:\n- Novelty: a more efficient implementation of an existing method is not enough novelty for NeurIPS, especially given that for the presented experiments SS-Convs only result in a 2x speedup wrt ST-Convs, and authors have to actually use a single-stage architecture to reach real-time performance on 6D object pose tracking. \n\nQuestions for authors:\n- In Table 1, SS-Convs outperforms ST-Convs. How do you justify this, given that SS-Convs is a sparse implementation of ST-Convs? Are there any other differences between the two considered architectures, or is it due to the use of Submanifold SS-Convs whitin the architecture?  If so, it should be specified.\n\n- Are Submanifold convolutions. novel? From the paper, it is hard to say if the if the idea of using Submanifold SS-Convs is novel, or if it has already been propose for sparse 3D convolutions and adapted to this framework. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2uLHu2EP1qN", "reviewer": "Reviewer_hnKB", "summary": "This paper discusses the development of a novel method called Sparse Steerable Convolution for the estimation of an object’s 3D pose. As the pose estimation is usually done using SE(3)-equivariant deep feature learning, the proposed method provides more efficient computation using sparse tensors. The paper presents carefully the theoretical background of the method and experiments it via using two different benchmark image datasets for estimating the 6D object pose.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, scientifically sound and provides enough details about the method. The method is novel and provides means for efficient and still accurate object pose estimation. The paper gives credit for previous work. Novelty and contribution are clearly stated. The theoretical premises are carefully explained. The method is tested and analyzed using two very different benchmark image datasets and the results are compared to multiple state-of-the-art object pose estimation methods. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, two of the reviewers expressed concerns about the paper and ranked the paper below acceptance. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper (highest rate was 9!)  and recommended acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version.
Decision: Accept (Poster)