Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space

Abstract: As a basic component of SE(3)-equivariant deep feature learning, steerable convolution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3)-equivariance. Based on SS-Conv, we propose a general pipeline for precise estimation of object poses, wherein a key design is a Feature-Steering module that  takes the full advantage of SE(3)-equivariance and is able to conduct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Ablation studies also show the superiority of our SS-Conv over alternative convolutions in terms of both accuracy and efficiency. Our code is released publicly at https://github.com/Gorilla-Lab-SCUT/SS-Conv.


Reviews and author responses:

 Hi,

Thank you for the rebuttal. Mostly the authors address my comments well.

Only one point concerning c) Extension of SS-Conv to more general cases.

The PN_VLAD [2] is a weak baseline, and it would be better to use a stronger backbone as per [https://github.com/jac99/MinkLoc3D]

I would like to maintain the original rating.

Best,
Reviewer dFJh


 Hi,

Thanks. The authors' response addressed the reviewer's concern (I am not an expert on this topic). I'd like to maintain the original rating as the Sparse Steerable Convolution is still a valuable contribution. However, the authors are suggested modifying the paper according to all other comments.

Best,

Reviewer gTpb

This paper proposes a sparse SE(3)-equivariant convolution for voxel-based deep networks.  To avoid high computational costs of dense voxel processing, only active voxels are processed.  Every other layer increases the number of active voxels through dilation to account for the filters' range of influence.  The proposed sparse convolution is embedded into a network that first performs an approximative matching, then a refinement, and is applied to 6 DoF object pose estimation, 6 DoF category pose estimation, and 3D object tracking.  All networks are evaluated on standard datasets, showing very good results and improved computational performance.
  # Overview

The paper introduces a well thought-off and generic method can can be used as building block for many 3D applications. The method is well described, practical and shows very good results compared to state of the art, including an impressive 99.2% detection rate on LineMOD. The theoretical contribution is overall incremental over [24] and [9]. The exact contribution of the introduced method are not quite clear from the experiments.


# Details

* The evaluation is reasonably convincing regarding the overall performance of the proposed networks. It is performed on large, standard datasets, and the proposed method is compared to state of the art.  The results are very good, especially the refinement part ("Stage Two") of the network appears to have a significant effect on improving the predictions.

* The writing is good, the method is well motivated and explained and can likely be reproduced based on the paper.

* The introduced convolutions heavily relies on [24], especially in Sec. 3.1, 3.2.1, 3.3, and [9] in 3.2.2 (though, arguably, [9] has a more interesting approach to the sparsity problem). Overall, it appears to me that the SE(3)-equivariance is borrowed from [24], and the main contribution for the convolutions is the usage of a sparse instead of a dense voxel grid, coupled with dilating the region of active voxels over time.

  Regarding the overall network design, the main contribution is using a two-stage approach where the second network further refines the output of the first. This approach leads to significantly more accurate results, as is visible in the experiments.
  
* While the impact of the introduced sparse voxel grid is analyzed in ablation studies, those are not quite clear to me and several open questions are below.


# Open Questions

From most to least important.

* Table 1: Are the FPS computed for complete images, or for single patches (found by the preceeding object detection / instance segmentation network)? Overall, giving concrete timings for inference would be a plus (i.e. how much runtime does the network use for processing one LineMOD image? How much runtime does the instance segmentation network take?). Does the FPS include the instance segmentation, or is it only the time spent in the proposed network, processing the detected instances? I am especially curious about this because while sparse voxels often have a theoretical advantage, that is often mitigated in practice because of the required out-of-order memory access (here lines 2--7 in Algorithm 1), which is not well suited for modern GPU architectures.

  Also in Table 1, are the ADD(S) and FPS for ST-Conv also measured using the proposed two-stage network?

* Sec. 3.1 / Figure 2: I am unsure what the input to the MLPs is, since the number of active sites can change, but MLPs need a constant number of inputs. Maybe the effect of the Tensor-to-Point-Block should be described in 1-2 sentences.

* Sec. 3.3: What is the motivation for using a different normalization and activation for l=0?

* Why is there a difference in the ADD(S)-metric of the proposed method between Table 1 and Table 2 (SS-Conv vs. Ours)?

* Supplementary Material: When using a "ResNet50" like backbone (Appendix A), how are skip connections implemented if the dimensions of the output of a layer are larger than that of the input (as done according to Eq.(10) of main paper)?

* l.260--261: What does "feature channels are kept consistent" mean exactly?

* l.193: Which instance segmentation network was used, exactly? I believe this should be mentioned in the paper.


# Typos etc., no need to respond:

* l.267: I believe "marginally" is the wrong word here, since the gap is quite large. Similar in l.138.

* For consistency, think about making the best values bold in Tables 1 and 4.


# Potentially interesting citiations

The following citiations might be interesting for the paper, though there is no requirement to include them:

* Zhao, Yongheng, et al. "Quaternion equivariant capsule networks for 3d point clouds." European Conference on Computer Vision. Springer, Cham, 2020.
    
* Shen, Wen, et al. "3d-rotation-equivariant quaternion neural networks." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. Springer International Publishing, 2020.
 Societal impact was not discussed and was not expected by the reviewer for topic at hand. Limitations of the method were not discussed.

 I'd like to thank the authors for the clarifications and explanations and will raise the rating to a 6. Overall, I find this to be a paper of value that introduces a method with a good theoretical foundation and of highly practical value. Similar to reviewer zYbA, I still have some reservations regarding the novelty of the method over [24] and [9], which is why it is not a 7 or higher.

The paper proposed to equip the 3D steerable convolution with the sparsity handling capability for more effective use of the computing resources. In particular, only the activated grid locations among the feature map (sigma_{n+1}(x)) are computed for feature values. To achieve this goal, a rule book R is constructed before conducting the sparse convolution propagation. This results in an iterative forwarding procedure among the network forwarding flows. The authors cultivated several useful applications of the proposed framework and conducted several experiments to indicate that the proposed approach seems to be able to outperform existing works.  This paper aims to handle sparsity for steerable convolution for better practical use. I believe the motivation is good and the problem addressed in this paper provides a nice example to better deploying theoretical contribution.

My concerns toward the weaknesses of the paper are as follows and I wish the authors address them in the rebuttal.

This paper seems not particularly exciting with respect to its novelty, and particularly, to me this paper seems to be a quick combination of [24] and [9]. My major concern is whether using the techniques of the spherical harmonics in Sec. 3.2.1 are particularly important under the sparsity settings. If so, I hope the authors can provide a clear explanation in both the rebuttal and the final version, on why this part (spherical harmonics in Sec. 3.2.1) is essential to the sparsity setting, e.g. without applying spherical harmonics, why existing art cannot be well suited for the sparsity setting. I understand that the spherical harmonics satisfy the rotation steering constraints, which is an important property to be used, but its connection to the sparse convolution has not been thoroughly discussed. Quick issues for Sec. 3.2.1: m seems not defined in L137 (Is it an iterator or the number of bases in a row of the spherical harmonics?). It would also be helpful to state that each element in the matrix (Eq 9) is in the size of s by s by s.

There are also concerns from the experiments. First, I am concerned that the speed acceleration seems not particularly exciting (one time faster than the non-sparse counterpart). Second, I did not find comparisons to the non-sparse counterpart, e.g. [24]. It is OK to be slightly worse than the non-sparse counterpart, but it would be essential to see its trade-off between the performance drop and the speed acceleration. Third, I appreciate that the authors cultivated several important applications as a result of the proposed framework, but I also wish to see its comparison on the original task (e.g. 3D object classification or recognition) to the existing art.

Please address my concerns in the rebuttal. The authors have addressed the limitations and potential social impact in the paper.

 I appreciate the authors' efforts to address my concerns. I am willing to raise the score to "6", as the authors addressed most of my concerns (details below). Though, I still believe the paper seems to be a combination of [9] and [24], and Sec. 3 (before Sec 3.1) + Sec  3.2.1 needs proofreading (details below), but overall I believe this paper would be fit for the conference proceedings.

I now get that using the spherical harmonics is because it is regarded as one viable way [24] to resolve the constraint in Eq. (4) for fitting into the sparse setting. I am not an expert on theoretical analysis, and I guess maybe adding one paragraph between Sec.3 and Sec. 3.1, illustrating the overall logic flow between different parts of Sec. 3 (i.e. different components of the approach and why they are motivated), might help me get the message faster. It would be an elevator overview of the approach, e.g. what difficulties would arise when getting into the sparse setting (refer to Sec. 3.1), and why finding the basis kernel that satisfies the rotation-steering is essential (Sec. 3.2), and then how SH satisfies that (Sec. 3.2.1) and then this component is fused into the system (later sections). The current version would leave me confused on how the modules are connected and why each part is necessary. In addition, I wish Sec.3.2.1 can get proofreading. The current version seems mainly for the expert readers in this field. 

Experimental side, I believe adding the new results into the supplementary would help. I still believe the SS-conv vs ST-conv comparison seems insufficient to some extent (I wish more experimental data points on more tasks). For the response above, is there any reason why the ST-conv results are not included? (Is it because it is not realistic to train with the non-sparse counterpart?)

 
We highly appreciate the reviewer’s comments and thank the reviewer for the suggestion on more experimental comparisons between Steerable convolution (ST-Conv) and our proposed Sparse Steerable Convolution (SS-Conv). In the submitted paper, we compare ST-Conv and SS-Conv in Sec. 5.1 in terms of accuracy and efficiency on LineMOD dataset. Training times with the same iterations are shown in Fig 3(b), where SS-Conv takes about 2 hours while ST-Conv takes about 6 hours; during inference, SS-Conv also runs much faster than ST-Conv, as shown by the improvement on FPS metric in Table 1. 

In the supplementary material, we further explain that the efficiency of SS-Conv improves with the increased sizes of data batches during both training and inference (cf. Fig. 2 in the supplementary material); for example, for a plain backbone consisting of standard Conv layers (dubbed Plain24 in Fig. 2), SS-Conv brings about 16x speedup w.r.t. ST-Conv (111.8 FPS versus 7.1 FPS) at the batch size of 13; more importantly, SS-Conv costs much less memory and can thus support a batch size as large as 512, while ST-Conv can be only run at the extreme batch size of 13 on the GPU with 12G memory; running with larger batch sizes further improves the efficiency of our proposed SS-Conv (as shown in Fig. 2, FPS goes to 206.5 when running with the batch size of 512).

We also thank for the constructive suggestion on showing statistics of the data sparsity. To illustrate the sparsity, we quantify the CAD models of the 13 objects in LineMOD dataset into $64\times64\times64$ voxels, each of which covers an area of $0.96m\times0.96m\times0.96m$ as done in our experiments, and count the number of occupied grids for each voxel. As shown in the table below, the average occupied ratio is less than 1%; in the real-world application, the number would further decrease because the observed input is partial and even heavily occluded. 


| Object      | Num. of Occupied Grids | Occupied Ratio |
| :----:      | :--------------------: | :------------: |
| ape         |        1022             |      0.39%     |
| bench       |        3744             |      1.44%     |
| camera      |        2726             |      1.04%     |
| can         |        3119             |      1.19%     |
| cat         |        1665             |      0.62%     |
| driller     |        3355             |      1.28%     |
| duck        |        1258             |      0.48%     |
| egg         |        2464             |      0.94%     |
| glue        |        1258            |       0.48%    |
| hole        |        2359            |       0.90%    |
| iron        |        3984            |      1.52%    |
| lamp        |        3774            |      1.44%    |
| phone       |        2778             |     1.06%     |
|  __mean__   |        2577            |      0.98%     |

For a fair comparison, the network architecture and training strategies of Dense-Conv are the same as those of SS-Conv. In the submitted supplementary material, we give an illustration of the architecture in Fig. 1, and the specifics of the backbone (Plain12) are also given in Fig. 3 (by replacing SS-Conv with Dense-Conv). The inputs of the network based on Dense-Conv are dense voxels, represented as 64x64x64xC tensors, where C denotes the number of feature channels. More experimental details are shown in L259-L264 in the submitted paper and L2-L10 in the supplementary material.




 
We thank the reviewer for agreeing with the usefulness of our proposed Sparse Steerable Convolution (SS-Conv), and address the reviewer’s concerns as follows.

## Concerns on the novelty of SS-Conv.

We first address the reviewer’s major concern on the importance of the combination of SParse Convolution (SP-Conv) and rotation-steering kernels on pose-related semantic tasks.

SP-Convs have been proven to be very efficient on various 3D tasks, e.g. 3D object detection [1] and 3D instance segmentation [2]; however, for our focused tasks, estimation and tracking of object poses, they perform poorly (e.g., 62.8% on ADD(S) metric for LineMOD dataset cf. Table 1), due to the difficulty of learning in SE(3) space. With the rotation-steering kernels, our SS-Conv is capable to learn SE(3)-equivariant features, which could maintain the information of object pose along with hierarchical feature learning and make it easier to decode the object pose out; this is also verified in Table 1, where our SS-Conv outperforms SP-Conv by a large margin (93.5% versus 62.8%) on ADD(S), demonstrating the importance of SE(3)-equivariant feature learning on pose estimation.

Another contribution of our paper is to propose a general SS-Conv based pipeline (cf. Fig. 2) for pose estimation, and based on the steerability of the learned features, a novel Feature-Steering module wherein is designed to enable the second stage of iterative pose refinement. The second stage of refinement not only gives us finer predictions through learning residual poses but also provides auxiliary supervision on the SE(3)-equivariant feature of backbone. We emphasize that the base of such a two-stage pipeline is exactly the steerability of features; however, for the original SP-Conv based network, Feature-Steering module loses its function, and the pose refinement naturally breaks down.

We note again that SE(3)-equivariant deep feature learning is essential for pose-related 3D semantic tasks; it is yet less explored until very recently, possibly due to the less convenient usage of existing techniques (e.g., the huge time- and memory-costs for steerable convolutions). We expect that SS-Conv proposed in the present paper would improve the convenience of learning SE(3)-equivariant features for pose-related 3D semantic tasks and consequently would inspire new algorithms that perform even better on these tasks. 

For the quick issues, ‘$m$’ in Eq. (8) is a scalar superparameter, denoting the mean of the Gaussian radial function. In our experiments, we set ‘$m$’ in Eq. (7) as a series of integers ${0, 1, …, int(s/2)}$, where $s \times s \times s$ is the spatial size of kernel $\kappa$. Besides, in Eq. (9), $\kappa(x)$ is an element of the whole kernel $\kappa$ at location $x$; the size of $\kappa(x)$ is $K_{n+1} \times K_{n}$, while that of $\kappa$ is $K_{n+1} \times K_{n} \times s \times s \times s$. We thank the reviewer for pointing out those problems, and we will explicitly state them in the paper accordingly. 

&nbsp; 

## Concerns on the experiments.

Next, we make responses to reviewer’s concerns on the experiments individually as follows. 

1.	__The speed acceleration of SS-Conv over ST-Conv:__ In the supplementary material, we further explain that the efficiency of SS-Conv improves with the increased sizes of data batches during both training and inference (cf. Fig. 2 in the supplementary material); for example, for a plain backbone consisting of standard Conv layers (dubbed Plain24 in Fig. 2), SS-Conv brings about 16x speedup w.r.t. ST-Conv (111.8 FPS versus 7.1 FPS) at the batch size of 13; more importantly, SS-Conv costs much less memory and can thus support a batch size as large as 512, while ST-Conv can be only run at the extreme batch size of 13 on the GPU with 12G memory; running with larger batch sizes further improves the efficiency of our proposed SS-Conv (as shown in Fig. 2, FPS goes to 206.5 when running with the batch size of 512).

2.	__The performance comparison between SS-Conv and ST-Conv:__ In the submitted paper, we evaluate the performance in Sec. 5.1 of both architectures based on SS-Convs and ST-Convs, separately. The results are given out on the ADD(S) metric on LineMOD dataset, as shown in Table 1, where SS-Conv performs a little better than ST-Conv (93.5 versus 92.8), due to the use of Submanifold SS-Conv, which alleviates the “submanifold dilation problem” (cf. L149-L156 in Sec. 3.2.2).

3.	__Results on 3D object recognition of SS-Conv:__ We evaluate the performance of SS-Conv on ModelNet40 dataset [3] for 3D object recognition. The backbone we use is the small Plain12 (cf. Fig. 3 in the supplementary material) with the input resolution of $64\times64\times64$. Following [6], the training data is augmented with random azimuthal rotation to train the model; we evaluate the model on data with azimuthal rotation (z/z) or arbitrary rotation (z/SO(3)). The results of both SP-Conv and SS-Conv are shown in the table below, indicating that when encountering arbitrary rotations, SS-Conv is very robust due to its properties of equivariance and invariance, while the performance of SP-Conv drops rapidly.

|&emsp;&emsp;Methods       |  z/z | z/SO(3) |
| :----------------: | :--: | :-----: |
|  PointNet [4]      | 89.2 |  14.7   |
|  PointNet++ [5]    | 89.3 |  28.6   |
|  Spherical CNN [6] | 88.9 |  78.6   |
|  SP-Conv           | 81.3 |  21.9   |
|  SS-Conv           | 85.6 |  80.8   |

&nbsp; 

### __Reference:__
[1] He C, Zeng H, Huang J, et al. Structure aware single-stage 3d object detection from point cloud[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 11873-11882.

[2] Jiang L, Zhao H, Shi S, et al. Pointgroup: Dual-set point grouping for 3d instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 4867-4876.

[3] Wu Z, Song S, Khosla A, et al. 3d shapenets: A deep representation for volumetric shapes[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1912-1920.

[4] Qi C R, Su H, Mo K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 652-660.

[5] Qi C R, Yi L, Su H, et al. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space[J]. Advances in Neural Information Processing Systems, 2017, 30.

[6] Esteves C, Allen-Blanchette C, Makadia A, et al. Learning so (3) equivariant representations with spherical cnns[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 52-68.



 
We appreciate the reviewer’s comments on the strengths of our proposed Sparse Steerable Convolutions (SS-Conv). Our responses to the reviewer’s concerns are made individually as follows.

__a) Discussions of other sparse 3D convolutions.__

__Reply:__ We thank the reviewer’s suggestion and will supplement more discussions on the literature of 3D sparse convolutions, including Minkowski convolution [1]. As far as we know, submanifold sparse convolution [2] and Minkowski convolution share the same theory and thus share similar mathematical formulations. In [1], a rich open-source differentiation library for sparse tensors is released, and we will also enrich our library for steerable sparse tensors to benefit the community.

__b) Parameterizations of the rotation in Fig. 2.__

__Reply:__ We use in Fig. 2 a continuous 6-dimensional representation $r_{6d}$ proposed in [3] for the $3\times3$ rotation $r$. Specifically, $r_{6d}$ is defined as the first two columns of $r$, and during inference, the third column of $r$ could be computed via cross product of the first two columns. More details about the theory can be found in [3].

__c) Extension of SS-Conv to more general cases.__

__Reply:__ It’s correct that in our paper, we confirm the advantages of our proposed SS-Conv on instance-level or category-level point clouds since we focus on the tasks of object pose estimation and tracking. The advantages of SS-Conv on running speed and memory consumption (cf. Fig. 2 in the supplementary material) also make it possible to extend SS-Conv to large-scale data, e.g., 3D scan of a street or a room. 

In this short rebuttal period, we make a simple trial to apply SS-Conv to the task of point cloud based retrieval for place recognition [4]; the challenge of this task lies in the rotation-invariant local feature extraction and fusion from a large-scale point cloud. To evaluate the capacity of feature learning of SS-Conv, we simply replace the backbone of PN_VLAD [4] with SS-Conv based Plain12 (cf. Fig. 3 in the supplementary material) and conduct experiments on the open-source Oxford RobotCar dataset [5]. The results are shown in the table below, and our SS-Conv improves the original result by a margin of 7.22% on the average recall at top 1%.

| &emsp;&emsp;&emsp;&ensp;Method|  Avg recall @ 1%  |
| :------------------: | :---------------: |
|     PN_VLAD [2]      |      80.09%       |
| PN_VLAD with SS-Conv |      87.31%       |


We greatly thank for the reviewer’s curiousness about the extension of SS-Conv to more general cases, and we also feel excited to explore more potentials of SS-Conv. Through the simple experiment above, we believe in the capacity of SS-Conv to handle large-scale point cloud, and will continue to explore its potentials on more complex tasks, like 3D object detection and instance segmentation, in the future work.

### __Reference:__

[1] Choy C, Gwak J Y, Savarese S. 4d spatio-temporal convnets: Minkowski convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 3075-3084.

[2] Graham B, Engelcke M, Van Der Maaten L. 3d semantic segmentation with submanifold sparse convolutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9224-9232.

[3] Zhou Y, Barnes C, Lu J, et al. On the continuity of rotation representations in neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 5745-5753.

[4] Uy M A, Lee G H. Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4470-4479.

[5] Maddern W, Pascoe G, Linegar C, et al. 1 year, 1000 km: The oxford robotcar dataset[J]. The International Journal of Robotics Research, 2017, 36(1): 3-15.



 We greatly appreciate the reviewer’s constructive comments, and make a response to reviewer’s open questions individually as follows.

__Q1: Table 1: Are the FPS computed for complete images, or for single patches (found by the preceeding object detection / instance segmentation network)? Also in Table 1, are the ADD(S) and FPS for ST-Conv also measured using the proposed two-stage network?__

__Reply:__ For LineMOD dataset, there is only one target object considered in each RGBD image. We first crop out the RGBD segment via MaskRCNN, and then estimate the 6D object pose via our proposed network. In Table 1, we only consider the time spent on the proposed network to compute FPS, and the time spent on MaskRCNN is not included. We agree that the construction of a rulebook needs extra time and memory, but our proposed SS-Conv still takes obvious advantages on speed due to the extreme sparsity of input data on our focused datasets, e.g., the average ratio of occupied grids in the whole input voxel is lower than 1% for LineMOD dataset.

For fair comparisons, all the quantitative results of different convolutions in Table 1 are obtained via the same single-stage architecture, which is illustrated in Fig. 1 in the submitted supplementary material. The specifics of the backbone (denoted as Plain12) are also given in Fig. 3 in the supplementary material.

&nbsp;

__Q2: Sec. 3.1 / Figure 2: I am unsure what the input to the MLPs is, since the number of active sites can change, but MLPs need a constant number of inputs. Maybe the effect of the Tensor-to-Point-Block should be described in 1-2 sentences.__

__Reply:__ The input to the MLPs is the point-wise feature with a dimension of $N \times C$, where $N$ denotes the number of sampled points, and $C$ denotes the number of feature channels. For a depth segment, we transform it into a point cloud and randomly sample a small set with a fixed number (e.g., 1024 as did in our experiments). Given a feature map, the Tensor-to-Point Module uses the features at the active sites to obtain the point features via interpolation, which is achieved based on the relative coordinates of the active sites to the points. More details are given in [1]. We thank for the suggestion and will illustrate the Tensor-to-Point Module in the paper accordingly.

&nbsp;

__Q3: Sec. 3.3: What is the motivation for using a different normalization and activation for l=0?__

__Reply:__ Features with l=0 correspond to scalar fields, which are invariant to any rotations. So for these features, conventional normalizations (e.g., BatchNorm) and activations (e.g., ReLU or sigmoid) can be directly used, without special designs.

&nbsp;

__Q4: Why is there a difference in the ADD(S)-metric of the proposed method between Table 1 and Table 2 (SS-Conv vs. Ours)?__

__Reply:__ The result in Table 1 is obtained via the architecture (A1) in Fig. 1 in the supplementary material, while that in Table 2 is obtained via the architecture (A2) in Fig. 2 in the submitted paper. There are 3 differences between A1 and A2, which are listed as follows:
1. A1 uses a backbone with 12 convolutional layers (Plain12), while A2 uses a backbone with 24 convolutional layers (Plain24). The specifics of both backbones are given in Fig. 3 in the supplementary material.  
2. A1 uses a global feature to estimate poses, while A2 makes point-wise predictions.
3. A1 is a one-stage framework, while A2 adopts a second stage for pose refinement.

&nbsp;

__Q5: Supplementary Material: When using a "ResNet50" like backbone (Appendix A), how are skip connections implemented if the dimensions of the output of a layer are larger than that of the input (as done according to Eq.(10) of main paper)?__

__Reply:__ We illustrate the skip connections at the bottom of Fig. 3 in the submitted supplementary material; if the output dimension of a residual block is different from the input dimension, we will use another SS-Conv to change the dimension of the input for feature addition.

&nbsp;

__Q6: l.260--261: What does "feature channels are kept consistent" mean exactly?__

__Reply:__ In Table 1, the networks based on 4 different convolutions share the same architecture, and we use the same output feature dimensions for corresponding convolutional layers across different models. For example, the output feature dimensions are all set as 32 for the first convolutional layers of all the 4 models. We will improve the writing here to avoid the misunderstanding.

&nbsp;

__Q7: l.193: Which instance segmentation network was used, exactly? I believe this should be mentioned in the paper.__

__Reply:__ For fair comparisons, we use the segmentation results provided by the previous state-of-the-art works. Specifically, for LineMOD dataset, we use the segmentation results used in DenseFusion [2], which are obtained via a U-Net for semantic segmentation; for REAL275, we use the segmentation results used in SPD [3], which are obtained via MaskRCNN implemented by matterport [4]. We appreciate the suggestion and will add the statement of segmentation networks in the paper.

We also appreciate the reviewer’s comments on typos, references, etc. We will correct the typos and make more discussions among related works. For limitation and society impact, we discuss them in Sec. 6 briefly, and will continue to lead to more discussions in depth.

&nbsp;

### __Reference:__

[1] He C, Zeng H, Huang J, et al. Structure aware single-stage 3d object detection from point cloud[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 11873-11882.

[2] Wang C, Xu D, Zhu Y, et al. Densefusion: 6d object pose estimation by iterative dense fusion[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 3343-3352.

[3] Tian M, Ang M H, Lee G H. Shape prior deformation for categorical 6d object pose and size estimation[C]//European Conference on Computer Vision. Springer, Cham, 2020: 530-546.

[4] Abdulla, W.: Mask r-cnn for object detection and instance segmentation on keras
and tensorflow (2017). https://github.com/matterport/Mask RCNN



The paper presents a Sparse Steerable Convolution to accelerate steerable convolution (ST-Conv) via sparse tensors, while still preserving the SE(3)-equivariance. 

The authors also show its application to 3d semantic analysis including the tasks of instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking and they achieve superior performance on these datasets.  **Originality**: The authors present a new Sparse Steerable Convolution (SS-Conv) that speeds up the existing steerable convolution (ST-Conv), by borrowing benefits from recently introduced sparse-tensor-based computation frameworks. It is clear that this paper presents a non-trivial attempt in making the steerable convolution more practical to handle sparse data, e.g., irregular and sparse 3D point clouds.

**Quality**: The paper is technically sound. The motivation of using sparse tensor computation is intuitive. However, the authors are suggested to conduct experiments to directly compare steerable convolution and the proposed sparse steerable convolution along several dimensions, including memory consumption, training/inference times under a similar resolution. Also, it could be more intuitive to actually quantify the sparsity of the tackled datasets. For example, to count the number of occupied grids/voxels with a fixed resolution.  

**Clarity**: The paper is well-written. Most of the experimental results supported the claims. 

It is not clear how the authors conduct the experiment of Dense-Conv. From reading the paper, the reviewer didn’t find any detail of Dense-Conv including architecture, input resolutions, input types (image-like dense tensors???). 

**Significance**: The reviewer believes that Sparse Steerable Convolution could be a useful operator for tasks such as pose estimation and tracking in real-world applications, which could potentially make a larger impact once it is published.
 Please refer to Main Review for making potential improvements to the paper.


a) This paper aims to learn SE(3)-equivariant deep features;

b) The major contribution is propose a Sparse Steerable Convolution (SS-Conv) to address the shortcoming of steerable convolution;

c) Experiments on three tasks of 3D object semantic analysis show the effectiveness of the proposed method.  Strong points.

a) The pipeline of Sparse Steerable Convolution is given in detail;

b) Experimental results are promising.


Weak points.

a) I cannot find significant weaknesses of this paper, only minor comments.

The main contribution of this paper is changing the Steerable Convolution to sparse. As such, I would see comparisons/discussions of the proposed method with respect to some off-the-shelf sparse 3D convolutions other than [9].

For example, the Minkowski Engine. https://github.com/NVIDIA/MinkowskiEngine

When checking equation 6, I found it is similar to the Minkowski Engine, other than the SE(3)-equivariant constraint.

b) It is not clear how the rotation is parametrized throughout the paper.
Specifically, in Figure 2, how many parameters are used to denote r_1 or r_2?

c) If I understand correctly, the proposed method can handle instance-level or category-level point clouds. Can this paper handle more general cases, e.g., 3D scans of a room? 






 Yes.

Meta-review: The paper presents a Sparse Steerable Convolution to accelerate steerable convolution (ST-Conv) via sparse tensors, while still preserving the SE(3)-equivariance.
The authors also show application of the proposed method to 3D semantic analysis including the tasks of instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking and they achieve superior performance on these datasets.
All reviewers agree on the soundness of the proposed method and its efficiency over steerable convolutions or dense convolutions. 


Decision: accept