Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper builds upon Weiler's [24] steerable filters on volumetric data
and creates a new equivariance by design by sparsifying group convolutions.
Authors are inspired by Graham's [9] sparsification of convolution that takes into account the state of a voxel (active /inactive) without sacrificing the steerability. 
The kernel construction follows the spherical harmonics weighted by the Clebsch-Gordan coefficients and radially weighted with a Gaussian.
The paper contains an extensive experimentation, comparing several 3D convolutions, and applying the method on 6dof instance pose, tracking and simultaneous class pose and size.



main_review: This paper develops a new efficient scheme for steerable convolution on volumetric data. Volumetric convolutions are very expensive and any way to accelerate them has tremendous practical impact.

The paper takes the two main ideas of 3D steerability [24] and sparsified convolutions [9] to design a tensor-based sparsified convolution that is steerable in SO(3). The authors provide all details of the novel design. The reader needs some background reading on steerability and to understand the general steerability constraint (4) that is applicable for any dimension of the input and the output (the formula is much simpler for scalar fields). 

The active sites (voxels) are stored in a hash table $H$ updated by the active/inactive state. This is a very elegant design indeed and $H$ together with the feature vector $F$ constitute the vector that is updated from layer to layer.

The paper is one of the few, might be next to Esteves ICML 2019 (equivariant embeddings that are used for relative SO(3) computations)  that contains extensive experimentation on non-classification tasks (no MNIST, Cifar10 etc) but standard 6dof object pose benchmarks like LineMOD and REAL275. 

These applications alone could be a paper in a mainstream vision conference.

For instance-based 6dof pose the authors segment the volume creating the initial hash table and start with the vector (R,G,B,1) as feature input. After several layers, features are converted into point coordinates that are regressed to compute the pose. 
At this point, the reader has really to go to the appendix to understand the loss functions. While ground-truth poses are used the loss functions are ICP-like. Please move the losses to the main text. The Appendix is highly informative and like the paper itself a delight to read.

The method outperforms all methods on pose and size and comparable or better to DenseFusion of G2L on instance-based 6dof pose estimation.

POSTREBUTTAL: I read all reviews and the authors' responses and I stay with my positive mark: The combination of steerability and sparisty is not a simple integration, and the experiments on pose are novel in the context of steerability. The reader can learn a lot by reading the paper.
limitations_and_societal_impact: Limitations explained in Conclusion.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper discusses the development of a novel method called Sparse Steerable Convolution for the estimation of an object’s 3D pose. As the pose estimation is usually done using SE(3)-equivariant deep feature learning, the proposed method provides more efficient computation using sparse tensors. The paper presents carefully the theoretical background of the method and experiments it via using two different benchmark image datasets for estimating the 6D object pose.
main_review: The paper is well written, scientifically sound and provides enough details about the method. The method is novel and provides means for efficient and still accurate object pose estimation. The paper gives credit for previous work. Novelty and contribution are clearly stated. The theoretical premises are carefully explained. The method is tested and analyzed using two very different benchmark image datasets and the results are compared to multiple state-of-the-art object pose estimation methods. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes sparse steerable convolution that efficiently learns SE(3)-equivariant features and demonstrates its usefulness in object pose estimation tasks. Essentially, the proposed method is a combination of 3D steerable cnn [24] and submanifold sparse convolutional networks [9]. The proposed method, being able to extract SE(3)-equivariant features, is proven to be useful in 6D pose estimation, outperforming previous state-of-the-art methods
main_review: This work to my best understanding is a nice engineering effort to bring 3D steerable convolution to sparse voxels. The experiments show that the property of SE(3)-equivariance extracts better information of object poses while being more efficient than its dense counterpart.

However, I am concerned about the novelty of this work. To my best understanding, almost all of the theoretical basis of 3D steerable convolution came from its previous work [24] and submanifold sparse convolutional networks [9], and this work mostly reiterates previous literature. I am not sure what makes this novel, especially considering the venue of the conference. Although I agree that in engineering perspective, the proposed sparse steerable convolutions is efficient and reasonable, considering the venue of the conference, I am not sure if this work satisfies the novelty claim that it makes. If there is any unique problem being sparse, I believe it should better be highlighted. The comparison in the supplementary material only discusses engineering perspectives such as speed or memory consumption, which is the novelty of sparse convolution, not this work.

limitations_and_societal_impact: It would be helpful if the authors highlight failure cases of this work compared to a regular sparse convolution neural networks.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends steerable convolution by taking advantage of the sparsity of 3D data, such that the speed in both training and inference is significantly faster than the original one [24]. The method itself is motivated by SP-Conv [9], e.g. the site state definition and “submanifold dilation”. In evaluation, it is demonstrated that the method outperforms or is comparable to the state-of-the-arts in 3D object pose estimation, tracking and size estimation. 
main_review: The paper is well-written, as it provides a gentle review of the background works which this one is built upon, and most of its discussion about related works and motivations is adequate and precise.  I’m also fully convinced by the practical impact of this work -- the noticeable speed up of steerable convolution is certainly something the community desires. 

However, as a reviewer, I do concern the “novelty” of this work, as it reads to me, is "just" applying the engineering details proposed by  [9], e.g. “submanifold conv” to steerable conv [24].  I fail to see the technical novelty which is actually owned by this paper. I’d like to hear from other reviewers regarding this, and the author is of course welcomed to point out their contribution if I unfortunately missed.

Other minors regarding to quantitative comparison: (1) It would be nice if the authors also compare the running speed in table 2-4, or at least comment on their relative speeds. This potentially could make the readers appreciate more about the practical impact of this work.

(2) In table3, the proposed method leads in almost every metric, except IoU50, which shows a big gap when compared to  FS-Net[4]’s 92.2% accuracy. It seems that the proposed method has more catastrophic failures. Could the author show / comment on the type / reason of failures. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes a reformulation of Steerable Convolutions (ST-Conv) , dubbed Sparse Steerable Convolutions (SS-Conv),  that allows them to run on sparse voxel grid while still preserving SO3 equivariance.

Specifically, accelerations is achieved in two ways: 1) convolutions are run only on activated voxels, 2) in the receptive field of activated sites, only active input features are convolved.
To further increase efficiency and alleviate the "dilation" problem of sparse convolutions, authors also propose a modification to SS-Convs, dubbed Submanifold SS-Convs, where output voxel state is kept consistent with the input one, making it possible to construct a deep but efficient network. In practice Manifold SS-Convs are alternated to SS-Convs to guarantee high performance and efficeincy.

The proposed method is first compared to ST-Convs, Dense Convs, and Sparse convs on the LineMOD dataset, where it demonstrates similar performance to ST-Convs while being 2x faster. Furthermore, it is validated through the following 3D semantical analysis tasks: instance-level 6D pose estimation, category-level 6D pose and size estimation, and category level 6D pose tracking. 

main_review: Strengths:
- Sparse implementation of ST-Convs could be beneficial for community working with sparse 3D data, as allows for efficiency that is on par with standard 3D convolutions
- Proposed application-specific architectures exploit Steerable convolutions to achieve state-of-the-art performance and demonstrate the effectiveness on steerable convolutions on tasks that had not been tackled by this architecture before

Weaknesses:
- Novelty: a more efficient implementation of an existing method is not enough novelty for NeurIPS, especially given that for the presented experiments SS-Convs only result in a 2x speedup wrt ST-Convs, and authors have to actually use a single-stage architecture to reach real-time performance on 6D object pose tracking. 

Questions for authors:
- In Table 1, SS-Convs outperforms ST-Convs. How do you justify this, given that SS-Convs is a sparse implementation of ST-Convs? Are there any other differences between the two considered architectures, or is it due to the use of Submanifold SS-Convs whitin the architecture?  If so, it should be specified.

- Are Submanifold convolutions. novel? From the paper, it is hard to say if the if the idea of using Submanifold SS-Convs is novel, or if it has already been propose for sparse 3D convolutions and adapted to this framework. 


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper extends the concepts of sparse manifold convolution to the steerable convolution. In particular, a sparse steerable convolution operation is proposed to reduce the redundant computation in volumetric data, while maintaining the SE(3) equivariance. Extensive experiments conducted on three tasks (i.e., instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking) demonstrated the efficiency and effectiveness of the proposed SS-Conv.

main_review: Overall, the idea of this paper is straightforward and feasible. It is desirable to extend the sparse convolution to steerable convolution, especially in light of the expensive computational cost.  The reported experimental results also demonstrate better efficiency of the proposed SSConv operation.

However, the real technical contribution of this paper is somewhat limited, since the sparse convolution has been well-developed and achieving remarkable performance in other fields such as semantic segmentation. In addition, several well-optimized libraries such as SparseConvNet and Minkowski engine further simplified the implementation. 

Experiments:

-Please increase the readability of Figure 3, the fonts are too small to be recognized.

-Learning generalized local descriptors from 3D point clouds also requires rotation and translation invariance. I am wondering if the proposed SSConv can be integrated into the FCGF [1] framework to achieve better registration performance.

[1] Fully convolutional geometric features. In ICCV, 2019.

-Please also report the FLOPS of different implementations in Table 1.

-In Table 2, the performance of the proposed approach is inferior to G2L [3] when the second stage is not used, what is the possible reason? what is the impact of the second stage on the proposed framework? What is the extra computational cost introduced by the second stage?


Minor comments:

Line 82, the reference is missing.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: Initially, two of the reviewers expressed concerns about the paper and ranked the paper below acceptance. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper (highest rate was 9!)  and recommended acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version.
Decision: Accept (Poster)