Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors modify PointNet++ by replacing the vanilla SA layers in PointNet++ by a new  Anisotropic Separable Set Abstraction layer. The obtained neural network, called PointNetv3, is faster and more accurate.
main_review: PointNet++ which is a variant of PointNet is a notable neural network using for processing point clouds. PointNet++ contains two main components: Set Abstraction SA (to sub-samples the point clouds and groups the neighborhood) and a local feature extraction (which is PointNet). This paper aims to modify the SA layer to increase the efficiency and accuracy.

The authors propose a novel Anisotropic Separable Set Abstraction (ASSA) component to replace the SA. First, the MLPs in the SA layer are separated to before and after the reduction layer and are connected by a residual connection. Second, the reduction layer is replaced by an Anisotropic Reduction layer to effectively aggregate local information. This ASSA is then used to replace the SA layer in PointNet++ to obtain PointNetv3.


Various experiments with classification on ModelNet40, semantic segmentation on S3DIS and part segmentation on ShapeNet shows that the modification is indeed very useful and PointNetv3 achieves better accuracy and runs faster in comparison with PointNet++ and previous methods.

The paper is very well-written. The result is novel and interesting.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposed a novel neural network backbone architecture for 3D point cloud processing that focuses on reducing the repeated computation inside the grouped neighborhood. The author proposed a nice technique to decompose the learning of the relative 3D coordinate values and the learning of features from previous layers so that the majority of repetitive computation can be saved. The experiments are done on several classification and segmentation benchmarks.
main_review: Strong points:

- This paper addressed the problem of repetitive feature computation in PointNet++-style architecture. The problem is important yet has been long ignored. The proposed idea is simple but elegant and effective. 

- The experiment results are strong. When viewing performance and model runtime together, the proposed method significantly outperforms previous methods by a notable margin.


Weak points: 

- The paper does not include any visualization that helps understand why would the proposed method work. The authors could provide a visualization figure of reduction $\mathcal{R}$ in Equation (2) by showing what area is activated during the reduction operation in a local neighborhood, similar to Figure 19 of the PointNet paper; or by showing the critical point set that is activated, similar to Figure 18 of the PointNet paper.

- It is good that the authors provide Equation (2) for ASSA module, which is the main contribution of the paper. However, the other two proposed PreConv SA and Separable SA are not explained by math equations which makes it really hard for readers to understand and compare with ASSA.

- Figure 3 is very confusing. It does not reflect what ASSA is doing and what difference it is compared to SA and others. I suggest that the authors remove Figure 3 and simply use math equations to explain the four SAs.

- Minor but important notation issue: in Equation (2), the author messed up with the set bracket and function bracket, i.e. the whole set indexed by $j$ should be inside the bracket of $\mathcal{R} ()$ instead of outside of it.

- Minor problem: there should be a line break at line 253?

- Though the author pointed out that the theoretical speed-up of ASSA compared to vanilla SA is $K=32$, why does the actual speed-up is much smaller than this? The author can decompose the actual experiment latency numbers of each part (e.g. data loading, encoder, decoder, etc.) similar to Figure 2 to show the actual speed-up due to ASSA.

- Though this work is on point cloud deep learning, conceptually it is not a direct extension of PointNet and PointNet++. Therefore, it is inappropriate to name this paper "PointNetv3". Instead, the author should emphasize "anisotropic" in the title and the network name. I'm willing to increase the review score to 8 as long as the authors are committed to using another name other than "PointNetv3".
 
=================================

After rebuttal:

I'm satisfied with the response from the authors. It resolved most of the concerns in my review. 

I regard this paper as a very strong submission. It solves an important but long-ignored problem in point cloud deep learning. The solution is beautiful. I'm happy to increase my review score to 9.

There could be further improvement to the paper. Please see my official comment to the authors below.
limitations_and_societal_impact: No such issue was found.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors present PointNetV3, which is a faster and more accurate version of PointNet++.  The authors first analyze the latency bottleneck of PointNet++ and find that computation takes up to 70% of total latency, and then propose to reduce the computation cost via several techniques. Namely, separate SA modules are used to reduce the computation cost of vanilla SA (since MLP after reduction is much more efficient); Anisotropic reduction is applied to achieve higher accuracy (original PointNet++ shares conv. weights for all neighbors, which is suboptimal). The authors also analyze the effectiveness of width and depth scaling to improve the efficiency-accuracy tradeoff of PointNetV3 models. Experiment results on S3DIS, ModelNet40 and ShapeNet Part are impressive.
main_review: This paper attempts to analyze the efficiency bottleneck of the widely used PointNet++ architecture in 3D deep learning and improve its efficiency-accuracy tradeoff. I feel that the motivation / analysis of this paper is thorough and the proposed solution is simple and effective. The experiment results show that PointNet++-like architecture can achieve significantly better accuracy-efficiency tradeoff comparing with more complicated design such as KPConv, which is interesting. The overall writing quality of this paper is also good.

However, I still believe that there are some challenges to be addressed in order to make this paper more influential.

- The evaluation benchmarks chosen in this paper are all very small datasets, I'm quite curious whether PointNetV3 can be applied to larger datasets such as SemanticKITTI, nuScenes and Waymo to achieve state-of-the-art level accuracy. Even for indoor tasks, I believe ScanNet might be a better benchmark in 2021 comparing with S3DIS.

- It seems that this paper assumes the number of input points per sample is around 15,000. This will be sufficient for most object / indoor-level datasets, but not quite sufficient for LiDAR perception tasks. For example, each scene on SemanticKITTI has ~120K points, and for Waymo each scene contains ~150K points. Can PointNetV3 scale up to these large scale scenes and perform competitively? Notice that at >100K points per scene, neighborhood query might become a significant bottleneck.

- The comparison with point-based methods are really impressive, but SparseConv-based methods are not compared against. These methods are particularly strong at **one-pass inference** for the scenes. That is to say, we do not need to divide the input into sliding windows. I wonder if PointNetV3 can have advantage over SparseConv-based methods in terms of **per-scene latency** (not 16x15000 batched inference latency) and accuracy. Again, it is better to compare at large-scale scenes to make PointNetV3 more influential.

I would raise my score if these concerns (especially the latter two) are addressed properly.

===

I've updated the score to 6 after reading the author response.
limitations_and_societal_impact: As I have indicated in the main review, the major weakness of this paper is that all experiments are done on small-scale datasets and the number of input points in the point cloud is not large (around 15K). It will be great if the authors can present some clue that the proposed method can scale up to large-scale input, in terms of both efficiency and accuracy.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the limitations of Set Abstraction in PointNet++. and proposed its variants including PreConv SA, Seperable SA, and Anisotropic Separable SA to reduce the computational cost and improve the accuracy. 
main_review: The proposed method provided an improved version to the original set abstraction and increase the inference speed while maintaining almost the same or even better speed compared to PointNet++ and PosPool.

The paper is well-organized. I think the description of evolution from Vanilla SA and ASSA.

Just a question on Equation (2), repeating f_i^{res} three times seems redundant even though multiplied by the distance in three directions. Have you tried other more efficient ways to incorporate anisotropic property, like using 1/3 channels as in PosPool, or simply addition?

limitations_and_societal_impact: N/A
review_title: null

Meta-review: The paper presents a new variant of PointNet that's faster and more accurate. This paper received positive reviews and all reviewers recommended acceptance. The reviewers find many positive points including impressive empirical results. AC does not find grounds to overturn this consensus recommendation. The authors should incorporate the suggestions of the reviews when revising the paper for the camera ready version. 
Decision: Accept (Spotlight)