Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: As the title suggests, the paper revisits and studies PointNet++, a popular neural network for point cloud representation learning. Based on the study, the paper discovers two results: first, it proposes a set of training strategies to boost the performance on PointNet++ so that it achieves SOTA results on various benchmarks; and second, it shows how PointNet++ can be modified and scaled to further boost performance.
strengths_and_weaknesses: Strengths:
- The paper achieves impressive performance on various benchmarks (87.7% on ScanObjectNN, 74.9% mean IOU on S3IS, 87.2 IOU on ShapeNetPart)
- The paper conducts a systematic study showing the improvement in performance because of various factors (Table 4 and Table 5). In my opinion, such findings are useful.
- The paper adopts ways to improve the PoinNet++  architecture. This could be useful for future research as they can use a more improved and modern version of PointNet++.
- The paper shows how the findings about training strategies are generalizable to network architectures in Table 6, however, this result should be made more rigorous as discussed in the weaknesses section.

Weaknesses:
- The study showing that the findings about training strategies are generalizable to other networks is very useful (as discussed in the strengths) but limited. Specifically, only two other networks have been evaluated. I would suggest evaluating at least one simple model like SimpleView and a recent model like PointMLP.
- It is unclear why information about FLOPs has been omitted in Table 2 and Table 3. It would be important for readers to know the strengths and limitations of various methods. Similarly, information about Parameter Count is only omitted in Table 3 for the ShapeNetPart dataset. It is unclear why specific parameters are omitted for different datasets in the main paper.
- In Table 2, PointNet, DGCNN, and additional models (SimpleView, PointMLP) with improved training strategy should be added, so that readers can quantify the difference in performance purely as a result of network architecture and also compare models across parameter count, throughput and FLOPs.
- Potential misrepresentation of prior work: The paper says “However, SimpleView simply adopts the same training strategies as DGCNN. In contrast, we conduct a systematic study to quantify the effect of each data augmentation and optimization technique”. It seems like a misrepresentation of prior work as SimpleView does compare models across different augmentation and optimization schemes, and finds that the ones used in DGCNN are better. The paper should provide more information about what is meant in this comparison.
- Information about scaling is confusing. From my understanding, PointNeXt-S is the smaller valiant (L182-L185), however, in Table 4 PointNeXt-S is mentioned alongside Scale-up. Similarly, scaling rules for ShapeNetPart are not consistent (Table 3). Does the usual scaled-up model not perform well on this dataset? If so, this should be discussed.

questions: Several questions have been asked in the Weakness sections. Overall the paper has merits but there are parts where the paper is weak. I will update my review based on the rebuttal.
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to improve PointNet++ by slightly modifying PointNet++, increasing model scale and employing a few training strategies. Comprehensive experiments on S3DIS for semantic segmentation, ScanObjectNN for real-world object classification, and ShapeNetPart for object part segmentation. 
strengths_and_weaknesses: 1. Although the novelty seems limited, the paper may have a positive effect on the community. Some existing works (even actually also based on PointNet++) just follow the trend and apply popular techniques to point clouds without very convincing reasons. The paper may cool down such behaviours. 

2. The paper looks solid. The proposed method achieves several SOTAs on multiple tasks and datasets. 
questions: 1. The novelty seems a bit limited. The receptive field scaling (normalization) looks interesting and technically sound. However, according to the ablation study, the improvement is not that significant.  

2. In Table 4 and Table 5, it cloud be better to include SGD as a reference. 

limitations: No significant limitations.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper explores a variety of tricks to improve the performance of PointNet++ for point cloud classification and segmentation. They first show that by incorporating modern training strategies and data augmentations, the vanilla version of PointNet++ can be improved significantly. Then, to scale the PointNet++ for better performance, they propose the InvResMLP that adopts residual connection and separate MLPs in Set Abstraction layer to support scalable PointNet++, where the relative coordinates are also normalized to ease the model optimization. The experiments demonstrate that the proposed improved version of PointNet++ (PointNeXt) achieves promising performance on 3D semantic segmentation (S3DIS), 3D object classification (ScanObjectNN) and 3D part segmentation (ShapeNetPart). 
strengths_and_weaknesses: Strengths:
* The paper conducts extensive experiments to explore various data augmentation and training strategies, which greatly improve the performance of vanilla PointNet++.
* Although simple, the proposed normalization of relative coordinate is reasonable for easing the optimization.   
* The experiments show that the proposed InvResMLP can enable a larger model to improve the model performance. 

Weaknesses:
*  Although the reviewer agrees that the authors have systematically explored the potential of vanilla PointNet++, the technical contribution of this paper is still not enough to be published at NeurIPS: see below comments.  
* The explored data augmentation and training strategies are well-known tricks for training a point cloud model, and it is also well-known that vanilla PointNet++ can achieve much better performance than the original officially reported performance. 
* The proposed InvResMLPis is also straightforward by adding a residual connection (to solve vanishing gradient) and moving more MLPs after max-pooling (to reduce calculations). 
* The PointNet++ is hard to scalable to larger point cloud scenes, such as some outdoor scenes, and also the performance of PointNet++ series is inferior to sparse-voxel-based networks (like in larger dataset ScanNet). This paper does not show that an improved version of PointNet++ can still be the next-generation framework for point cloud processing in such a larger dataset.    

questions: How about the performance of PointNeXt in larger datasets like ScanNet? Can it be extended to larger outdoor scenes like SemanticKITTI?
limitations: The limitations have been well discussed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, authors work on the point cloud understanding task and point out that we may need to revisit the training startegies and scaling methods in thi field. By detailed analysis and experiments, authors claimed that a large improvement in this field is introduced by these two aspects, rather than the network architecture designs in recent years. Furthermore, authors also introduce micro designs to improvement the PointNet++, and achieve state-of-the-art performance in three benchmarks. 

I like this paper, revisting the training strategies and micro designs instead of proposing something "novel" and new. I will give an accpet. 
strengths_and_weaknesses: Strengths:
1. I like the motivation in this paper. A revisiting in point cloud understanding is necessary and can help others greatly.
2. The detailed experiments and analysis make the experimental results solid. 
3. The presentation and organization are good. 

In general, I like this paper and recommend accept.

Weaknesses:
1. In L138, the authors claimed that "the radius is dataset-specific ..." This phenomenon is not a finding. We can always expect detailed hyper-parameters can cause varying performances on different datasets. A general fixed setting is appreciated.
2. InvResMLP present nothing new. the Inv-Res-Block has been proposed by MobileNetV2 and ConvNeXt already. 
3. For the experiments, I acknowledge authors repeated all experiments three times. However, a critical issue is that all three datasets are small. To prove the main idea's effectiveness, I suggest authors conduct experiments on large datasets, like ScanNet for segmentation and SUN RGB-D for object detection. 

questions: 1. In L45， authors claimed that "For the S3DIS segmentation benchmark, can increase by 13.6% ". Is this simply achieved by training strategies (i.e., data augmentation and optimization techniques) and without any modifications to network structure?

2. I wonder why the authors consider the InvBlock design? What if we change it to a bottleneck structure in ResNet? 

3. From Tab.4 and Tab. 5, we can see that normalizing the relative positions by Eq. (2) could only improve the performance by 0.3\% for both ScanObjectNN and S3DIS. Compared to other modifications, it could not be considered a "significant influence (L138)".

4. In L142, the authors claimed: "without normalization, ... larger weight..." I am interested in the distributions of position encoding layer weights. Also, if the authors cloud explain L143, "This makes ..., ... weight decay..." What results would weight decay cause?
limitations: Authors addressed the limitations and  potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper presents a series of training strategies and settings that can improve PointNet++ to match the performance of state-of-the-art architectures. The AC agrees with reviewer jBW5 that the novelty of the paper is limited and some phenomena were observed before. However, the detailed training strategies might have the potential to benefit the research community. Open-sourcing the code will be therefore important.


confidence: Less certain
Decision: Accept