Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses several protocols including data augmentation, point distribution, loss function, ensemble scheme, and testing models, which serves as a kindly reminder that the training protocol matters. As claimed, earlier work like PointNet++ can still achieve comparable performance to more recent methods. Such observations are useful, as different methods are supposed to be developed and measured under a unified setting. Moreover, the authors investigated a new projection-based SimpleView method by converting point clouds into depth images, achieving SOTA performance without pretrained CNNs.

Although such detailed reviews over the training protocols are helpful, the authors failed to propose any novel methods or evaluation metrics (notice the SimpleView is also a commonly used methods in other multi-view framework), resulting in relatively inadequate novelty and originality. In practice, when we evaluate or re-implement these deep point cloud networks, we naturally attempt to add and remove some basic tricks. I believe there are a lot of people already knowing what has been explored in this paper. Nevertheless, this work still has a great practical value that helps us quantitatively understand how different protocols influence several commonly-used models. Moreover, different methods are compared only for point cloud classification task on ModelNet40, which only rely on global feature and cannot fully demonstrate the representation ability among different methods. For SimpleView, providing the image resolution is preferred.  Additionally, some typos, e.g. in "randon" in Table 1, should also be corrected. 

Pros:
This paper investigates protocols for various point cloud networks for point cloud classification, with an observation that simple PointNet++ and SimpleView can already achieve competitive performance

Cons:
This paper lacks academic insights and novelty, making it more like a technical report. Besides, it only focuses on the classification task, which cannot convincingly conclude the representation ability of different methods. Some typos and missing of experiment settings

review_title: This paper revisits various training protocols and experimental settings for point cloud classification. Moreover, this paper proposes a simple yet effective projection-based SimpleView method, which achieves state-of-the-art performance on the ModelNet40 classification benchmark. However, I will consider it as a valuable technical report, instead of an appropriate academic paper for ICLR.

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper systematically analyzes performance of point cloud classification methods with strict control over the setup and hyperparameters including data augmentation, loss, input representation, model epoch selection. Through extensive experiments, this paper reveals that there seem no real improvements due to the proposed model architecture in the last two years and all performance improvements come from the training/testing setup and hyperparameter tuning. 

Paper Strengths:
1. This paper provides important findings of the source of performance improvements in the domain of point cloud classification. If the findings are correct, it will imply that many efforts prior work has been put in terms of innovating new network architecture are not very effective as we expected, sometimes might be less useful than using a better training/testing protocol. 
2. The presentation (language and logic) of the paper is clear (though part of the reason is that there is no involving technical description ), I really enjoy reading this paper, and I appreciate the authors for their efforts.
3. The provided experiments are really extensive, with 7 tables and 2 datasets, which makes the findings quite convincing. 

Paper Weaknesses:
1. As this paper has written, there is no scientific novelty in the paper. In my mind, It is totally okay to not propose any new method/architecture but it would be nice to see a solution to the revealed problem. Specifically, this paper has revealed an important problem that prior work in innovating network architecture does not seem to clearly bring improvements. However, there is no clear solution/direction proposed in the paper to this problem. What should we do as a community next? Should researchers in point cloud classification stop innovating these useless and sophisticated network architectures? Should we just limit our research by exploring different types of data augmentation? I would guess the answer to these questions is no. But then, what should we do? It would be nice if the paper can bring further insight into this direction
2. The paper’s findings/claims are mostly based on experiments, so it would be nice if the paper can release its code to be used by the community and validate if it is really true that the efforts put in neural network architecture are not useful in terms of performance in point cloud classification. Without careful validation by the community, the strong claim made in this paper could be misleading and discouraging further research for innovating new approaches in this domain
3. Although many quantitative experiments are provided, it would be nice if this paper can add more qualitative analysis to help readers understand where the improvements are. 
4. I believe there are 100+ methods proposed in the recent 3 years at many conferences/journals for point cloud processing and classification, but only 4 are evaluated in this paper. It would be nice if the authors can add more representative baselines to validate the claims in order to double confirm if it is true that there is no performance gain from the architecture side. It might be possible that the claim is true within the compared 4 baselines and there might be other methods which really improves performance solely because they have a better network

Detailed Comments
1. It might be good to combine the third contribution in the introduction with the second one?
2. In the section of “our protocol”, this paper uses the fixed set of 1024 points instead of re-sampling different points at each epoch. Why? As the paper described, using the re-sampling point strategy can effectively increase the training dataset of points, which is good. It would be nice to justify why not using such a good strategy. Also, this strategy is fair unlike the one using feedback from the test set which is kind of cheating. 
3. In the section of “generating depth images from point cloud”, this paper uses two strategies, one using the minimum depth of all points which makes sense to me. But the latter one which uses a weighted average of depth does not seem to be reasonable as it is essentially creating imaginary non-existing points which correspond to the averaged depth

Justification:

My decision is made mainly because I feel the findings in this paper are important and need to be shared with the community. Sometimes, people care too much about novelty and try to design more and more sophisticated networks. If it turns out these innovative networks do not really bring performance gain, we need to re-think what we should do. However, there are spaces to be improved in this paper as I pointed out above. It would be nice to see some revision in the final version if the paper is accepted, including potential solutions/directions about where we should go and more analysis (qualitative and baselines) to support the claim.

Post-rebuttal review

After reading the authors' responses and other reviews, I would like to stick to my original rating to accept this paper. Though there are minor problems that I still have concerns (e.g., why not using re-sampling for all methods which is a better strategy than using fixed points), I am satisfied with the responses to my primary concerns about the paper. 


review_title: An application paper with no scientific novelty, but might bring a large impact to the community

final_score: rating: 7: Good paper, accept
review_content: This paper studies the factors that are related to point cloud classification but independent of model architecture. Then a light-weight projection-based model is proposed. Substantial experiments are conducted to show how the auxiliary factors affect the evaluation results and the proposed method can perform at similar level compared with state-of-the-art methods.

On the whole, I think this is a good paper as it addresses one of the most important problem of current research in point cloud classification.  We need to distinguish what actually brings the performance improvement: is it the evaluation scheme or the new model architecture itself. Extensive experiments show that 1) different methods indeed use different protocols and 2) data augmentation, loss function, voting play important role in the model performance. It's valuable to know what kind of data augmentation and loss function can improve performance generally in controlled setting. The paper is well written, rigorous and easy to follow. I think this paper would be a good contribution to the community. 

Some questions I have:
(1)What's the reason that performances of architectures in Table3 under RSCNN/DGCNN outperforms the performance in Table4? Is it because RSCNN/DGCNN use feedback from test set?
(2)For performance in Table 6,  do you train the the state-of-the-art method with the same protocol for SimpleView?
(3)Have you tried what the performance would be given different amount of data for different architecture under same protocol?
(4)For other more sophisticated operations in related projection-based methods, have you tried to see if they are useful under same protocol?

--------
After discussion:

After reading the author's response as well as the opinions from other reviewers, I will stick to my original rating. The authors resolve most of my questions and concerns and I look forward to a revised version of the paper.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Summary:
In this paper, the author(s) do a careful analysis on the classification performance of various modern point cloud processing networks and show empirically that with evaluation protocol set the same for different models, PointNet++, which is a relatively old model, has similar or better performance than newly proposed methods. The author(s) also show a simple projection based baseline SimpleView that can work surprisingly well on point cloud classification task. They evaluate methods on ModelNet40 and ScanObjectNN datasets.
-----------------
Pros:
1. The paper is well written and easy to follow.
2. The author(s) conduct detailed break-downs on the evaluation protocols used by various modern point processing models, which I think is valuable to research on point cloud data. It is also surprising that factors beyond architecture design can make such a difference in the evaluation. I think following the same evaluation protocol can be very helpful for subsequent research. This also indicates evaluating point processing models solely on ModelNet40 might not be a good practice since it poses bias to some augmentation/architecture, which might be false if models are applied in real-world data. These messages are worth to be known by the community.
3. The author(s) also show that a surprisingly simple baseline that makes use of different view projections can work quite well on point cloud object detection.
-----------------
Cons:
1. It would be nice if the author(s) can include some failure mode analysis on different models. Since the SimpleView does not operate directly on point clouds, it might have distinctive failure modes compared with other models. It might be helpful for designing point cloud processing models in the future.
2. It is unsure how such projection-based methods, like the MVCNN and SimpleView, handle larger scenes (e.g. with covered objects or a scene scan in autonomous driving scenarios) and per-point tasks such as part segmentations.
-----------------
Misc:
1. Will you release source code?
2. The paper format seems not complying with ICLR format, especially the font?
3. The overall figure and the projected images in figure 2 do not match？
4. Page 6 last line, choice -> choices
-----------------
Post-rebuttal review

I carefully read through the rebuttal and other reviews and I would like to keep my original rating. The author(s) addressed my concerns and I think it is a good paper for the community.
review_title: An interesting and nice paper on analysing various modern point cloud processing models

Meta-review: This paper received three recommendations of accept and one recommendation of reject.   The paper is mixed.  The results presented are both compelling and will have impact on the community.  The AC does not agree with R2's views that the paper requires proposal of a novel method for acceptance.  At the same time, the AC also does not agree with the views of the other reviewers that the current experiments alone are enough to carry the paper without more conclusive statements.  As hinted by R3, simply pointing out the problems is not enough without proposing how to adjust our models and experimentation protocols in the future is insufficient.  

In its current state, the paper would make for a good workshop submission.  Alternatively, the AC suggests to the authors to expand on the SimpleView baseline and or propose alternative solutions or protocols.
Decision: Reject