Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: At a higher level, this paper is proposing several ways to measure the sensitivity and smoothness of representations learnt by 3D point cloud networks. This paper presents several ways to measure the sensitivity of point cloud networks to rotation, scale, translation and local structure. The Paper also proposes different metrics to evaluate the smoothness of encoded local 3d structures and the complexity of representations learnt by the network. The analysis is done using several point-based neural networks trained on the ModelNet40 classification task and shapenet part segmentation task.
main_review: 1. Paper is well written and claims are stated clearly.
2. The ideas explored in the paper are important for understanding how much point-based networks are vulnerable to different perturbations in the input and also how smooth/complex the learned representations are. This might lead to the design of better architecture that is robust to these perturbations.
3. Experiments are done on a variety of architectures and elucidate the ideas presented in the paper.
limitations_and_societal_impact: 1. This is not a limitation per se, but it would be interesting to see how much data augmentation changes sensitivity to different perturbations.
2. Furthermore, there are several works on rotation invariant neural networks for 3d shapes. I wonder if the representations learnt by these architectures are different when evaluated using proposed metrics.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces metrics that reflect the representation quality, and properties, acquired by a deep-net that is applied in 3D Point-Cloud (PC) processing tasks. Concretely, the authors introduce six novel metrics that in aggregate capture different types of regional sensitivities e.g., the sensitivity of the DNN in rotations on specific regions/rotation-angles of the input PC. They further explore the extent that modern architectures in standard benchmarks acquire “spatial smoothness” --  do nearby points have ‘similar’ effects/importance for the end task? And finally, they explore the acquired representation complexity of the DNN by adapting a 'multi-order interaction' metric to PC data.  

These metrics are being applied on a well-selected subset of modern PC-DNN architectures such PointNet++ or DGCNN when these are trained either for PC-object classification, or part-based object-segmentation. The analysis of the metrics in these tasks/architectures highlights a number of interesting points that oftentimes are very simple/intuitive: e.g., PointNet fails to encode local 3D structures (e.g., compared to PointNet++) — or includes results that are more subtle, but still highly intuitive & rich: e.g., adversarially training a PC-DNN with rotation/translation attacks; will force it to learn more global PC structures compared to standard training; which is intuitive as the local structures, per the new metrics, are those that on average are more susceptible to rotations.
main_review: This paper is original, written with clarity, and introduces ideas, per the evaluation metrics that I think will be very helpful for the PC-Deep-Learning community. The exposition is clear and most findings well align with intuitions known by practitioners.  For instance, it is widely known that PointNet based variants do not cope well with rotations and that ad-hoc mechanisms that are typically applied to address this issue (e.g., the pc/feature rotation transformation loss in the original PointNet) have very limited success -- which is partially why a lot of work keeps happening in rotation-invariant or equivariant PC-DNNs. This paper brings a very elegant instrument to measure such a dependence (among others): the Shapley values. These can capture the sensitivity of different PC "sub-regions" to a specific quantity of interest, not only rotations but other structures like edge vs. surface vs. volume like structures. To this end, another interesting finding is that PointNet++ is the network that pays most attention to intrinsic 3D structures: "masses", which can be why in many settings is still the better performing, e.g., see [1].

The findings regarding how the adversarially trained net shifted its attention from orientations and positions to structural information and increased its spatial smoothness are important and they demonstrate a use case of how the new metrics can be used to examine/analyze the behavior of black-box models, training losses, etc.

The only concern that I have with the paper is that IMO lacks extensive use-cases analysis. While the networks architectures tested seem adequate; the datasets/tasks are less extensive. E.g., why not using ShapeNet  (or ModelNet50) for a larger classification problem? How is the training size affects the different metrics? How about trying auto-encoders or other forms of learning used for point-cloud processing? While, this seems like an extra mile that another paper can do; if you had included analysis say on AutoEncoders, GANs etc. to complete the picture, I would argue this paper should have very strong acceptance. 


Minor:
L187. is represented ->  are
Figure 3. (b) -> are these objects supposed to be at different angles? I cannot tell. 


[1] Goyal et al. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline




limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies how to understand the quality of knowledge representations encoded in point cloud DNNs. Based upon the Shapley value, the paper presents six metrics to evaluate the model vulnerability toward different types of input variations. In addition, it presents an analysis regarding the smoothness and multi-order interactions of these attributions, pointing out the restrictions of current point DNNs. The paper makes interesting observations and draws useful conclusions, supported by a range of experimental evaluations.
main_review: This is a diagnosis paper studying an important problem. The submission is clearly written, making it easy to follow. The content is also well organized with enough technical details provided. Generally speaking, the submission is technically sound. The motivation behind different metric designs is clear and the experimental results successfully support the main claims. The diagnosis is quite new for 3D point cloud analysis. Its difference from previous works is also clearly discussed. The related work is adequate as far as I can tell.

I still have some questions and I hope the authors could provide more explanations to strengthen the submission.
1. Is the conclusions task dependent? Would the observation change if analyzing with another task, say semantic segmentation?
2. Is the smoothness score defined for a specific transformation or averaged across all types of transformations considered?
3. Is there any intuition why does the biased knife category appears for adversarial training?
4. Why does the GCNN mainly encodes extremely low-order or high-order interactions but not something in between?
5. How does the size of the selected regions influence the observations and analysis?

I think the paper is making an interesting analysis which reveals the limitations of current point DNNs and could provide intuitions for future designs.
limitations_and_societal_impact: As I mentioned, there are some details unclear. Addressing them could potentially strengthen the submission. It would be interesting to include sparse voxel conv based methods given that it is one of the main streams for sparse point cloud analysis. It would also be interesting to see whether rotation equivariant networks could mitigate the rotation sensitivity issue.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper measures the sensitivity of various point cloud classification networks (like PointNet, PointNet++, DGCNN) to rotation, translation, scale, and local 3D structure. It also proposes ways to measure the 3D smoothness of a network. Based on these analyses, the work provides various insights. One of the main insights being that the DNNs don’t handle rotation very well. Another insight is that training with adversarial perturbations helps.
main_review: Strengths

- The kind of analysis used in the paper to quantify different aspects of DNN could be useful for 3D point cloud processing.

Weakness

- The data augmentation could have a very significant impact on the conclusions in the paper (as explained below) but those details are missing. Specifically, it is unclear what kind of data augmentation strategies are used while training. For most of the best-performing models on point cloud classification, rotation augmentation is not used while training the DNN [1]. Only scale and translation augmentations are used. This is because while testing the object models are already axis-aligned. Hence, rotation augmentation does not help in improving the test set performance.  If this standard practice (i.e using scale and translation augmentation and not using rotation augmentation) is used in this paper as well, then the insights provided in the paper are very much expected, as a DNN would only be invariant to changes it has seen while training. Moreover, it would explain why adversarially trained GCNN, which would take rotated samples as input while training would be less sensitive to rotation. This would severely limit the contribution of the current work. Therefore without knowing this detail it would be difficult to judge the importance of the work.

- The paper lacks discussion about prior works which also point out that DNN for point cloud processing is sensitive to rotation (for example [1, 2]). The paper should add such a discussion to contextualize the findings with regards to what is already known.

- It would also be good to mention the accuracy of the DNNs used for the analysis to make sure that well-trained DNNs have been used. More details about the exact training protocol should also be provided.

- The resetting of absent points (L124-L132) could introduce unnecessary artifacts impacting the performance of DNNs. Most DNNs like PointNet, PointNet++, DGCNN should be able to handle the different number of points by default. It would be important to compare if simply removing points and replacing missing points with center lead to the same performance.

[1] Revisiting point cloud shape classification with a simple and effective baseline

[2] PointCNN: Convolution On -Transformed Points, NeuRIPS 2018

Minor Comments:

- It seems like many insights mentioned in the introduction repeat the same information. It would be useful to mention the insights in a more concise form. The reviewer feels that simpler and more fundamental insights would be more useful than many complicated ones; at least in the introduction.


Overall Recommendation:

In the current version, critical information is missing. Hence, one cannot judge how useful the contributions are. There are some other issues as well (see weak points). I look forward to the response by the authors. For now, I would recommend a borderline reject rating for the paper.

limitations_and_societal_impact: Please refer to weaknesses discussed in the main review.
review_title: null

Meta-review: This paper introduces metrics that reflect the representation quality, and properties, acquired by a deep-net that is applied in 3D Point-Cloud (PC) processing tasks. 
This paper presents several ways to measure the sensitivity of point cloud networks to rotation, scale, translation and local structure.
All reviewers laud the originality, clarity of the manuscript, and the ideas. 
The initial paper lacked comparative analysis on different datasets, which the rebuttal seems to have addressed. 
All reviewers agree that this paper provides an interesting result for the community to justify acceptance. 
Decision: Accept (Poster)