Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper systematically analyzes robustness of self-supervised adversarial training against adversarial attacks, which conducts thorough evaluation on several representative point cloud classifiers with different types of adversaries. Beyond point cloud classification, the task of part segmentation is also compared and analyzed. 
main_review: The paper reveals positive impact of adversarial training on improving adversarial robustness of multiple point cloud classifiers. I really appreciate that the authors conduct a thorough evaluation under different settings. The observation that robust local features are vital to achieve adversarial robustness is interesting and also meets our common knowledge.  
limitations_and_societal_impact: The following concerns can be important to further improve the paper:
1. There exists a large number of abbr. to make the paper very difficult to digest. Only the important and popular terms are suggested to use abbr.. More importantly, the definition of performance metrics CA and RA is missing in the paper. I suppose CA and RA could refer to clean accuracy and robust accuracy, but their forms remain unclear to the reviewer.

2. Technically, the threat models such as PS, PD, and PA and their experimental settings are important and sensitive to adversarial robustness. However, generation of adversaries should be well investigated and evaluated, which is missing in the experiments. For example, how existence of outliers affect adversarial robustness?

3. The APT performs better than AJT in experiments. The authors claim that distribution gap between SSL and original recognition tasks leads to poor classification performance of AJT. However, in UDA on point cloud classification, SSL is verified its effectiveness to mitigate domain gap (see [a]). Can you explain different performance of your paper and [a]?
[a] I.  Achituve, H. Maron, and G. Chechik. Self-supervised learning for domain adaptation on point cloud. CVPR 2021. 

4. Insight given in the experiments of part segmentation encourages a more interesting SO(3)/SO(3) setting used to evaluate adversarial robustness of rotation-equivariant methods.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work builds on recent adversarial attacks on 3D point clouds and recent self-supervised methods on 3D ( 3D jigsaw[28] and 3D rotation prediction[27]) to evaluate the robustness of the 3D models that were trained in such SSL setup. Comprehensive experiments are conducted on 3 different benchmarks in 3D classification and one in 3D part segmentation that show a marginal improvement in robustness by adding SSL pre-training before performing a standard adversarial training. Finally, the authors perform an ensemble of different robust models to improve the robustness of 3D point cloud classification.  
main_review: **Strengths** :

- The work provides comprehensive 3D classification experiments on several 3D datasets, including the realistic ScanObjectNN. 

- The first work to investigate the effect of self-supervision on 3D deep learning robustness, as well as studying the robustness of 3D segmentation.


**Weaknesses**:

- The paper lacks technical novelty. All the SSL and adversarial attacks and defenses are developed in previous works, which makes the manuscript more or less an evaluation paper.  The proposed Joint SSL adversarial Training (AJT) is shown in the paper to be less effective than the naive SSL pre-training (APF). Performing an ensemble of robust models is not novel since it was shown before in [b]

- One of the findings of the paper ( that DGCNN is more robust than other architectures) was already found in [73] in their comprehensive study of different 3D architectures, including DGCNN. 

- The work has a flawed experimentation setup. While the authors did show improvement in robustness, there are some caveats in the setup. The authors used a single $\epsilon$ value for the PGD attack ( 0.005) based on a rejected ICLR’21 submission [21]. Published works (like [12,19,73]) usually report several $\epsilon$ values for PGD attack or run on C&W formulation to make sure that the defense/attack methods work on generic attacks and not for that specific $\epsilon$ ( that might be too small for the attack to work).

- The uncertainty in the results is not quantified. It is not clear whether the small improvements over AT baseline ( that can be as low as 1%) are statistically significant. Reporting the variance of the results is highly appreciated in this scenario. 



**Minor issues** :

- Some grammar mistakes need to be fixed ( line 117 e.g. )

- The paper misses recent work on 3D attacks [a], and ensemble defenses [b].

- The presentation of the paper can be improved. For example, Table 2  has 323 reported results, which is very difficult to follow.  Using plots, charts, and other visualizations can improve the presentation of the paper. The text size should be consistent in all of the figures and plots. Fig 3 has tiny legends for example.



[a] Tsai et.al, Robust adversarial objects against deep learning models. AAAI Conference on Artificial Intelligence (2020)

[b]Florina et.al,  Ensemble Adversarial Training: Attacks and Defenses, ICLR (2018)

limitations_and_societal_impact: Adding a section about the vulnerabilities of 3D deep learning methods in safety-critical applications ( like self-driving cars systems) would be appreciated.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper systematically studies the impact of various self-supervised learning proxy tasks on different architectures and threat models for 3D point clouds. Experiments on three datasets imply that self-supervised learning (SSL), as a pretraining task, improves the robustness of the 3D point-cloud-based model for classification. It is observed that DGCNN and the 3D jigsaw SSL task are more effective w.r.t. robustness, which is claimed to benefit from more robust local features.
main_review: Strongness:
+ Comprehensive experiments on multiple datasets
+ The overall message is clear: DGCNN and the 3D jigsaw SSL task are more effective w.r.t. robustness.

---

Weakness and Questions:
1. The clarity of the paper can be improved. 
    - L164: $D_t$ is not well defined yet. I assume that it is a data point indexed by $t$.
    - There are many abbreviations, e.g. ST, AT, CA, etc. It is better to highlight those words where they are defined, so that the readers are able to quickly localize their meanings when they read the experiment section. Alternatively, the definitions of those abbreviations can be emphasized again in the table or figure captions. Besides, the definition of RA seems to be missing, while I assume that it stands for the accuracy after the attack.
    - (minor) The overview paragraph of each section is not informative enough currently. Although it actually describes the following content, it fails to guide me when I first read it.
2. From Table 2, it seems that the impact of SSL (self-supervised learning) and AT (adversarial training) are not complementary. For example, AT + SSL (jigsaw) seems to perform similarly or even worse than SSL only. Can the authors explain it? 
3. Is there any data augmentation in ST and AT baseline? If no, can the authors add baselines with data augmentation similar to your SSL tasks (rotation and jigsaw)?
4. It is questionable whether the contribution of the paper meets the standard of NeurIPS. From my perspective, the paper focuses on scientific observations by comparing different existing network architectures and pretext tasks w.r.t robustness. According to the related works of the paper, there are no significant technical contributions. However, the observations proposed by this paper are interesting: learning robust local features are important. Despite the comparison with PointNet and PCT, the lack of deeper analysis on local features is a drawback. For example, a common practice of SSL is contrastive learning, and there are several works about 3D SSL (like PointContrast), which also focus on local feature learning. More pretext tasks relevant to local feature learning can be included to verify the point. Besides, the autoencoder (shape reconstruction) can also rely on local features, like NSFA(Detail preserved point cloud completion via separated feature aggregation.), which can provide an apple-to-apple comparison about the impact of local feature learning.

---

Typo:
L201: default -> by default

---
**Update after rebuttal**
According to the additional results presented in the rebuttal, I slightly raise my score to 6 (Marginally above the acceptance threshold), and hope to see the additional results are included in the revision.


limitations_and_societal_impact: The authors have not adequately addressed the limitations.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents studies on exploring the impact of various self-supervised learning tasks on three different architectures and three different threat models for 3D point clouds:
(1) three primary backbone architecture: a shared multi-layer-perceptron-based network PointNet, DGCNN, and a transformer-based network Point Cloud Transformer (PCT) [17];
(2) three threat models: point shifting (PS), point dropping (PD), and point adding (PA); and
(3) three self-supervised learning pretext task: 3D Rotation (predict 3D rotation angles), 3D Jigsaw, and Autoencoder (point cloud reconstruction).
Then, two strategies were used to integrate self-supervised learning and adversarial training, including (1) adversarial pre-training for fine-tuning (APF), which uses the SSL tasks only for pre-training, and (2) adversarial joint training (AJT), which jointly trains the SSL task with the recognition task.

Overall, the contribution of this paper is not on the method (Section 2) but on the experiments that compare different architectures, strategies, and models, and then show that what combinations will lead to better robustness.

In short, the experiment considers four datasets (ModelNet40, ModelNet10, ScanObjectNN, and ShapeNetPart) and two tasks: classification and segmentation.
main_review: Major weaknesses:

(1) The technical novelty is weak, as the contribution is more on the experiments and results rather than the method.

(2) For the threat models:
- From Figure 2, Point addition (PA) seems to be just adding random points near the surface, so the attack looks weak.
- Line 176 says that a larger number of modifications would be easy to detect. This argument sounds strange, cause this work is to explore how AT (adversarial training) can help to improve the robustness of the trained network, so dropping just 10% points does not sound sufficient.  It is better to perform the experiment by trying different (and larger) amount of point drops. 
- The point drop is only 10%; doing so does not really affect global tasks, particularly classification. See Fig.5 in this paper: "SampleNet: Differentiable Point Cloud Sampling".  Dropping large proportion of points (>10%) still can't affect the classification accuracy.  Therefore, the setting for PD looks too weak, particularly for the classification experiment.
- Also, from Figure 2, it seems that all the salient points were on the bottom of the chair. Does it make sense? Why not consider existing methods for 3D key point detection?

(3) About the third architecture, i.e., PCT:
- The description on PCT [17] is too short and not informative in the paper. From supp., it seems that this PCT is developed based on EdgeConv (DGCNN) with k=32 (?).  Since this paper [17] is not published yet, should motivate why this is a good point transformer to pick, since there are other point transformers on arxiv, e.g., https://github.com/Strawberry-Eat-Mango/PCT_Pytorch and https://github.com/lucidrains/point-transformer-pytorch
- But in fact, why not also try other point transformers at the same time and see if you have consistent results?  The result in Section 3.1 seems to suggest that DGCNN is better than PCT [17], but I wonder if it is also true for other transformers (see above).

(4) I can see that the authors have tried their best to discuss the "insight" in each experiment but it is still very hard to understand the connections between the experimental results and the underlying reasons.

Overall, my rating is more borderline.

======================================================

Other comments:

Line 69: please use the full word for AT here. It was defined too early in line 33.

Line 177: what is the meaning of 14,5?  14 or 5?

Missing reference:
- Adversarial robustness: From self-supervised pre-training to fine-tuning. CVPR 2020.
limitations_and_societal_impact: Even though this is an exploratory paper that focuses on the experiments, the authors better discuss the limitations of the experiments.
review_title: null

Meta-review: I recommend acceptance. This is solid experimental work on an interesting and relevant question: the adversarial robustness properties of the most popular 3D point cloud architectures. The question is relevant as a lot of follow up work and engineering efforts are built on these and similar architectures. Even though I accept some reviewer’s concerns that the paper might be thin on novelty, my opinion is that this solid experimental study and its findings will be of interest to the community. 
Decision: Accept (Poster)