Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper first points out the drawback of state-of-the-art defenses for 3D point cloud classification, via designing adaptive attack. And then the author analyzes the effect of adversarial training on this task. Based on such analysis, the authors propose a deep symmetric pooling operation which can enhance the adversarial robustness for adversarial training.

The presentation of the paper is clear and easy to follow. And there are extensive experiments and sufficient experimental analysis to support the claim of this paper. However, there are still some concerns that need to be solved.

1. The authors utilize the white-box attack to analyze the robustness of different pooling operations, while they should also compare the defense effect of various pooling operations under black-box attack.
2. In table 2, although the adaptive PGD attack has high success rates, the attack with L_{gather} has higher performance in some cases. What is the intrinsic reason for its superiority?
3. In the experiments of adversarial training, the authors propose to select PGD for the training. However, the authors evaluate the performance of trained models with PGD only, e.g. the results in Tables 3 and 4.  It is necessary for the authors to evaluate the results under various attack types.

review_title: This paper analyzes the robustness of 3D point cloud classification, while there are some concerns need to be figured out.

final_score: rating: 7: Good paper, accept
review_content: The paper addresses the problem of adversarial robustness in 3D point cloud representations. It claims that two of the previous defense designs do not prevent adaptive attacks. The authors then propose to use adversarial training (AT) to improve the robustness. It claims that the standard MAX pooling operation within PointNet-derivates contribute to the weaknesses. It then proposes a new pooling operation that improves the robustness under AT.

The paper claims contributions in:

-- Demonstration that current defenses do not provide real robustness.

-- MAX and SUM pooling strategies weaken models trained under AT.

-- A new sorting-based pooling operation, DeepSym, that works the best with AT against adaptive attacks.

I agree with the authors' assessment of the paper.

Strengths:

-- Interesting insights: As promised, the authors applied adaptive attacks against recent works that aimed to improve robustness. The experiments are clear and discussions make sense. The discovery that sorting-based pooling layers are more robust under AT is also insightful.

-- Strong experimental results: On the one task of ModelNet40, the authors showed significant improvement using three methods when using AT, and further improvements when using the proposed DeepSym layer.

Neutral:

-- Novelty: The majority of claimed novelties are straightforward applications: 1) application of AT on points, 2) usage of attention or sorting-based layer to improve robustness. The proposal of DeepSym is somewhat novel, but not sure if sufficient for a publication in this venue.

Weaknesses:

-- There is only one task: All experiments were done on ModelNet40. Having a second dataset would be nice. Or even some toy examples.

Clarification:

-- I imagine that the adaptive attacks are white box attacks? I couldn't find in the paper if white or black attack....

Conclusion:
The paper provides interesting insights and strong experimental results. Although the novelty is slightly lackluster, I would be happy to see this paper at the conference.


review_title: Interesting work with good experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies adversarial robustness of point cloud classification models. In particular, this paper analyzes the effects of pooling layers and conducts extensive ablation studies. In addition, this paper proposes a DeepSym operation, which is built on top of both the sorting-based pooling and the parameterized pooling. The paper shows empirical improvements under different types of attacks. I summarize the pros and cons as follows.

Pros:
1.  Robustness of point cloud classification is an important problem to study. Compared to existing works on robustness, point cloud models have unique architectures, e.g, the pooling operation, which could be an important factor to the final performance. This paper studies an important problem. 
2. This paper conducts multiple experiments of different pooling layers under different types of attacks. 
3. Empirically, this paper shows improvements under different attacks. 

Cons:
1. My biggest concern of this paper is its novelty. The sorting-based pooling and parameterized pooling layers are not new and have been studied extensively in existing papers.  FSPool proposed the sorting-based pooling while the parameterized pool layers have been proposed in multiple papers (e.g, PMA). Can the authors state the novelty/difference between this proposed method and existing ones?
2. The proposed pooling layers introduce additional parameters compared to sorting-based pooling layers, this makes the comparison less fair. Can the author talk about the number of parameters introduced by the proposed pooling layers. In particular, I'd like to see how much computational overhead introduced by the additional parameters? Is it possible to measure the inference time and compare to existing sorting-based methods (e.g, FSPool)? 
3. The proposed pooling layers shows performance decrease when there is no adversarial attack. I'd like to see more discussion regarding this. Is this method specific to adversarial defense? Or this paper is proposing a general pooling layer? 
review_title: Interesting direction, limited novelty and performance improvement

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
1. The paper performs the first security analysis of defenses and designs adaptive attacks, which demonstrate current defense designs are vulnerable.
2. The paper proves that sorting-based parametric pooling operations can improve the model’s robustness. 
3. Based on the existing problems, a deep symmetric pooling operation, DeepSym, is proposed.

Cons:
1. DeepSym seems to be simple and slightly lacking in novelty.
2. Unlike existing methods, DeepSym applies deeper neural networks. Therefore, the computational cost (both time and space consuming) of the DeepSym should be discussed in the paper.

review_title: The paper proves that state-of-the-art defenses against 3D adversarial point clouds are still vulnerable to adaptive attacks, and proposes a deep symmetric pooling operatio, DeepSym, to improve the model’s robustuness.

Meta-review: The authors develop novel adaptive adversarial attacks for 3D Point Cloud Classification tasks. They show that many existing defenses are broken by develop a novel pooling operation, DeepSym, and demonstrate that using this they can achieve significant improvements in adversarial robustness of 3D Point Cloud Classification. 

All reviewers agreed that the paper makes interesting contributions and that the setting of 3D point cloud classification is interesting from a security perspective. The shared concern of the reviewers was around novelty. This was addressed in the rebuttal to some extent, but there remained some lingering questions that made this paper borderline and unfortunately, the program committee had to decide to reject it.

I would urge the authors to revise their manuscript to clarify the novelty relative to prior work (particular those that use similar pooling operations) and resubmit to a future venue. 
Decision: Reject