Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes PointDP, an adversarial purification method that leverages a diffusion model as a pre-processing module to defend against 3D adversary attacks for point clouds.  PointDP consists of two components (1) an off-the-shelf 3D point cloud diffusion model and (2) a classifier. Given an input point cloud, PointDP takes the following steps: (i) adding noise to the input data gradually via the diffusion process of the diffusion model, (ii) purifying the noised data step by step to get the reversed sample via the reverse process of a diffusion model, and (iii) feeding the reversed sample to the final classifier. Since PointDP does not rely on any types of pre-defined adversarial examples for training, it can defend against diverse unseen threats.

This paper also revisits the existing works via designing various types of strong adaptive attacks. It demonstrates that standard adversarial training suffers from gradient obfuscation in the point cloud recognition models as the unstructured point cloud data format requires unique architectural designs to digest.

The authors evaluate PointDP with multiple representative point cloud models and sixteen attacks and show significant improvement over existing state-of-the-art methods.
strength_and_weaknesses: +: The paper demonstrates that standard adversarial training has a major limitation in its application in 3D point cloud models due to architecture designs.
+: The proposed PointDP method leverages diffusion models to purify adversarial 3D point clouds. 
+: Experiments on six representative models show that PointDP outperforms previous state-of-the-art purification methods by a large margin.


clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easier to read. It could benefit from more rationale and background to make it easier for a non-expert like the reviewer. For example, why we first add noise and then remove them?
summary_of_the_review: Overall the paper has shown good results using the diffusion method for adversial attack defense for 3D point clouds. I am not an expert in this area but do believe this should be considered novel.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work the authors propose to use denoting diffusion models in order to aid in robustness against adversarial examples for neural networks trained on point-cloud data. The paper is very similar to what has recently been produced for 2D images in [7, 36] (using the paper’s citation numbering). The underlying principle of the work is sound and the primary contribution of this paper appears to be applying the same framework to 3D point cloud inputs. As a secondary contribution, the authors discuss how gradient obfuscation makes standard adversarial training less effective due to the intractability of computing strong white-box adversarial examples for point cloud networks. Thus, the authors argue (and empirically show) that their diffusion-driven purification is a better alternative. One question that could be interesting for future study is if one can design an adversarial example such that the purification method fails (due to poor performance of the diffusion model) but this is outside of the scope of the current work. Modulo some errors and omissions in presentation I found this paper nicely written and think it has an interesting contribution.
strength_and_weaknesses: Strengths:

The paper is well-written. Specifically the exposition on denoising diffusion models is clear and concise. Another primary strength of this paper is the experimental evaluation which is comprehensive. Hopefully the authors release their code as I am sure open implementation of 16 different attacks on point clouds would serve as a good resource to the community. 

Weaknesses:

One odd weakness of this work is the incorrect recounting of related work for attacks on point cloud models. The first attacks on point cloud models dates to roughly 3 years prior to the work the authors claim is the first. It is my understanding that [1A] was the first to demonstrate the vulnerability of neural networks on 3D data. This ought to be corrected in the related works of this paper. In addition the method of [1A], though very simplistic, has been shown to work very well in a black-box setting and even the white-box attack does not require gradients so this would be interesting to see how this plays into the gradient obfuscation argument given in this paper. 

Another odd weakness of this work is the lack of reporting on the computational cost of training the denoising delusion model. Though I am not an expert, this seems like a very costly training that must be done. In addition, ablations on the quality of the purification would be interesting. The authors provide Figure 3 which I find very interesting and think that a better understanding of what makes a good purification model and what leads to a poor purification model would strengthen the contribution of the paper. That being said, the experiments of this paper are extensive and comprehensive so such ablations are not required for a convincing study. In addition, the authors state in table 1 that models mostly have 0% robustness and while I understand that these are not reported in the main text, they ought to be fully reported in the appendix on submission for completion. 

Another place I think the paper could have been strengthened, and I am curious about why the authors choose not to perform this step, is that in [7] the authors show that diffusion-driven purification can easily be used to get certifications of model robustness. Is such an approach not computationally feasible in this setting or was it just not explored?


[1A] - https://openaccess.thecvf.com/content_CVPR_2019/papers/Wicker_Robustness_of_3D_Deep_Learning_in_an_Adversarial_Setting_CVPR_2019_paper.pdf
clarity,_quality,_novelty_and_reproducibility: The paper is clear, relatively novel, and mostly reproducible. I do not see the full details for reproducing the diffusion architecture, so I would urge the authors to declare what portions of the source code will be made open source if the paper is accepted.
summary_of_the_review: Overall, though the technical contribution is not wholly novel, I find the experimental analysis convincing and think it makes a strong empirical contribution that is of interest. There are omissions that need to be corrected and the authors should add in their rebuttal exactly what will be made open source.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a simple but effective model, named PointDP, which leverages diffusion models to defend against 3D adversarial attacks. The experimental results show that PointDP achieves significantly better adversarial robustness than state-of-the-art methods.
strength_and_weaknesses: **Strengths**
- The paper is clearly written.
- The proposed method is simple and effective.


**Weaknesses**
- The technical contributions of PointDP over the diffusion model should be clearly demonstrated. Otherwise, the proposed method could be regarded as an application of the diffusion model in denoising.
- The differences between the diffusion model in PointDP and [31] should be discussed.
clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written and with sufficient experiments.
- The novelty should be evaluated after the rebuttal. It is not clear how many differences between the proposed method and [31].
- The authors provide the supplementary material and the work is thus highly reproducible.
summary_of_the_review: The authors propose a simple but effective method to defend against 3D adversarial attacks. But the technical contributions are not clear. I will change my rating if the authors compare the diffusion model more closely.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a pretraining diffusion model for defending against 3D adversarial point clouds. They also shows the limitation of robust training for defense due to gradient obfuscation. The results have shown better qualitative performance than SOTA methods. 
strength_and_weaknesses: Pros: 
+) The idea is clear and easy to follow. They show that adversarial training does not work well on 3D point cloud recognition task. 
+) They deploy diffusion model for recovering/purifying 3D point clouds, which further increase the correct detection ability. 

Cons:
-) The methodology is straightford with limited technical contributions.
-) Diffusion model itself is a powerful generative model, capable of denoising. Limited novelty on technical contribution on purifying adversarial 3D point clouds. The authors are suggested to clarify their contributions. 
-) A lot of missing experiments, e.g. [1, 2]. It is suggested to clarify the advantage of the method with more experimental results. 
-) Missing evaluations on larger distortions, e.g. perturbation is 0.1. 
-) typo error: P3: Donget al.->Dong et al.

[1] Kaidong Li et al, Robust Structured Declarative Classifiers for 3D Point Clouds: Defending Adversarial Attacks with Implicit Gradients, CVPR22
[2] Hang Zhou et al, DUP-Net: Denoiser and upsampler network for 3d adversarial point clouds defense, ICCV19
clarity,_quality,_novelty_and_reproducibility: +) The paper is well written.
+) The idea and experiments are clear.
+) The first diffusion model based 3D adversarial point cloud defense. 
summary_of_the_review: The paper has shown potential of deploying diffusion model for 3D adversarial point cloud defense. The major problem is the lack of novely on designing their diffusion models, as well as experiments on the recent works. Therefore, it is difficult to justify their performance improvements.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes PointDP, an adversarial purification method that leverages a diffusion model as a pre-processing module to defend against 3D adversary attacks for point clouds. The rigorous formulation of adaptive attacks on diffusion models and the robustness evaluation protocol are also studied. Extensive experiments have been conducted to illustrate its superior robustness.  

Strengths: The proposed PointDP method leverages diffusion models to purify adversarial 3D point clouds. The Experiments on six representative models show that PointDP outperforms previous state-of-the-art purification methods by a large margin. The experimental analysis is convincing and it makes a strong empirical contribution to point clouds applications. The paper is clearly written and the method is effective. It is an important finding that diffusion-driven purification could provide actual robustness to 3D point clouds. 

Weakness: The technical contribution and novelty of the paper is limited. The diffusion model was proposed to perform data purification in the past research [ICML 2022], and point cloud diffusion models was proposed in [CVPR 2021], this paper seems to combine those ideas and apply it to 3D point clouds. 
justification_for_why_not_higher_score: The technical contribution and novelty is limited. Diffusion model was proposed to perform data purification and enhance the robustness of image classification in past research, while this paper apply the similar technology to 3D point clouds applications.  Of course, the new application and experimental analysis are useful to show that diffusion model is efficient for defending against adversarial point clouds. However, the technical contribution is incremental, as they only apply diffusion models to this new task without much reasoning.  The other contributions such as the rigorous formulation of adaptive attacks on diffusion models and the robustness evaluation protocol are well received.
Decision: Reject