Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to solve the potential classification problem of the adversarial samples with the help of diffusion models. Specifically, the authors firstly give some analysis of the regions of the conditional generation. They claim that the conditionally generated samples will concentrate on the regions around the original adversarial sample. Then through majority voting, the proposed method can help find the correct labels. Experiments show that the proposed method does help improve the performance.
strength_and_weaknesses: Strength:
- The paper gives some theoretical analysis about the conditional generation, which can help clarify the understanding of the problem.
- The experiments show that the proposed method can improve the robust of the classification of adversarial inputs.

Weakness:
- In general, it seems that the classification of the adversarial samples is overkilled. To improve the robustness, the authors need to run the time-consuming diffusion models more than 10 times to get the conditional generation and then run the classifiers. It is unreasonable.
- The theory part just includes some clarifications of common sense, it hard to get some novel ideas through the definitions or theorems. 
    - It is also unclear if it is proper to bound the complex data support  with hyper-balls. For example, in the data manifold, if the data labeled by 1 is supported in a 2-dimensional rectangular like region with large length and width ratio, can we use definition 3.2 or theorem 3.3 to measure the results?
    - In theorem 3.4, KL divergence may not be a good method to measure the similarity of two distributions since it is not a distance. 
clarity,_quality,_novelty_and_reproducibility: The paper is in general clear and easy to follow. And there seems no reproducibility problem. But the novelty is limited, and the theory cannot support the claim very well.

summary_of_the_review: For one thing, the paper help clarify the conditional distribution of the diffusion models, which may help the community; for another, the theoretical contribution seems limited and the proposed method overkills the adversarial classification problem in terms of resource and time.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method, DensePure, designed to improve the certified robustness of a pretrained model (i.e. classifier). Specifically, DensePure uses the diffusion model to denoise the adversarial input to get multiple reversed samples, which are then passed through the off-the-shelf classifier, followed by majority voting of inferred labels to make the final prediction. The extensive experiments demonstrate the effectiveness of DensePure by evaluating its certified robustness given a standard model and show it is consistently better than existing methods on ImageNet.
strength_and_weaknesses: Quality/Clarity: the paper is well written and the techniques presented are easy to follow. Its motivation is clear to use diffusion model to denoise input to improve classification accuracy. On a technical level, I do not see much new contribution, where most equations are from diffusion model. For the BEiT in Table 2, do we use the same BEiT model for evaluation while comparing to Carlini 2022? We do not know its gain from the model itself or from the label voting. Also it is better to add experimental comparision w/o diffusion model.

Originality/significance: the idea is interesting, which uses diffusion model to improve data quality to improve classification performance. Another contribution is the label voting from multiple samples. However, both diffusion model and pretrained classifier are known, which makes DensePure an incremental approach. In addition, the diffusion model is only for Gaussian noise, which limit the application of this approach. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The idea to include denoising step with diffusion model to improve classification performance is incremental. And more experiments are needed in Table 2 (see above for details)
summary_of_the_review: DensePure with the diffusion step can improve the robustness of prediction in the late stage. Overall, it is a good paper, I am ok if it is accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the conditions under which diffusion model can work well for purification of adversarially perturbed samples. A simple diffusion-based purification method named DensePure is proposed by using majority vote  and achieves higher certified accuracy in comparison with other method on CIFAR-10 and ImageNet. 
strength_and_weaknesses: Strengths:

S1) Theoretical strength: This paper theoretically analyzes why and how diffusion-based purification model can enhance the adversarial robustness of a given classifier for the first time. The robust region of a sample (under a deterministic inverse purification process and a base classifier)  is characterized as the union of several convex sub-robust regions (indexed by samples with the same ground truth label). The newly characterized robust region may provide with a larger robust radius than other methods, which indicates better robustness. 

S2) Modelling/algorithmic strength:  The proposed DensePure performs consistently better than existing methods on ImageNet, with 7% improvement on average. 

Weaknesses:

W1) Time complexity of DensePure: As it requires repeating the reverse process multiple times, DensePure is really time-consuming and this drawback may prevent it from further applications to large datasets.

W2) Claimed but seemingly not well-supported contributions:  The authors claimed their first contributions as "We prove that under constrained data density property, an adversarial example can be recovered back to the original clean sample with high probability via the reverse process of a diffusion model". However, it seems not well-supported. The reason is as follows:

Let $x_0$ be the clean sample and $x_a$ be its adversarially perturbed sample. 
Theorem 3.1 characterizes the distribution of inversed variable $\hat{x}_0$ from a scaled adversary $x\_{a,t}=\sqrt{\alpha}\_t x\_a$ as 
$$ \mathbb{P}(\hat{x}_0=x|\hat{x}\_t=x\_{a,t}) \propto p(x)\cdot\frac{1}{\sqrt{(2\pi\sigma^2_t)^n}}\textnormal{exp}\big(\frac{-\|\|x-x_a\|\|_2^2}{2\sigma^2_t}\big)$$ Letting $x=x_0$, we obtain
$$ \mathbb{P}(\hat{x}_0=x_0|\hat{x}\_t=x\_{a,t}) \propto p(x_0)\cdot\frac{1}{\sqrt{(2\pi\sigma^2_t)^n}}\textnormal{exp}\big(\frac{-\|\|x_0-x_a\|\|_2^2}{2\sigma^2_t}\big)$$

which seems unable to directly imply "$\mathbb{P}(\hat{x}_0=x_0|\hat{x}\_t=x\_{a,t})$ is high" or "$\mathbb{P}(\|\|\hat{x}_0-x_0\|\|\le \delta |\hat{x}\_t=x\_{a,t})$ is high with a small $\delta$". 

So, I failed to understand why "under constrained data density property, an adversarial example can be recovered back to the original clean sample with high probability". What is the "constrained data density property"? Can the authors provide upper bounds on $\|\|\hat{x}_0-x_0\|\|$ to verify "an adversarial example can be recovered back to the original clean sample with high probability"? Did I miss something?

----------------After rebuttal------------
In the rebuttal, both of my main concerns about "Weakness 1: Time complexity of DensePure" and "Weakness 2: Claimed but seemingly not well-supported contributions" have been well explained. Therefore, I decided to change my score from 6 to 8.
clarity,_quality,_novelty_and_reproducibility: Clarity score 7/10: In general, this paper is well-written. However, it still has typos. For example, "the robust region for data region with ground-truth label under $\mathbb{P} (\cdot; t)$" should be "the robust region for data region with ground-truth label under $\mathcal{P} (\cdot; t)$".

Quality score 6/10: The theoretical results as well as the proposed DensePure are technically sound. However, the first claimed contribution seems not well supported. (See Weakness (W2) for more details. If I misunderstood this paper, the authors please point it out directly.) 

Novelty score 7/10: To the best of my knowledge, this paper theoretically analyzes why and how diffusion model performs well in purification by proposing three interesting and novel theorems. 

Reproductivity score: 6/10: Although the  code is not shared, I think the details in the supplementary materials are sufficient for reproduction. 

summary_of_the_review: In general, this paper provides novel theoretical results and an effective algorithm.  I suggest "marginally above the acceptance threshold" mainly due to "Weakness (W2) Claimed but seemingly not well-supported contributions". See Weakness (W2) for more details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed to increased robustness to adversarial attacks of an off the shelf classifier by using a (of the shelf) diffusion model as a data augmentation preprocessing step.


strength_and_weaknesses: Strength:
- simplicity of the framework
- use of off the shelves diffusion model and classifiers
clarity,_quality,_novelty_and_reproducibility: Sections 1 and 2 are well written and could be understand by a reader unfamiliar with adversarial attacks/robustness (which is my case).
I made the educated guest for section 3 and 4.
In section 5, experimentation are well described and reproducible.
summary_of_the_review: The framework is simple and clever.
I could not verify the theoretical analysis part as it is out of my skills.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:  This paper studies the conditions under which diffusion model can work well for purification of adversarially perturbed samples. Then a simple diffusion-based purification method named DensePure is proposed by using majority vote. The extensive experiments demonstrate the effectiveness of DensePure by evaluating its certified robustness given a standard model and show it is consistently better than existing methods on ImageNet.

Strengths: Theoretical strength: This paper theoretically analyzes why and how diffusion-based purification model can enhance the adversarial robustness of a given classifier for the first time. The robust region of a sample is characterized as the union of several convex sub-robust regions. The proposed DensePure performs consistently better than existing methods on ImageNet, with 7% improvement on average.

Weakness: DensePure is really time-consuming and this drawback may prevent it from further applications to large datasets. Both diffusion model for purification and random smoothing technique are existing methods, and thus DensePure is a straightforward combination of these two methods. 
justification_for_why_not_higher_score: The technical contribution seems limited and the proposed method overkills the adversarial classification problem in terms of resource and time.
justification_for_why_not_lower_score: The theoretically analysis of why diffusion model is able to perform purification task is well grounded and important.
Decision: Accept: poster