Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper observes that a state-of-the-art Diffusion models could be a good denoiser applicable for Denoised smoothing when combined with pre-trained classifiers, specifically to construct an off-the-shelf smoothed classifier that offers robustness certificates. The paper establishes several practices for this pipeline which enables a significant improvements in performance compared to previous similar attempts, e.g., the use of one-shot denoising unlike standard practice, and a proper way to scale the given noisy image to make the pipeline in compliance to the certification protocol.
strength_and_weaknesses: **Strength**

* The paper is easy-to-follow, and proposes a simple, easy-to-use method
* The paper presents an extensive evaluation covering large-scale dataset such as ImageNet, as well as exploring diverse architectures such as ViT
* The practice established in the paper could be useful in the future works given its significant improvements compared to previous similar attempts.


**Weakness**

* Although the paper claims a good practical performances by utilizing pre-trained models, but lacks on validating the effectiveness of the methodology itself: here, I could see the method can be also applied in the standard, non-off-the-shelf setup by training each diffusion model and classifier on the target dataset, e.g., CIFAR-10 from scratch for a fairer comparison - which I think a good ablation study to add. 
* The evaluation could also be strengthen by comparing Average Certified Radius (ACR) of the models which is often in the literature for a metric that considers both accuracy and robustness. 
* A discussion on the cost for certification (in other words, inference cost/overhead) compared to existing model would be helpful and worth to be added.


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The technical novelty can be questionable, as the method itself does not introduce new component over existing method of Denoised smoothing and Diffusion model. 
summary_of_the_review: Despite its lack of technical novelty, I think the empirical novelty of the observation and its practical significance could overweigh it thus for now I lean to accept for the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for certifying the adversarial robustness of off-the-shelf models with pre-trained Denoising Diffusion Probabilistic Model. The main idea is to use a denoiser after introducing noise for robustness certification and well-trained diffusion models can work well as denoisers. The noise introduced by the diffusion models can be used seen as the smoothing process and single-step denoising is then applied. Their main contribution is extending the denoising-based certification model to using a diffusion model. They take advantage of the recent advances in diffusion models and greatly improve the quality of denoising and certification. The experiments on ImageNet and CIFAR both show significant improvement in certification accuracy.
strength_and_weaknesses: Strength: 
- An elegant solution to a challenging problem. The intuition and method are both relatively simple, yet highly effective. Although it is not so novel as the idea of using a denoiser is already proposed, using a diffusion model is novel, and pre-trained diffusion models significantly improve denoising accuracy. 
- The one-step denoising is well-thought-of. It reduces computation costs by a great deal. Yet I'm curious why the certification accuracy is worse when the denoising result is more accurate.

Weakness:
- I hope to see more details on the hyperparameters for the diffusion process, including the noise scale during diffusion \alpha_t, and the number of diffusion steps chosen during smoothing as it is related to the smoothing level. 
- I would like to see details on the backbone model for baseline models, as using ViT VS. ResNet could be very different in terms of clean accuracy on ImageNet.
clarity,_quality,_novelty_and_reproducibility: - The writing is overall clear and in good quality. 
- Reproduction details are given yet no code is given.
- The work seems reasonably novel to me. It's a smart application of diffusion models in the certification robustness field.
summary_of_the_review: I lean to accept this paper as it wins me over with a simple yet effective method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Based on the framework of denoised randomized smoothing (Salman et al. (2020)), the authors applied larger diffusion models as the denoiser and observed significantly improved certified robustness (when the perturbation range is not too larger) compared to existing methods. 
strength_and_weaknesses: Strength: The result shows the benefit of using the off-the-shelf diffusion model as the denoiser for improving certified robustness. Some limitations (e.g., fast degraded certified accuracy with increased perturbation levels) and some ablation studies in terms of one-step and multi-step denoising are discussed.

Weakness: There is no new contribution to the methodology of denoised randomized smoothing. My rating is based on the reported numerical improvement and the discussion.
clarity,_quality,_novelty_and_reproducibility: Clarity: very well organized
Quality: Provide new insights with an in-depth discussion
Novelty: There is no new contribution to the methodology of denoised randomized smoothing. But I think the overall insights and findings are sufficiently novel.
Reproducibility: Code is not provided; can't not verify reproducibility
summary_of_the_review: In general, the claims are well supported by the presented results, and both advantages and limitations are discussed. 

To understand whether the improved certified accuracy comes from the data itself, I would like to see an ablation study that uses the one-shot denoised images from diffusion models as data inputs, and then apply them to standard randomized smoothing. If there are performance gains compared to unperturbed images, then this result will show the improvement is due to the fact that denoised images are more robust to Gaussian perturbations than clean images.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper demonstrates that one can achieve state-of-the-art certified $l_2$-robustness using the existing method of denoised smoothing, by simply replacing the custom denoiser by an off-the-shelf diffusion model. 
strength_and_weaknesses: The method is technically very simple, and yet highly effective, which I view as a major strength. The result is significant because it is a "scale is all you need" type result for certified robustness, which has the potential to have a big impact on this subfield as a whole.
The authors also provide some additional experiments as ablations, which I believe to be valuable, i.e. showing that fine-tuning can help and demonstrating that a variety of noise levels improve the denoisers over more restricted noise levels.
The authors compare to many baselines which is good.

However, one point of criticism is that the authors do not actually provide any results on full diffusion as opposed to one-shot denoising. They provide convincing intuitive explanations of why one would expect that one-shot denoising is better suited for this task, but the examples they provide might be quite cherry-picked. I suspect that the reason why they do not provide numbers for comparison is that, as they admit, it is far too expensive to run this many steps on every single forward pass (of which many thousands are needed for randomized smoothing). I think the authors need to either a) include numbers for full diffusion (preferred option) or b) weaken their language in Section 3 where they claim in the paragraph on "High accuracy" that "Section 5 experimentally validates this hypothesis."

Another minor criticism is about the misleading title. I understand that this title is good for calling attention to this paper but I find "for free" to be very misleading, given that of course the accuracy vs. (certified) robustness tradeoff still persists and, crucially, that each verified sample still takes orders of magnitude more resources than a single prediction. A less snappy, though more accurate title would be much better.
clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly. The only thing I believe should be clarified better is the latency description in Section 4 ("is 1.5 seconds per image on an A100 GPU at a batch size of 32"). Presumably, this is not the time it takes to verify an image but just to classify it so make this crystal-clear to all readers.
I have no negative comments about quality, novelty or reproducibility.
summary_of_the_review: I believe this paper is well-written and has significant results, especially because of its simplicity. I argue that this paper should be accepted at ICLR, under some minor revisions that I describe in my main review.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All four reviewers recommend accepting the paper (6-8-6-8). The paper proposes using a state-of-the-art diffusion model as a denoiser for Denoised Smoothing, resulting in a simple and yet highly effective solution for certifiably robust pre-trained classifiers. The experimental validation is strong. Minor concerns raised by reviewers were answered in the rebuttal. 

Without questioning the validity of the method, the title is considered to be slightly misleading. The authors are encouraged to consider a more accurate title for the paper.
justification_for_why_not_higher_score: The execution is solid and the results show the method is highly effective. However, the contribution is also incremental.
justification_for_why_not_lower_score: All reviewers are positive, and all minor concerns were addressed.
Decision: Accept: poster