Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a denoising-based method for randomized smoothing that converts a base classifier into a smoothed one with p-robustness to adversarial examples. It considers a practical setting where the retraining/finetuning of the base classifier is largely inapplicable (e.g. the commercial classification service with only API provided to users).  To do this, it adopts a recently proposed methodology termed denoised smoothing [1] by prepending a custom-trained denoiser to the pretrained classifier. The major novelty of this work lies at the proposed denoising method using learned score function. The new denoising method only requires training one score network and is readily applicable to defend various $l_p$ adversaries, which is a key feature not available in [1].  The experiments show the proposed method outperforms the previous denoising-based approach, and is sometimes on par with the white-box approach [2] that manipulates the classifier. 

Basically, this is an incremental work over [1] but the contributions claimed are perceived myself (though I have to admit I'm an expert on image denoising, instead of adversarial defense) 
However, I do have some concerns about the method and the experiments, listed as follows:

- The major advantage of the score-function-based denoiser is the flexibility to handle various noise types and levels.  I don't expect it can beat the specialized Gaussian denoiser [1] under Gaussian perturbation setting.  As it is the case on Table 1/2, I'm wondering what's the benefit of  the proposed denoiser over the state-of-the-art Gaussian denoisers (as used in [1]) under Gaussian noise setting?
- The flexibility to tackle various $l_p$ adversary, the key feature of the proposed method is not thoroughly evaluated. In Table 2, I suggest the authors to add comparisons to [1] with denoiser trained on Gaussian noise setting, as well as ones trained with noise type aligned with the test setting. 

[1] Denoised Smoothing: A Provable Defense for Pretrained Classifiers, Arxiv 2020  
[2] Certified Adversarial Robustness via Randomized Smoothing, ICML 2019

review_title: Good work with some incremental novel contributions

final_score: rating: 3: Clear rejection
review_content: Update: I have read the author's response and decided to keep my review, confidence, and score.

---

Summary: randomized smoothing is a method to construct provably robust classifiers via additive Gaussian noises on the input. The authors propose to learn score functions as a means to denoise the randomized image prior to a trained classification model. As the denoising + pre-trained classifier architecture is already proposed, the contribution is only limited to the choice of using a score function. The justification and realization of the method is limited for two main reasons. See below. 

1. Efficiency: one of the most critical bottleneck of randomized smoothing methods is the slow prediction time. The score-function based generative / denoising models are known for their slow sampling time, so the proposed method undermines randomized smoothing in efficiency. 

2. Many design choices in this paper is not well justified. 

2-1) How good does the RHS of Eq. (12) approximates the gradient descent procedure? 

2-2) Even if the true gradient descent can be executed, the bound in Eq. (13) seems very bad in high dimension, thus the smoothed classifier will not be accurate unless the pre-trained classifier is already robust in the local region. 

2-3) Clearly using the same score function for multiple $\sigma$ is suboptimal. Although the authors mentioned this part as an advantage, but it is not clearly compared to existing methods. Would existing methods fail if they use the same denoising function for multiple $\sigma$?


review_title: Official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary & contribution: 

in this paper the authors propose an improved method for randomized smoothing (for provable defense) which performs better and is less computationally expensive than previous work. More specifically the authors propose two image denoising algorithms (based on score estimation) that can be applied regardless of noise level/type. In fact, the paper is mainly an improvement of Salman et al. (2020) with a blind denoiser. Therefore, it is not needed to train different models for different kind of attacks. Another advantage of the method is that one does not need to retrain the classifier and does not even need information from the pretrained classifier while the method from Salman et al.  requires access to the classifier.


Strengths: 

-strong quantitative results. Experimental section is promising. The gap with white box smoothing is small on cifar. The method outperforms Salman et al.

-Only need to train one score network to handle various types of noise type/level 

-Denoiser doesn't need access to the pretrained classifier.

Weaknesses:

-In my opinion writing can be improved. I am not very familiar with the literature and it took me some time to understand section 3. More specifically I did not understand the motivation for using score based denoisers rather than a more "standard" algorithm for blind denoising. 

-Method seems to be effective for low-resolution images only. The gap with white box increases on Imagenet.

Questions for the authors:

-If I understand well, one-step denoiser can only handle gaussian noise whereas multi-step can be applied to any log concave distribution? What is the advantage of one-step denoiser over multistep, does it perform better for gaussian noise? 

-It is not clear to me why the method does not make use of the pretrained classifier but still outperforms Salman et al. which can acess the classifier? Is it only due to the denoiser's performance?

-I do not understand well the motivation for using score-based methods. Why scored based denoising in particular and not other blind denoising models? The current method is impacted by the size of the images, while most of the existing blind denoising algorithms are not affected by the size of the image. Actually, why even focus on learning base methods? Cannot simpler methods with handcrafted priors do the job? I might also be wrong. Could you please elaborate?

-Qualitatively speaking it seems to me that the visual performances are not very good when compared to existing denoising algorithms (maybe a quantitative comparison in term of PSNR with other algorithms would be relevant, rather than only showing qualitative results). Also the multi step denoiser seems to give a lot of artefacts in figure 1. 

-Aren't there missing reference in the related work regarding blind denoising ? 

-p5 "the noise makes the support of the score function to be whole space, [...] non-Gaussian or off-the-manifold samples". I don't understand that part. 


At this stage I give a weak accept, but I would consider raising my score if authors answer my concerns.

Typos:

p8 "without any re-trianing."

p7 "smoothingon" 

p4 " matching obejctiv" 
review_title: Promising results,  clarifications would be appreciated

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method based on denoising to protect classifiers from adversarial attacks. Unlike existing methods based on randomized smoothing with various noise distributions to retrain several classifiers, the proposed one uses denoising as the preprocess of the classifier. The experimental results demonstrate the proposed method has good performance.
review_title: I am not an expertise of the area of this paper. But I think the idea is interesting.

Meta-review: The paper proposes an improved method for randomized smoothing, reducing computationally complexity compared with some previous works. The authors propose to learn score functions to denoise the randomized image prior to feeding it to a trained classification model. More specifically,  two image denoising algorithms based on score estimation are proposed to be applied regardless of noise level/type.


Strengths:
- The paper shows strong quantitative results. The gap with white box smoothing is small on cifar, outperforming Salman et al. However according to the authors, the performance advantage could be mainly attributed to (1)  the use of better network architecture and (2) the multi-scale training, not the major contribution of a score-based denoiser. 
- The denoiser doesn't require access to the pre-trained classifiers.
- The proposed method only requires training of one score network to handle various types of noise type/level, although reviewers have raised concerns about motivation to having a method that only needs one denoiser for multiple noise levels -  the computational bottleneck of randomized smoothing is the prediction time rather than training time and  using the same score function for multiple noise levels could be suboptimal.

Weaknesses:
- There are some concerns about the significance of the contribution as well as novelty of the work, as the denoising + pre-trained classifier architecture is already proposed. Specifically, the work can be seen as incremental to [1], although the work uses a score-based image denoiser whereas [1] uses a CNN based image denoiser and this work is more efficient as it requires only one score network, while [1] trained multiple denoisers with respect to each noise levels. 
- Reviewers have expressed concerns on the prediction efficiency of score-function based generative / denoising models.  The proposed method might exacerbate the weakness of randomized smoothing (i.e., slow prediction), especially in high-dimensions. 
-The reviewers are curious to see the benefit of the proposed denoiser over the state-of-the-art Gaussian denoisers (as used in [1]) under Gaussian noise setting.
-Method seems to be effective for low-resolution images only. The gap with white box increases on Imagenet.

[1]. Salman, Hadi, et al. "Denoised Smoothing: A Provable Defense for Pretrained Classifiers." Advances in Neural Information Processing Systems 33 (2020).

Decision: Reject