(Certified!!) Adversarial Robustness for Free!
Authors: Nicholas Carlini
Florian Tramer
Krishnamurthy Dj Dvijotham
Leslie Rice
Mingjie Sun
J Zico Kolter
Venue: ICLR 2023 poster (2023)

Abstract: In this paper we show how to achieve state-of-the-art certified adversarial robustness to 2-norm bounded perturbations by relying exclusively on off-the-shelf pretrained models. To do so, we instantiate the denoised smoothing approach of Salman et al. by combining a pretrained denoising diffusion probabilistic model and a standard high-accuracy classifier. This allows us to certify 71% accuracy on ImageNet under adversarial perturbations constrained to be within a 2-norm of 0.5, an improvement of 14 percentage points over the prior certified SoTA using any approach, or an improvement of 30 percentage points over denoised smoothing. We obtain these results using only pretrained diffusion models and image classifiers, without requiring any fine tuning or retraining of model parameters.

Published as a conference paper at ICLR 2023
(CERTIFIED!!) ADVERSARIAL ROBUSTNESS FOR FREE!
Nicholas Carlini'!
Florian Tramer"!
Krishnamurthy (Dj) Dvijotham!
Lestie Rice² Mingjie Sun²
J. Zico Kolter
Google CCanneiie Mellon University
3Bosc Center for Al
ABSTRACT
In this paper we show how to achieve state-of-the-art certified adversarial robust- ness f2-norm bounded perturbations by relying exclusively off-the-shelf pre- trained models. To do so, we instantiate the denoised smoothing approach of Salman et al. (2020) by combining a pretrained denoising diffusion probabilistic model and standard high-accuracy classifier. This allows us to certify 71 11% ac- euracy on ImageNet under adversarial perturbations constrained to be within an & norm of & = 0.5, an improvement of 14 percentage points over the prior cer- tified SoTA using any approach, or an improvement of 30 percentage points over denoised smoothing. We obtain these results using only pretrained diffusion mod- els and image classifiers, without requiring any fine tuning or retraining of model parameters.
INTRODUCTION
Evaluating the robustness of deep learning models to norm bounded adversarial perturbations has been shown to be difficult (Athalye et al. 2018: Uesato et al., 2018). Certified defenses- as those based on bound propagation (Gowal al, 2018 Mirman al., 2018) or randomized smooth- ing (Lecuyer et al., 2019 Cohen et al. 2019 offer provable guarantees that model's predictions are robust to norm- coum-bouded adversarial perturbations, for large fraction of examples in the test set.
The current state-of-the-ard approaches certifyy robustness to adversarial perturbations bounded in the &g norm rely on randomized smoothing (Lecuyer et al. 2019; Cohen et al., 2019). By taking a majority vote the labels predicted by "base classifier" under random Gaussian perturbations of the input, if the correct class is output sufficiently often, then the defense's 's output on the original un-noised input guaranteed be robust l2 norm bounded adversarial perturbations.
Denoised smoothing (Salman et al. 2020) a certified defense that splits this one-step process into two. After randomly perturbing input, the defense first applies denoiser model that aims to remove the added noise, followed by standard classifier that guesses a label given this noised- then-denoised input. This enables applying randomized smoothing to pretrained black-box base classifiers, as long as the denoiser can produce clean images close to the base classifier's original training distribution.
We observe that the recent line of work on denoising diffusion probabilistic models "Sohl-Dickseee al., 2015 Ho et al., 2020 Nichol Dhariwal, 2021)-whie which achieve state-of-the-art results on image generation are perfect match for the denoising step in denoised smoothing defense. A forward diffusion process takes a source data distribution (e.g., images from some data distribution) and then adds Gaussian noise until the distribution converges to high-variance isotropic Gaussian. Denoising diffusion models are trained to invert this process. Thus, we can use a diffusion model as denoiser that recovers high quality denoised inputs from inputs perturbed with Gaussian noise.
In this paper. we combine state-of-the-ard. publicly available diffiusion models as denoisers with standard pretrained state-of-the-art classifiers. We show that the resulting denoised smoothing defense obtains significantly better certified robustness results-for perturbations of & norm of S 2 ImageNet and c < 0.5 CIFAR-10 compared to the "custom" denoisers trained in prior work (Salman el al. (2020). or in fact with any certifiably robust defense (even those that do rely on denoised smoothing). Code to reproduce our experiments is available https: http://gttttbbbb com/ettomtttoo www.eettt-iiiiieee wttp:/iffffssiooooooo ftntr//afafuuiooooooooosseeeeei
"Joint first authors

---

Published as a conference paper at ICLR 2023
2 BACKGROUND
Adversarial examples (Biggio et al. 2013 Szegedy et al. 2014) are inputs x'' = x constructed by taking some input æ (with true label )) and adding a perturbation (that is assumed to be imperceptible and hence label-preserving) so that given classifier f misclassifies the perturbed input, i.e., (x 5) # y. The "smallness" of á is quantified by its Euclidean norm, and we constrain ||6|| < E. Even when considering exceptionally small perturbation budgets (c.g., 0.5) modern classifiers often have near-0% accuracy (Carlini & Wagner (017).
Randomized smoothing (Lecuyer et al., 2019: Cohen et al. 2019) is a technique to certify the robustness of arbitrary classifiers against adversarial examples under the l norm. Given input and base classifier f, randomized smoothing considers smooth version of f defined as:
g(x) == argmax Pr (f(x + 5) =c)
(1)
80A(0,02
Cohen et al. 2019) prove that the smooth classifier g is robust perturbations of l2 radius R, where the radius It grows with the classifier's "margin" (i.e., the difference in probabilities assigned to the most likely and second most-likely classes). As the probability in Equation [ cannot be efficiently computed when the base classifier f is neural network, Cohen et al. (2019) instantiate this defense by sampling a small number m of noise instances (e.g., the outputs of the base classifier f on m noisy versions of the input. To compute lower-bound on this defense's 's robust radius R. they estimate the probabilities Pr[f(x + 8) for each class label by sampling a large number N of noise instances ō (e.g.. N 100,000). See Cohen et al. (2019) for details.
I0) and taking majority vote over
Denoised smoothing (Salman et al. 2020 an instantiation of randomized smoothing, where the base classifier f is composed of denoiser denoise followed by standard classifier feie:
f(r (x+ fer(denoise( + +
Given a very good denoiser (i.e., denoise(z + 6) = x with high probability for 5 ~~ N (0, oT)),, we can expect the base classifier's accuracy on noisy images be similar the clean accuracy of the standard classifier ferr- [Salman et al. (2020) instantiate their denoised smoothing technique by training custom denoiser models with Gaussian noise augmentation, combined with off-the-shelff pretrained classifiers.
Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015 Ho et al., 2020 Nichol
Dhariwal (02T) are a form of generative model that work by learning model that can reverse time diffusion process of the form 24 v1 - B3 -t-- 3r we, we ~ N (0. I) with xop coming from the data distribution, and the 3r being fixed (or Icarned) variance parameters. The diffusion process transforms images from the target data distribution to purely random noise over time. The reverse process then synthesizes images from the data distribution starting with random Gaussian noise. In this paper will not make of diffusion models in the typical way; instead it suffices to understand just one single property about how they are trained.
Given clean training image [-1, 10m-bre,, diffusion model selects timestep N+ from some fixed schedule and then samples a noisy image Fe of the form
Ft
Vou x + -I-vill a4 IO,
(3)
where the factor aa is a constant derived from the timestamp that determines the amount of noise to be added to the image (the noise magnitude increases monotonically with t).
The diffusion model is then trained (loosely speaking) to minimize the discrepancy between and denoi se t); that is, to predict what the original (un-noised) image should look like after ap- plying the noising step at timestep t
SSaac--ffthe-an diffusion models actually trained to predict the noise rather than the denoised image
directly (Ho 2020 Nichol & Dhariwal 2021
2

---

Published as a conference paper at ICLR 2023
Algorithm Noise. denoise, classify
Algorithm 2 Randomized smoothing (Cohen et al. 2019)
1: NDISEANDCIASSTFYGG
1: PREDICI(I, N,
2: ar + G
counts +-
3: Ir- - (=r
3:
for iorie , N } do
4: f d=:
4:
y No NONTTEAAECCSSSSTY(....... counts [v] counts(y] +
5: - fa)(
5:
6: return y
6:
9A. ÚB t top two labels counts
7:
7:
AA, Bg + counts(aa countsjjp
8: GRTTTMESTEP(O)
if TIINOMPTEST
nB, 1/2) S n) then
9: t' +- find f s.t. I-a = 0²
9: return UA
10: return 1", Orp
10: else
11: return Abstain
Figure 1: Our approach can be implemented in under 15 lines of code, given off-the-shelf clas- sifier fer and an off-the-shelf diffusion model denoise. The PREDICT function is adapted from Cohen et al. (2019) and takes input number of noise samples N and statistical significance level 17 E (0, 1) and inherits the same robustness certificate proved in Cohen et al. (2019).
3 DIFFUSION DENOISED SMOOTHING
Our approach, Diffusion Denoised Smoothing (DDS), requires no new technical ideas on top of what was introduced in the section above.
Denoised smoothing via diffusion model. The only minor technicality required for method to map between the noise model required by randomized smoothing and the noise model used within diffusion models. Specifically, randomized smoothing requires data point augmented with additive Gaussian noise Trs ~ (xx,, IT),, whereas diffusion models assume the noise model 2; ~ N Vo,z, 0x I).. Scaling Ti yar and equating the variances yields the relationship
ag
(4)
at
Thus, in order to employ diffusion model for randomized smoothing at given noise level first find the timestep t* such that o² = 1-ax; the precise formula for this equation will depend on the schedule of the a terms used by the diffusion model, but this can typically be computed in closed form, even for reasonably complex diffusion schedules Next, we compute
Zz- aa + 8), 5 o0)
and apply the diffusion denoiser on to obtain an estimate of the denoised sample denoise((r::"))
And finally, we classify the estimated denoised image with an off-the-shelf classifier y = far(
(8)
The entirety of this algorithmic approach is shown in Figure
FFor example, in Nichol & Dhariwal
2021 the authors advocate for the schedule ar 7 for various values of T, and discussed in this reference. In
f(1/f(0), where f(1) cos 4/T+=
this case, for given desired value of o², some algebra yields the solution for
2(1 + s) esc (Vi + o² csc
2+
==
(5)
The actual formula here is unimportant and only shown as an illustration of how such computation can look in practice. Even when such closed form solution does not exist, because the schedules for a; monotonic decreasing, always find solution via ID root-finding methods necessary.
3

---

Published as a conference paper at ICLR 2023
To obtain a robustness certificate, we repeat the above denoising process many times (e.g.. 100,000) and compute the certification radius using the approach of Cohen et al. (2019) (note that since our diffusion models expects inputs in [-1, 1]4, we then divide the certified radius by 2 to obtain certified radius for inputs in [0, 1] as assumed in all prior work).
One-shot demoising. Readers familiar with diffusion models may recall that the standard process repeatedly applies "single-step" denoising operation rt-1 = d(zy; t) that aims to convert a noisy image some timestep 1 to a (slightly less) noisy image the previous timestep L- 1. The full diffusion process would then be defined by the following iterative procedure:
6; = d(d(... .. d(d(æ 6; f); 1). :....; ;);
In fact, each application of the one-step denoiser d consists of two steps: (1) an estimation of the fully denoised image from the current timestep i, and (2) computing (properly weighted, according to the diffusion model) average between this estimated denoised image and the noisy image at the 1. Thus, instead of performing the entire t-step diffusion process to denoise an image. it is also possible to run the diffusion step d once and simply output the best estimate for the denoised image x in one shot.
previous timestep
When diffusion model generates images from scratch (i.e., the denoiser is applied to pure noise), the iterative process gives higher fidelity outputs than this one-shot approach (Ho al.) 2222)). But here, where we aim to denoise one particular image, a one-shot approach has two advantages:
1. High accuracy: it turns that standard pretrained classifiers more accurate on one- shot denoised images compared to images denoised with the full t-steps of denoising, We hypothesize this is due the fact that when we first apply the single-step denoiser d at timestep f, the denoiser already has all the available information about æ. By applying the denoiser multiple times, we can only destroy information about x as each step adds new (slightly smaller) Gaussian noise. In fact, by using the iterative t-step denoising strategy, we are in essence pushing part of the classification task onto the denoiser, in order to decide how to fill in the image. Section 5 Jexperimentally validates this hypothesis.
2. Improved efficiency: instead of requiring several hundred (or thousand) forward passes to denoise any given image, we only require one single pass. This is especially important when we perform many thousand predictions as is required for randomized smoothing to obtain robustness certificate.
Related work. We are not the first to observe a connection between randomized smoothing and diffusion models. The work of Leel 2021) first studied this gobblem-however they do not obtain significant accuracy improvements, likely due to the fact that diffusion models available at the time that work was done were not good enough. Separately, NNie al. (2022) suggest that diffusion models might be able to provide strong empirical robustness to adversarial examples, as evaluated by robustness under adversarial attacks computed using existing attack algorithms; this is orthogonal to our results.
4 EVALUATION
We evaluate diffusion denoised smoothing on two standard datasets, CIFAR-10 and ImageNet, and find it gives state-of-the-art certified & robustness on both. On CIFAR-10, we draw N = 100,000 noise samples and on ImageNet we draw N 10,000 samples to certify the robustness following Cohen et al. (2019).
As is standard in prior work, we perform randomized smoothing for three different noise magnitudes, E (0.25, 0.5, 1.0]. For a fair comparison to prior work in Table 2 and Table I we give the best results reported each paper across these same three noise magnitudes. Note that prior work only uses three levels of noise due to the computational overhead; one benefit of using diffusion model is we could have used other amounts of noise without training a new denoiser model.
CIFAR-10 configuration. We denoise CIFAR-10 images with the 50M-parameter diffusion model from Nichol & Dhariwal 2021 The denoised images are classified with a 87M-parameter
https: thub. wwtp:/ccoo/ppaaaa /wmp//imp-vveeee ffusion
4

---

Published as a conference paper at ICLR 2023
Certified Accuracy at (%)
Method
Off Off-the-shelf
Extra data
0.5
1.0
1.5
2.0
3.0
PixeIDP Lecuyer el al.
2019
X
16.0
RS (Cohen al. 2019
X
(07 49.0
(7.00)37.0
(07.0)29.0
19.0
12.00)
SmoothAdy Salman
2019
X
(65 (65-0)
43.00)43.0
(64.0)37.0
(10-0)27.0
(10 (10.0:20.0
Consistency Jeong Shin
2020
X 50.0
44.0
(65.0) 34.0
(41.0)24.0
(4.00)
MACER Zhai et 2020
X
(68 57.0
(64.0) 43..0
(1..0)31.0
(46.0)25.0
(18 01 14.0
Boosting (Horväth ct al]
2022n
X (6.0
(87.0) 44.6
38.4
(3..6/21.2 28.6
DRT Yang et al. 2021
X 46..8
(86.2)44.4
(49,
(0,4
(40 (3,4
SmoothMix ( cone a.
2021
X (65.0) 50,0
(5.00)43.0
(65.0) 38.0
(40.0)26.0
(40.0)20.0
ACES
(orvaäh al.
2022b
X
(4.0
(37.2)
(5.66)3566
(39.9) (5..
19,8
Denoised Sallan et al
2020
X (3000)
(35.0)
14.0
(38.0)
Lee0 [T.e 2121
X
41.0
24.0
11.0
Ours
(82.8)71.1 (77.1)
54.3
(77.1)
38.1
(00.0)
29.5
(00 (00.0) 13.1
Table I: ImageNet certified top-1 accuracy for prior defenses on randomized smoothing and de- noised smoothing. Randomized smoothing techniques rely on pprpise- models (indicated by a empty circle). The work Horváth et al. (202bb) is an exception in that it selectively applies ei- ther a robust or accurate off-the- fff-the-shelf classifier (indicated by half full circle). Denoised smoothing (Sllan et al.) (020) use an off-the-shelf classifier but train their own denoiser (indicated by half full circle). Our base approach uses an off-the-shelf classifier and off-the-shelf denoiser (indicated by full circle). Each entry lists the certified accuracy, with the clean accuracy for that model in parentheses, using numbers taken from respective papers.
VīT-B/16 model (Dosovitskiy et al. (222)) that was pretrained on ImageNet-21k (Deng et al. 2009) (in 224 224 resolution) and finetuned CIFAR-I0. We use the implementation from Hugging- Face 44 which reaches 97.9% test accuracy on CIFAR-10. 10. In addition, we also report results with standard 36M parameter CIFAR-10 to 95.2% accuracy.
10 model (Zagoruyko Komodakis 2016) trained on
As is typical, we report results with images normalized to [o, 1]32x32x3 We obtain a throughput of 825 images per second through the diffusion model and ViT classifier on an A100 GPU at batch size of 1,000. We report robust accuracy results averaged over the entire CIFAR-10 test set.
ImageNet configuration. We denoise ImageNet images with the 552M-parameter class- unconditional diffusion model from Dhariwal & Nichol (021), and classify images with the 305M- parameter BEiT large model (Bao et al.) 2022) which reaches a 88.6% top-1 validation accuracy using the implementation from timm (Wightman] 2019). We report results for our images when normalized to [0, |221x224x3 to allow us to compare to prior work. The overall latency of this joint srooise-then-classee model is 1.5 seconds per image on an A100 GPU at a batch size of 32. We report results averaged over 1,000 images randomly selected from the ImageNet test set.
4.1 RESULTS
On both CIFAR-I0 and ImageNet we outperform the state-of-the-art denoised smoothing ap- proaches (i.e.. Salman et al. (2020) and Lee (202T) in every setting; see Table [ and Table 2 as well as Figure for detailed results. Perhaps even more impressively, we also outperform models trained with randomized smoothing at low E distortions (e S 0.5 on CIFAR-10, 10, and < 2 on Ima- geNet), and nearly match them at high e. Even though these randomized smoothing techniques train their models end-to-end and specifically design these models to have high accuracy on Gaussian noise, we find that our approach's use of off-the-shelf models yields superior robustness (and much higher clean accuracy an added bonus).
Interestingly, we find that using a diffusion model to perform the denoising step gives its most significant benefits when and are small: for example, while we reach 71.1% top-1 accuracy at
e = =05 on ImageNet, an improvement over prior work of +14 percentage points, when we reach 3 our scheme is percentage points worse than of Our hypothesis for this effect,
http:::/huuggingffcccccoo/
pactee-ppchhhhhhh 224- ci
5

---

Published as a conference paper at ICLR 2023
Certified Accuracy at (%)
Method
Off-the-shelf
Extra data
0.25
0.5
0.75
1.0
PixeIDP [Lecuyer ct al.
2019
X
(71.0)22.0
(44.0)2.0
RS (Cohen al.
2019)
X
61.0
(33.0)43.0
32.00
32.0
(26.0)
SmoothAdv (Salman ct al.
2019
X
(75.6) (7..
(75.6)57.6
(74.8)
(87.4)38.3
(57.4)
SmoothAdv (Salman et al.
2019
V
(4.93)7499
(80.1163.4
(8.91)
39.6
Consisteney Joong Shin
20200
X
6878
(75.8)58.1
(725
(52-3)
MACER Zhai et al. 2020
O
X
(819)
5910159.0
(66.0) 46.0
(06.4)3800
Boosting (Hovväth al.
2022a
X
70.6
(7..8)60.4
(71-6) 52.4
38.88
DRT Yang al. 2021
(81.5)
60.2
(71.9) 50.5
(56.1)
SmoothMix UJong ct a
2021
X
(77.1)
(74.2) (77.1)
(61.8)37.2
ACES Horväth et al. 20226
X
(7B.0) 69.0
(7.22)
(74:2) 0
(68.6) 37.8
Denoised (Salman et al.
2020
X
(72.0) 56.0
(62.0)41.0
(82.0) 28.0
(44.0) 19.0
1.ce 1lce 202T
X
60.0
42.0
28.0
19.0
Ours
(88.1)
(88. 63.00
(88. (88.1)45.3
(77.0) 32.1
Ours (+finctuning)
(9,3
(5.5
(87.3)
3555
Table 2: CIFAR-10 certified accuracy for prior defenses from the literature. The columns have the same meaning as in Table |
0,25
0.8
0.25
0.8
0.5
Certified accuracy
Certified accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3
Radius (f2)
Radius (f2)
(a) CIFAR-10
(b) ImageNet
Figure 2: Certified accuracy as a function of the & adversarial perturbation bound, when varying levels of Gaussian noise {0.25, 0.5, 1.0). Bounds are computed with 100,000 samples per run on CIFAR-10, and 10,000 on ImageNet.
which we explore further in Section 5 is that diffusion models are prone to "hallucinate" content when denoising extremely noisy images. Thus, instead of reinforeing the signal from the correct class, the diffusion model generates a signal from another class, thereby fooling the classifier.
CIFAR-10 ablation. The off-the-shelff classifiers use were pretrained on larger datasets than respectively CIFAR-10 and ImageNet. It is well known that the use of additional data can boost robustness, both for empirical (Schmidt al. 2018) and certified (Salman al., 2019) defenses. To investigate the role played by the pretrained model, we repeat our CIFAR-I 10 experiment using standard
10 model KZagoruyko & Komodakis 2016 trained solely on CIFAR-
10 to 95.2% accuracy. The results with this classifier (see Table 6a outperform prior denoised smoothing approaches, and are competitive with prior randomized smoothing results up s = 0.5.
The VIT classifier outperforms the ResNet because it is more robust to the distribution shift in- alleviate this, we can further finetune the
troduced by the noising-and- and-denoiiin procedure. 5) from the CIFAR-10 training set. This defense is
classifier denoised images denoise thus not strictly loff-the-shed anymore (although finetuning is negligible compared to the training time of the diffusion model and classifier). Table 6b shows that finetuned Mide-ResNet achieves results than non- finetuned VIT. Thus, with a minimal amount of training, we also surpass prior randomized smoothing results without relying any external data. If finetune
comparable-on

---

Published as a conference paper at ICLR 2023
Certified Accuracy at (%)
Method
Off sif-the-sheff
Extra data
0.25
0.5
0.75
1.0
Wide-ResNet
(83.8) 70.6
55.7
(63.8)40.0
(65.8)
ViT
(88 76.7
(88.1)63.0
(88.1)45.3
(77.
Wide-ResNet +finetune
(86.9) 76.7
63.8
(85.9)49.5
(7.40)3.44
VIT +finetune
(91.2) 79.3
(91.2)
(91.
(5.5
Table 3: Summary of our ablation on CIFAR-10. The diffusion model and Wide-ResNet classifier are trained solely on CIFAR-10, while the ViT classifier is pretrained on larger dataset. The finetuning results are obtained by taking an off-the-shelf diffusion model and classifier, and tuning the classifier on noised- teesedtten--eee then-denoised images from CIFAR-10.
the VIT model (Table 6d), further improve defense's clean accuracy and certified robustness at 6 I 0.5 by a couple of percentage points. Our ablation is summarized in Table 3
ANALYSIS AND DISCUSSION
We achieve state-of of-the-art certified accuracy using diffusion models despite the fact that we are not using these models as diffusion models but rather trivial denoisers. That is, instead of leveraging the fact that diffusion models can iteratively refine images across range of noise levels, we simply apply the diffusion model once for a fixed noise level, to perform one-shot denoising.
In this section we study why this approach outperforms prior work that trained straightforward denoisiers for denoised smoothing (Salman et al. 2020), and why using diffusion models for one- shot denoising performs better than the more involved iterative diffusion process. Last we show promising results of multi-step diffusion using advanced deterministic sampler.
5.1 FULL DIFFUSION VERSUS ONE-SHIOT DENOISING
When used generative models, diffusion models perform denoising through an iterative process that repeatedly refines an estimate of the final denoised image. When given an image T4 with noise of magnitude corresponding some diffusion timestep the model first predicts one-shot estimate of the denoised image Fo. and then constructs an estimate Vt-1 of the noised image at timestep t-1 1 by interpolating (with appropriate weights) between xy, 2t and fresh isotropic Gaussian noise I). The diffusion process is then applied recursively at timestep t- 1
Intuitively, it may be expected that when using diffusion model as a denoiser, one-shot denoising will produce more faithful results than the full iterative roverse-diffusig process. Indeed, each step of the reverse-diffusion process destroys information about the original image, since each step adds fresh Gaussian noise to the image. Thus, information theoretically at least, it should be easier to denoise an image in one-shot than over multiple iterations.
label: paper towel
0=1.00
prediction: paper towel
prediction: printer
Ine
Figure 3: Intuitive examples for why multi-step denoised images are less recognized by the classifier. From left to right: clean images, noisy images with a = 1.0, one-step denoised images, multi-step denoised images. For the denoised images, show the prediction by the pretrained BEiT model.
7

---

Published as a conference paper at ICLR 2023
We find that this is indeed the case. While the full reverse-diffusion process produces denoised images with more finegrained details (which is a good property for generating photorealistic im- ages from scratch), these details often not actually faithful to the original image want to denoise. Instead, diffusion models are prone to "hallucinate" salient detailed features during the iterative denoise-and-noise process. We illustrate some examples of this hallucination phenomenon in Figure 3 Here, we noise an original image (on the left) with large Gaussian noise (a = 1) and then apply either the full reverse-diffusion process (rightmost image) or a one-shot denoising at the appropriate timestep (2nd image to the right). As we can see, one-shot denoising produces mostly faithful, but blurry, reconstructions of the original image, with finegrained details lost due to noise. In contrast, iterative denoising "invents" new details that result in images that are ultimately more photorealistic but semantically different from the starting image. Additional examples (with multiple random seeds) are in Figure and Figure S
in the Appendix.
5.2 TRAINING ON RESTRICTED NOISE LEVELS
Given that one-shot denoising performs better than full multi-shot denoising, we now turn to un- derstanding our next question: if we are just using diffusion models as one-shot denoisers, then why diffusion models perform better compared to the straightforward denoisers trained in prior work (Salman et al. 2020 To investigate this, train seven new diffusion models CIFAR-10 10 with varying levels of Gaussian noise- the way towards a model trained on a single noise level, i.e., straightforward denoiser.
Recall that during standard training of a diffusion model, we sample timestep T uniformly from some range, add noise according to this timestep, and then train the model to predict the noise that has been added. The only difference between this process and the standard denoised smoothing training process (Salman al. (2020) is the fact that here Gaussian noise simultaneously. Therefore we now perform a comparative analysis of models trained on more restrictive noise levels. We select seven different levels of noise:
training multiple levels of
Three models are trained exclusively on Gaussian noise of fixed standard deviation re- 0.5,
spectively 0.25, denoiser on noise of a fixed magnitude.
1.0. This is identical training "straightforward"
One model is trained all three noise levels at the same time.
Two models are trained on noise uniformly selected from E [0, 0.25), and a E [0, 1.0].
One model trained using the full range of noise, from o [0, for some S > (the exact value of S depends on the chosen noise schedule for the diffusion model).
We then evaluate the clean accuracy of off-the-shelf VIT model each image when denoised (in one shot) with each of these diffusion models, where the images are noised with a standard deviation of either a 0.25, 0.5, or o 1.0. The results are summarized in Table 4
Noise at evaluation
Training noise
0.25
0.5
σ
1.0
E {0.25}
79.0
16.2
9.8
(0.5)
14.5
60.1
15.4
E (1.0)
13.9
13.5
35.5
(0.25, 0.5, 1.0)
81.6
68.1
43.0
E [0, 0.25]
84.5
14.5
9.9
a [0, 1.0]
84.0
71.6
46.0
S > 1] (standard)
85.5
72.3
44.8
Table 4: Clean accuracy of an off-the-shelf VIT classifier on images denoised with a diffusion model trained restricted levels of Gaussian noise. Diffusion models trained on more diverse noise ranges yield higher accuracy on one-shot denoised images, even compared to models trained on the specific noise level used at evaluation time.
As expected, training a new model on any one individual noise level, and then using that model to denoise images at that noise level, gives high downstream accuracy: for example, training diffusion
8

---

Published as a conference paper at ICLR 2023
model using = 0.25 noise and then evaluating this same noise level gives 79% accuracy. However if we then try and use this model to denoise images at a different noise level- say o = 0.5-the accuracy of the classifier drops to just 16%. If we train the diffusion model directly on = 0.5 noise, we instead get a much better classification accuracy of 60.1%, but without good generalization to lower or higher noise levels. Similarly, training noise of 1.0 only gives good results when denoising images with the same noise level.
More surprisingly, however, is that training on all three noise levels simultaneously gives better accuracy for denoising images at each noise level, compared to a diffusion model trained specifically and solely for that noise level. For example, when denoising images with 0.5 Gaussian noise, we get a classification accuracy of 68.1% when the diffusion model is trained on that noise level and additional lower and higher noise levels a value 8% higher than the accuracy of 60. % we get when training the diffusion model solely on 0.5 noise.
If we train on more granular noise levels, either in [0, 0.25] or in the full interval [0, 1]. the classi- fication accuracy on denoised images at the three individual noise levels further increases by a few percentage points. Quite surprisingly, the standard training regime which trains the diffusion model on noise from larger range [0, S] for some S >> 1 further improves the denoising capabilities at low noise levels (a = 0.25 and σ = 0.5), but slightly harms the accuracy for larger noise (ar = 1.0).
From this experiment, we can conclude that the (full) training process of diffusion models leads to much better, and more generalizable, one-shot denoising capabilities than when training a standalone denoiser single noise level as in prior work.
5.3 ADVANCED DETERMINISTIC MULTI-STED SAMPLER
we found that the denoised images from full multi-step diffusion have a tendency to deviate from the original clean image. This could be due the stochastic nature of the full reverse- diffusion process, since at each step a random noise is added. We notice a line of work (Song et al.) 2021 Karras al., 2022) on fast deterministic sampling of diffusion models. We show that with such an advanced sampler, multi-step diffusion is able to beat one-shot denoising.
In section 5.1
We consider the deterministic EDM sampler proposed by Karras al. (2022). We compare the rec- ognizability of images denoised by EDM sampler and one-shot denoising. We adapt EDM sampler for image denoising by setting the maximum noise sigma of the sampling noise schedule be the noise level found by [Equation 4 We use the suggested sampler setting from Karras et al.) (2022) on CIFAR-10, 10, where 18 reverse steps with 35 evaluations of the diffusion model are performed for each example. The result is summarized in Table We can see that the deterministic EDM sampler superior over one-shot denoising.
Classifier
Method
0.25
0.5
1.0
One-shot denoising
81.3
64.0
35.8
Wide-ResNet
EDM sampler
85.0
73.0
53.8
One-shot denoising
84.9
71.6
50.8
VIT
EDM sampler
86.1
73.1
54.0
Table 5: Clean accuracy (average 5 runs) of off-the-shelf CIFAR-10 classifiers evaluated images denoised by one-shot denoising and EDM sampler (Karras et al. 2022).
CONCLUSION
At present, training certified adversarially robust deep learning models requires specialized tech- niques explicitly designed for the purpose of performing provably robust classification (Cohen et al.) 2019). While this has proven effective, these models are extremely difficult to train to high accuracy, and degrade clean accuracy significantly.
We suggest an alternative approach is possible. By exclusively making of off-the-shelf mod- els designed to be ofate-of-thh-an at classification and image denoising, we can leverage the vast resources dedicated to training highly capable models for the purpose of robust classification.

---

Published as a conference paper at ICLR 2023
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283. PMLR, 2018.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In Interna- tional Conference on Learning Representations, 2022.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. Joint European conference machine learning and knowledge discovery in databuses, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy, pp. 39-57. JEEE, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Ad- vances Neural Information Processing Systems, 34, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2021.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint
2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Miklós Z Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Boosting randomized smoothing with variance reduced classifiers. In International Conference Learning Represen- tations, 2022a.
Miklós Z Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Robust and accurate- compositional architectures for randomized smoothing, arXiv preprint 2rXiv:2204.0048: 2022b.
Jongheon Jeong and Jinwoo Shin. Consistency regularization for certified robustness of smoothed classifiers. Advances in Neural Information Processing Systems, 33:10558- 2020.
Jongheon Jeong, Sejun Park, Minkyu Kim, Heung-Chame Lee, Do-Guk Kim, and Jinwoo Shin. Smoothmix: Training
smoothed classifiers for certified robustness. Ad- vances Neural Information Processing Systems, 34:30153-30165 2021.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 2022.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy, pp. 656-677 672. IEEE, 2019.
Kyungmin Lee. Provable defense by denoised smoothing with learned score function. ICLR Work- shop on Security and Safety in Machine Learning Systems, 2021.
10

---

Published as a conference paper at ICLR 2023
Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for prov- ably robust neural networks. In International Conference on Machine Learning, pp. 3578-3586. PMLR, 2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Weili Nie, Brandon Guo, Yujia Huang. Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint (xXxi:2205.07666 2022.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances Neural Information Processing Systems, 32, 2019.
Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33:21945-21957, 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances Neural Information Processing Systems, 31, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaramathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pp. 5025-5034. PMLR, 2018.
Ross Wightman. Pytorch image models.
httpss///gthabb..om///.......
Sytorch-image--e imweeh--emmmeemmmeeeeeeme 2019.
Zhuolin Yang. Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robustness for ensemble models and beyond. arXiv preprint arXiv:2107. 108,3 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. MACER: Attack-free free and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2020.

---

Published as a conference paper at ICLR 2023
A
APPENDIX
Certified Accuracy at (%)
Certified Accuracy at E (%)
Noise
0.0
0.25
0.5
0.75
1.0
Noise
0.0
0.25
0.5
0.75
1.0
0.25
83.8
70.6
55.7
40.0
0.0
a 0.25
85.9
76.7
63.8
49.5
0.0
0.5
65.8
54.7
43.7
34.2
26.1
a 0.5
74.5
66.0
56.1
45.7
36.4
1.0
33.2
28.0
22.8
18.0
13.6
a 1.0
55.1
48.7
42.3
35.8
29.9
(a) Wide-ResNett
(b) Finetuned Wise-ResNet
Certified Accuracy at (%)
Certified Accuracy at e (%)
Noise
0.0
0.25
0.5
0.75
1.0
Noise
0.25
0.5
0.75
1.0
a 0.25
88.1
76.7
63.0
45.3
0.0
a 0.25
91.2
79.3
65.5
48.7
0.0
0.5
77.0
65.8
53.4
41.8
32.1
0.5
81.5
67.0
56.1
45.3
35.5
a 1.0
49.5
40.3
33.3
26.1
20.2
σ = 1.0
65.1
48.4
41.7
35.2
29.0
(c) VIT
(d) Finetumed ViT
Table 6: Certified accuracy of four different classifiers on CIFAR-10 at varying levels of Gaussian noise all using the same diffusion model.
Certified Accuracy at (%)
Noise
0.0
0.5
1.0
1.5
2.0
3.0
0.25
82.8
71.1
0.0
0.0
0.0
0.0
о = 0.5
77.1
67.8
54.3
38.1
0.0
0.0
1.0
60.0
50.0
42.0
35.5
29.5
13.1
Table 7: Certified accuracy on ImageNet for varying levels of Gaussian noise a.
12

---

Published as a conference paper at ICLR 2023
(a) One-shot denoised images (o = 1.00)
(b) Multi-step denoised images (a 1.00)
Figure 4: Qualitative comparison of one-shot denoising and multi-step denoising. We show denoised images under random Gaussian noise (a = 1.00). A green border is applied when the denoised images are correctly classified while red border means that the classifier misclassifies the image.
13

---

Published as a conference paper at ICLR 2023
label: bookstore
e=1.00
prediction: bookstore
prediction: grocery store
label: ocean liner
0=1.00
predicton: ocean Iner
prediction: vending machine
Figure 5: Additional intuitive examples for why multi-step denoised images are less recognized by the classifier. From left to right: clean images, noisy images with 1.0, one-step denoised im- ages, multi-stey denoised images. For the denoised images, we show the prediction by the pretrained BEiT model.
14


Reviews:
[{"review_id": "mszzlADHtA", "reviewer": "Reviewer_PB4j", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for certifying the adversarial robustness of off-the-shelf models with pre-trained Denoising Diffusion Probabilistic Model. The main idea is to use a denoiser after introducing noise for robustness certification and well-trained diffusion models can work well as denoisers. The noise introduced by the diffusion models can be used seen as the smoothing process and single-step denoising is then applied. Their main contribution is extending the denoising-based certification model to using a diffusion model. They take advantage of the recent advances in diffusion models and greatly improve the quality of denoising and certification. The experiments on ImageNet and CIFAR both show significant improvement in certification accuracy.", "summary_of_the_review": "I lean to accept this paper as it wins me over with a simple yet effective method. ", "strength_and_weaknesses": "Strength: \n- An elegant solution to a challenging problem. The intuition and method are both relatively simple, yet highly effective. Although it is not so novel as the idea of using a denoiser is already proposed, using a diffusion model is novel, and pre-trained diffusion models significantly improve denoising accuracy. \n- The one-step denoising is well-thought-of. It reduces computation costs by a great deal. Yet I'm curious why the certification accuracy is worse when the denoising result is more accurate.\n\nWeakness:\n- I hope to see more details on the hyperparameters for the diffusion process, including the noise scale during diffusion \\alpha_t, and the number of diffusion steps chosen during smoothing as it is related to the smoothing level. \n- I would like to see details on the backbone model for baseline models, as using ViT VS. ResNet could be very different in terms of clean accuracy on ImageNet.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4jS6epc6vs", "reviewer": "Reviewer_kRsg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper observes that a state-of-the-art Diffusion models could be a good denoiser applicable for Denoised smoothing when combined with pre-trained classifiers, specifically to construct an off-the-shelf smoothed classifier that offers robustness certificates. The paper establishes several practices for this pipeline which enables a significant improvements in performance compared to previous similar attempts, e.g., the use of one-shot denoising unlike standard practice, and a proper way to scale the given noisy image to make the pipeline in compliance to the certification protocol.", "summary_of_the_review": "Despite its lack of technical novelty, I think the empirical novelty of the observation and its practical significance could overweigh it thus for now I lean to accept for the paper. ", "strength_and_weaknesses": "**Strength**\n\n* The paper is easy-to-follow, and proposes a simple, easy-to-use method\n* The paper presents an extensive evaluation covering large-scale dataset such as ImageNet, as well as exploring diverse architectures such as ViT\n* The practice established in the paper could be useful in the future works given its significant improvements compared to previous similar attempts.\n\n\n**Weakness**\n\n* Although the paper claims a good practical performances by utilizing pre-trained models, but lacks on validating the effectiveness of the methodology itself: here, I could see the method can be also applied in the standard, non-off-the-shelf setup by training each diffusion model and classifier on the target dataset, e.g., CIFAR-10 from scratch for a fairer comparison - which I think a good ablation study to add. \n* The evaluation could also be strengthen by comparing Average Certified Radius (ACR) of the models which is often in the literature for a metric that considers both accuracy and robustness. \n* A discussion on the cost for certification (in other words, inference cost/overhead) compared to existing model would be helpful and worth to be added.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3xxHdgb8E2", "reviewer": "Reviewer_VkUH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on the framework of denoised randomized smoothing (Salman et al. (2020)), the authors applied larger diffusion models as the denoiser and observed significantly improved certified robustness (when the perturbation range is not too larger) compared to existing methods. ", "summary_of_the_review": "In general, the claims are well supported by the presented results, and both advantages and limitations are discussed. \n\nTo understand whether the improved certified accuracy comes from the data itself, I would like to see an ablation study that uses the one-shot denoised images from diffusion models as data inputs, and then apply them to standard randomized smoothing. If there are performance gains compared to unperturbed images, then this result will show the improvement is due to the fact that denoised images are more robust to Gaussian perturbations than clean images.", "strength_and_weaknesses": "Strength: The result shows the benefit of using the off-the-shelf diffusion model as the denoiser for improving certified robustness. Some limitations (e.g., fast degraded certified accuracy with increased perturbation levels) and some ablation studies in terms of one-step and multi-step denoising are discussed.\n\nWeakness: There is no new contribution to the methodology of denoised randomized smoothing. My rating is based on the reported numerical improvement and the discussion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-j1fgA2JLR", "reviewer": "Reviewer_Kvhz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper demonstrates that one can achieve state-of-the-art certified $l_2$-robustness using the existing method of denoised smoothing, by simply replacing the custom denoiser by an off-the-shelf diffusion model. ", "summary_of_the_review": "I believe this paper is well-written and has significant results, especially because of its simplicity. I argue that this paper should be accepted at ICLR, under some minor revisions that I describe in my main review.", "strength_and_weaknesses": "The method is technically very simple, and yet highly effective, which I view as a major strength. The result is significant because it is a \"scale is all you need\" type result for certified robustness, which has the potential to have a big impact on this subfield as a whole.\nThe authors also provide some additional experiments as ablations, which I believe to be valuable, i.e. showing that fine-tuning can help and demonstrating that a variety of noise levels improve the denoisers over more restricted noise levels.\nThe authors compare to many baselines which is good.\n\nHowever, one point of criticism is that the authors do not actually provide any results on full diffusion as opposed to one-shot denoising. They provide convincing intuitive explanations of why one would expect that one-shot denoising is better suited for this task, but the examples they provide might be quite cherry-picked. I suspect that the reason why they do not provide numbers for comparison is that, as they admit, it is far too expensive to run this many steps on every single forward pass (of which many thousands are needed for randomized smoothing). I think the authors need to either a) include numbers for full diffusion (preferred option) or b) weaken their language in Section 3 where they claim in the paragraph on \"High accuracy\" that \"Section 5 experimentally validates this hypothesis.\"\n\nAnother minor criticism is about the misleading title. I understand that this title is good for calling attention to this paper but I find \"for free\" to be very misleading, given that of course the accuracy vs. (certified) robustness tradeoff still persists and, crucially, that each verified sample still takes orders of magnitude more resources than a single prediction. A less snappy, though more accurate title would be much better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster