Multi-scale Diffusion Denoised Smoothing
Authors: Jongheon Jeong
Jinwoo Shin
Venue: NeurIPS 2023 poster (2023)
Keywords: adversarial robustness
certified robustness
randomized smoothing
denoised smoothing
diffusion models
TL;DR: Overcoming the robustness-accuracy trade-off by combining smoothed classifiers of different scales.

Abstract: Along with recent diffusion models, randomized smoothing has become one of a few tangible approaches that offers adversarial robustness to models at scale, e.g., those of large pre-trained models. Specifically, one can perform randomized smoothing on any classifier via a simple "denoise-and-classify" pipeline, so-called denoised smoothing, given that an accurate denoiser is available - such as diffusion model. In this paper, we present scalable methods to address the current trade-off between certified robustness and accuracy in denoised smoothing. Our key idea is to "selectively" apply smoothing among multiple noise scales, coined multi-scale smoothing, which can be efficiently implemented with a single diffusion model. This approach also suggests a new objective to compare the collective robustness of multi-scale smoothed classifiers, and questions which representation of diffusion model would maximize the objective. To address this, we propose to further fine-tune diffusion model (a) to perform consistent denoising whenever the original image is recoverable, but (b) to generate rather diverse outputs otherwise. Our experiments show that the proposed multi-scale smoothing scheme, combined with diffusion fine-tuning, not only allows strong certified robustness at high noise scales but also maintains accuracy close to non-smoothed classifiers. Code is available at https://github.com/jh-jeong/smoothing-multiscale.

Multi-scale Diffusion Denoised Smoothing
Jongheon Jeong Jinwoo Shin
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, South Korea
{ jongheonnj,
Abstract
Along with recent diffusion models, randomized smoothing has become of a few tangible approaches that offers adversarial robustness to models at scale, e-g., those of large pre-trained models. Specifically, one can perform randomized smoothing on any classifier via a simple flenoise-and-classity pipeline, so-called denoised smoothing, given that accurate denoiser is available such diffusion model. In this paper, we present scalable methods to address the current trade-off between certified robustness and accuracy in denoised smoothing. Our key idea is to "selectively" apply smoothing among multiple noise scales, coined multi-scale smoothing, which can be efficiently implemented with single diffusion model. This approach also suggests new objective to compare the collective robustness of multi-scale smoothed classifiers, and questions which representation of diffusion model would maximize the objective. To address this, we propose to further fine- tune diffusion model (a) to perform consistent denoising whenever the original image is recoverable, but (b) to generate rather diverse outputs otherwise. Our experiments show that the proposed multi-scale smoothing scheme, combined with diffusion fine-tuning, not only allows strong certified robustness at high noise scales but also maintains accuracy close to non non-smoothed classifiers. Code available at attps://gtt tutt..u..... cwmm haa//jjjjjenge jeong/ smoothing wwwhchiinngnniiiieeeeeieiieeeeie multiscale.
Introduction
Arguably, one of the important lessons in modern deep learning is the effectiveness of massive data and model scaling [38, 74], which enabled many breakthroughs in recent years [8. 53, 55, 7]. Even with the largest amount of data available on the web and computational budgets. however, the worst-case behaviors of deep learning models are still challenging regularize. For example, large language models often leak private information in training data [10], and one can completely fool superhuman-leww Go agents with few meaningless moves [40]. Such unintentional behaviors can be of critical concerns in deploying deep learning into real-world systems, and the risk has been increasing as the capability deep learning continues to expand.
In this context, adversarial robusiness [61, 9] has been a seemingly-close milestone towards reliable deep learning. Specifically, neural networks are even fragile small, "imperceptible" scales of noise when it comes to the corst-case, and consequently there have been many efforts to obtain neural networks that are robust to such noise [47, 3, 75, 15, 65]. Although is reasonable premise that we humans have an inherent mechanism to correct against adversarial examples [21, 26]. yet so far the threat still persists in the context of deep learning, e.g., even in recent large vision-language models [48), and is known as hard to avoid without significant reduction in model performance [66, 64].
Randomized smoothing [41, 15], the focus in this paper, currently one of few techniques that has been successful in obtaining adversarial robustness from neural networks. Specifically, it constructs a smoothed classifier by taking majority vote from "base" classifier, e-g., neural network, over random Gaussian input noise. The technique is notable for its provable guarantees on adversarial
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dardise-and-claae
1,0
Dio
fata
P0
D
"Ship"
Dos
fota
Phe > 301
1.0
"Bird"
Drc
Do.26
fora
Phaa
"Abstain"
Po
"Abstain"
(a) Cascaded smoothing
(b) Diffusion calibration
An overview of the proposed approaches, (a) cascaded randomized smoothing (Section 3.2 and (b) diffusion calibration (Section 3.3) to attain better trade-off between accuracy and certified robustness in randomized smoothing, upon on the recent diffiuiin denoised smoothing scheme [11].
Figure I:
robustness, ie., it is certified defense [44], and its scalability to arbitrary model architectures. For example, it was the first certified defense that could offer adversarial robustness ImageNet [56].
The scalablity randomized smoothing has further expanded by Salman al. [58], which observed that randomized smoothing
be applied to any pre-trae pre-trained classifiers by prepending denoiser model, dubbed denoised smoothing. Combined with the recent diffusion- bassed models [31], denoised smoothing could provide the current state-of-the oftee---tthe--ts in &g f2-cettiieed robustness [11, 69, 44].
Despite its desirable properties, randomized smoothing practice is also at odds with model accuracy [15), similarly to other defenses [47, 76]. which makes it burdening to be applied in the real-world. For example, variance of Gaussian noise, or smoothing factor, is currently crucial hyperparameter to increase certified robustness at the cost of accuracy. The fundamental trade-off between accuracy and adversarial robustness has been well-evidenced in the literature [66, 75, 64], but it has been relatively under-explored whether they may may not be applicable in the context of randomized smoothing. For example, Tramèr et [64] demonstrate that pursuing E-uniform adversarial robustness neural networks may increase their vulnerability against invariance attacks, i.e., some semantics-altering perturbations possible inside the e-ball: yet, is still unclear whether such a vulnerability would be inevitable at non-uniformly robust models such smoothed classifiers.
In attempt to understand the accuracy-robustness trade-off in randomized smoothing, efforts have been made to increase the certified robustness given smoothed classifier can provide, e.g., through better training method [58, 73, 35-37]. For example, Salman al. [57] have employed adversarial training [47] in the context of randomized smoothing, and Jeong and Shin [35] have proposed to train a classifier with consistency regularization. Motivated by the trade-off among different smoothing factors, some works have alternatively proposed to perform smoothing with input-dependent factors
[1, 13, 67]: unfortunately, subsequent works [60, 36] have later shown that such schemes do not provide valid robustness certificates, reflecting its brittleness in overcoming the trade-off.
Contribution. In this paper, we develop practical method overcome the current frontier of trade-off in randomized smoothing, particularly upon the architectural benefits that the recent diffusion denoised smoothing [11] can offer. Specifically, we first propose to aggregate multiple smoothed classifiers of different smoothing factors obtain their collective robustness (see Figure ((a),, which leverages the scale-free free nature of diffusion denoised smoothing. In this way, the model can decide which smoothed classifier use for each input, maintaining the overall accuracy. Next, we fine-tune a diffusion model for randomized smoothing through the "denoise-and and-classiff"" pipeline (see Figure 1(b)), as efficient alternative of the full fine-tuning of (potentially larger) pre- trained classifiers. Here, we identify the over-confieence of diffusion models, e.g., to a specific class given certain backgrounds, and design a regularization objective to mitigate the issue.
major challenge towards accurate accurate-yet-rooust randomized smoothing,
In our experiments, we evaluate our proposed schemes on CIFAR-10 [39] and ImageNet [56], two of standard benchmarks for certified (2-robustness, particularly considering practical scenarios of applying diffusion denoised smoothing to large pre-trained models such as CLIP [53]. Overall, the

---

results consistently highlight that (a) the proposed multi-scale smoothing scheme upon the recent diffusion denoised smoothing [11] can significantly improve the accuracy of smoothed inferences while maintaining their certified robustness at larger radii, and (b) our fine-tuning scheme of diffusion models can additively improve the results not only in certified robustness but also in accuracy - by simply replacing the denoiser model without any further adaptation. For example, we could improve certifications from a diffusion denoised smoothing based classifier by 30.6% + 72.5% in clean accuracy, while also improving its certified robustness at 2.0 by 12.6% 14.1%. %. We observe that the collective robustness from our proposed multi-scale smoothing does not always correspond to their individual certified robustness, which has been major evaluation in the literature, but rather to their "calibration" across models: which opens up a new direction to pursue for a practical use of randomized smoothing.
2 Preliminaries
Adversarial robustness and randomized smoothing. For a given classifier s X + + 3/ where E X C Rd and y E y (1,... ..). adversarial robustness refers to the behavior of ʃ in making consistent predictions at the worst-case perturbations under semantic-preserving restrictions.
Specifically, for samples from a data distribution (x Pasta (x, y). it requires f(x 6) y for every perturbation that a threat model defines, e.g., an l2-ball | S c. One of ways to quantify adversarial robustness is to measure the following average minimum-distanee of adversarial perturbations [9], i.e., R(f: Pdata) := -(x.y))PPdnn | f(x'+ I|x - -x|12]
The essential challenge in achieving adversarial robustness stems from that evaluating this (and further optimizing on it) is usually infeasible. Randomized smoothing [41, 15] bypasses this difficulty by constructing a new classifier f from f instead of letting f to directly model the robustness: specifically, it transforms the base classifier f with a certain smoothing measure, where in this paper we focus on the case of Gaussian distributions (0, I²1):
f(x) == arg max IP o [f(x + cc] oE)
Then, the robustness of at (x, y). namely R(f:x, y), can be lower-bounded in terms of the certified radius R(,, x. e.g., Cohen et al. [15] showed that the following bound holds, which is tight for fz-adversarial threat models:
R(f: x. v) 2 a. 4 (D)(x, v)) =: R(f. x. v)) where p,(X, y) = Pslf(x + 5) = yl.
(2)
provided that f(x) = y, otherwise R(J;; x, y) =- 0. Here, we remark that the formula for certified radius is essentially a function of p.. which is the accuracy of fx + 5) over 5.
Denoised smoothing. Essentially, randomized smoothing requires f to make accurate classification of daussian-corruppe inputs. A possible design of in this regard is to concatenate Gaussian denoiser, say denoise(-), with any standard classifier fata, so-called denoised smoothing [58]:
[xx 8) : fana (deoossis 5)).
Under this design, an ideal denoiser denoise(-) should "accurately" recover x from x + 6, i.e.. denoise(x + 8) x (in terms of their semanties perform classification) with high probability of ~~, I²)). Denoised smoothing offers a more scalable framework for randomized smoothing, considering that (a) standard classifiers (rather than those specialized to Gaussian noise) are nowadays easier to obtain in the paradigm of large pre-trained models, and (b) the recent developments in diffusion models [31] has supplied denoisers strong enough for the framework. In particular, Lee [42] has firstly explored the connection between diffusion models and randomized smoothing; Carlini et al. [11] has further observed that latest diffusion models combined with a pre-trained classifier provides a state-of-the-art design of randomized smoothing.
Diffusion models. In principle, diffusion models [59. 31] aims to generate a given data distribution Pasta (x) via an iterative denoising process from a Gaussian noise &T ~ (0, for a certain T > 0. Specifically, it first assumes the following diffusion process which maps Pdata to (0, I2):: dx, p(x,, t)dt+ or (f)dw,, where t E j0, T) and w; denotes the standard Brownian motion. Based
s denotes the cumulative distribution function of (0 12).
3

---

on this, diffusion models first train a score model sx(x,)) t) ~ V (x()) via score matching [34], and use the model to solve the probabilistic flow from &T ~ (0, T to Xo for sampling. The score estimator (x, t) is often parametrized by denoiser D(x; (t)) in practice, viz., V px(): (D(x; (()) -x)/o(t) which establishes its close relationship with denoised smoothing.
3 Method
Consider a classification task from X' to y where training data {(x,, y:)} Paata (x, y) is available, and let f X + y be a classifier. We denote fo to be the smoothed classifier of f with respect to the smoothing factor >> O, defined by (1). In this work, we aim to better understand trade-off of fo occurs, with a partieular consideration of the recent
how the uccuracy-robustten denoised smoothing scheme. Generally speaking, the trade-off implies the following: for a given model, there exists sample that the model gets "wrong" is optimized for (adversarial) robustness on another sample. In this respect, we start by taking a closer look at what it means by a model gets wrong, particularly when is from randomized smoothing.
3.1 Over-smoothing and over-confidence in randomized smoothing
Consider smoothed classifier fo. and suppose there exists a sample (x, y) where fo makes enror: ie., fo (x) + y. Our intuition here is to separate possible scenarios of (x) making an error into two distinct cases, based the prediction confidence of (x).. Specifically, define the confidence of a smoothed classifier fa based on the definition of randomized smoothing (1) and (2):
Pf. (x) == max Pj. (x. max Po 8) =y]-
(4)
Intuitively, this notion of "smoothed" confidence measures how consistent the base classifier f is in classifying x + 6 over Gaussian noise 5 ~ (00 o²T). In cases when f is modeled by denoised smoothing, achieving high confidence requires the denoiser to accurately "bounce- back"" a given noisy image x + 5 into one that falls into class with high probability over 5.
Given a smoothed confidence p- :- -Pj. (x). we propose to distinguish two cases of model errors, which are both peculiar randomized smoothing. by considering a certain threshold Po p. Namely, we interpret an error of fo at x either as (a) p S Po: the model is over-smoothing or (b) p > Po the model is having an over-confidence on the input:
Over-smoothing (p S. Po): On one hand, it is unavoidable that the mutual information [xx + 8: y) between input and its class label absolutely decrease from smoothing with larger variance o², although using larger can increase the maximum certifiable radius of so in practice (2). Here, a "weel-calibrated" smoothed classifier should output a prediction close the uniform distribution across y, leading to low prediction confidence. In terms of denoised smoothing, the expectation clearer: x + gets closer to the pure Gaussian noise, the denoiser D should generate more diverse outputs hence in y as well. Essentially, this corresponds to an accuracy-robustne trade-off from choosing a specific o for a given (e.g., coormatiio-theoreetie capacity of data.
2. Over-confidence (p> po): On the other hand, it is also possible for model fo to be incorrect but with a high confidence. Compared to the werrshhothing case, this scenario rather signals a "miscalibration" and corresponds to trade-off from model biases: even across smoothed models with a fixed a, the balance between accuracy and robustness can be different depending on how each model assigns robustness in its decision boundary per-sample basis. When viewed in terms of denoised smoothing, this occurrence can reveal an implicit bias of the denoiser function D, e-g., that of diffusion models. For example, a denoiser might be trained adapt some spurious cues in training data, e.g., their backgrounds, as also illustrated in Figure ((b).
In the subsequent sections, Section 3.2 and 3.3, we introduce two methods to exhibit better accuracy- robustness trade-off in randomized smoothing, each of which focuses on the individual scenarios of over-smoothing and over-confidence, respectively. Specifically, Section 3.2 proposes to use cascaded inference of multiple smoothed classifiers across different smoothing factors mitigate the limit of using single smoothing factor. Next, in Section 3.3, we propose to calibrate diffusion models to reduce its over-confidence particularly in denoised smoothing.

---

3.2 Cascaded randomized smoothing
To overcome the trade-off between accuracy and certified robustness from over- smoothio i.e., from choosing specific o, we propose to combine multiple smoothed classifiers with different o's s. In nutshell, we design a pipeline of smoothed inferences that each input (possibly with different noise resilience) can adaptively select which model to use for its prediction. Here, the primary challenge is to make "correct", so that the proposed pipeline does not break the existing statistical guarantees on certified robustness that each smoothed classifier makes.
Specifically, we now assume K distinct smoothing faccoor, say 0 01 < < ok, and their corresponding smoothed classifiers of f. namely fo..". ... fox- For a given imput x, our desiderata is (a) to maximize robustness certification at x based on the smoothed inferences available from the individual models, say Pj., (x); ................................ : iPi (x), while (b) minimizing the access to each of the models those require a separate Monte Calro integration in practice. In these respects, we propose simple "predict-or-abee -b-attii--- policy, coined cascaded randomized smoothing:
fxx(x)
if P (x) > po.
casc(x; (Po, 1 1= = casc(x: 1KT)) if Pfa (x) S Po and K > 1,
(5)
ABSTAIN
otherwise,
where ABSTAIN & Y' denotes an artificial class to indicate "undecidable". Intuitively, the pipeline starts from computing fox (x), the model with highest σ, but takes its output only if its (smoothed) confidence f/, (x) exceeds a certain threshold po:² otherwise, it tries a smaller noise scale, say ØK-1 and so on, applying the same abstention policy of P,z- (x) S Po- In this way, it can early-stop the computation higher if it is confident enough, so can maintain higher certified robustness. while avoiding unnecessary accesses to other models of smaller ø.
Next, we ask whether this pipeline can indeed provide robustness certification: i.e., how much one can ensure casc(x + 5) = casc(x) in its neighborhood 6. Theorem 3.1 below shows that one can indeed enjoy the most certified radius from for where casc(x) =: ÿ halts, as long as the preceding models are either keep abstaining or output ý over ö:*
Theorem 3.1. Let fa..". Saxx X + y be smoothed classifiers with < 0r < ... < <o.- Suppose casc(x; {fa, ): =: y halts at for for some k. Consider any and Pi,. E [0, 1] that satisfy the following: (a) p S Pfay (x, .), and (b) p 2 plg (x,c) for ke > k and e E y. Then,
it holds that casc(x 8; MK) = à for any ||ō||| R. where:
1k
R min
Ok --1
min yzG
Oo x --1 (1-P
(6)
Overall, the proposed multi-scale smoothing scheme (and Theorem 3.1) raise the importance of "abstaining well" in randomized smoothing: if a smoothed classifier can perfectly detect and abstain its potential errors, one could overcome the trade-off between accuracy and certified robustness by joining more accurate model afterward. The option to abstain in randomized smoothing was originally adopted to make its statistical guarantees correct in practice. Here, we extend this usage to also rule out less-confident predictions for more conservative decision making. As discussed in Section 3, now the over-confidence becomes major challenge in this matter: e.g., such samples can potentially bypass the abstention policy of cascaded smoothing, which motivates our fine-tuning scheme presented in Section 3.3.
Certification. We implement our proposed cascaded smoothing to make "statistically consistent" predictions across different noise samples, considering certain significance level (e.g., a 0.001): in similar fashion as Cohen et al. [15]. Roughly speaking, for given input it makes predictions only when the (1 a) -onfidencc interval of (>(x) does not overlap with po upon i.i.d. noise samples (otherwise it abstains). The more details can be found in Appendix D.2.
We also discuss several other possible (and more sophisticated) designs in Appendix E. In our experiments, we simply use Po = 0.5 for all a's. See Appendix C.3 for an ablation study with po- 'The proof of Theorem 3.1 is provided in Appendix D.
5

---

3.3 Calibrating diffusion models through smoothing
Next, we move on to the over-confidence issue in randomized smoothing; viz., fo often makes errors with high confidence to wrong classes. We propose to fine-tune a given fo to make rather diverse outputs when it misclassifies, i.e., towards more "calibrated" fo. By doing so, we aim to cast the issue of over-confidence as that of over seerhhio which can be casier to handle in practice, e.g., by abstaining. In this work, we particularly focus fine-tuning only the denoiser model D in the context of denoised smoothing, i.e., f == fara o D for a standard classifier faea: this offers a more scalable approach to improve certified robustness of pre-trained models, given that fine-tuning the entire classifier model in denoised smoothing can be computationally prohibitive in practice.
Specifically, given base classifier :- fata D and training data D, we aim to fine-tune D to improve the certified robustness of for To this end, we leverage the confidence information of the backbone classifier fata fairly assuming it as an "oracle". which became somewhat reasonable given the recent off- aff-the-shelf models available and apply different losses depending on the confidence information per-sample basis. We propose two losses in this matter: (a) Brier loss for either correct or under-confident samples, and (b) anti- loss for incorrect, over-confident samples.
Brier loss. For a given training sample (x, 1), we adopt the Brier (or "squared") loss [6] to regularize the denoiser function D to promote the confidence of Sata [ata(D(x + 6)) towards y. which can be beneficial to increase the smoothed confidence Pf. (x) that impacts the certified robustness at x. Compared to the cross-entropy (or "log") loss, a more widely-used form in such purpose, we observe that the Brier loss can be favorable in such fine-tuning of D through feta, in sense that the loss is less prone to "over-optimize" the confidence at values closer to 1.
Here, an important detail is that we do not apply the regularization to incorrect-yet--onfiden samples: i.e., whenever feea( + 5)) + y and Pera (6) := max. F + 5)) > Po, where Feta is the soft prediction of Sata. This corresponds to the case when D(x + 8) rather outputs a "realistic" off-class sample, which will be handled by the anti-consistency loss we propose. Overall, we have:
Larier x. v) = Es[1 = y or (6) S po] Fta D(x 8)) -yyl
where we denote üs- [(x 5) and e, is the y-th unit vector in RD
Anti-consistency loss. On the other hand, the anti-consistency loss aims to detect whether the sample is over-confident, and penalizes it accordingly. The challenge here is that identifying over-confidence in a smoothed classifier requires checking for Pj. (x) > Po (4), which can be infeasible during training. We instead propose a simpler condition to this end. which only takes two independent Gaussian noise, say 51, 02 ~ AO0 A000,'11 I²I). Specifically, we identify (x, V) as cver-confident whenever (a) û == f(x+ and 8/2 == f(x+ + 52) match, while (b) they are incorrect, i.e., 인1 # y. Intuitively, such a case signals that the denoiser D is often making a complete flip to the semantics of x + 8 (see Figure ((b) for an example), which we aim to penalize. A care should be taken, however, considering the possibility that D(x + 5) indeed generates an in-distribution sample that falls into a different class: in this case, penalizing it may result in a decreased robustness of that sample. In these respects, our design of anti-consistency loss forces the two samples simply to have different predictions, by keeping at least one prediction as the original, while penalizing the counterpart. Denoting P1 == Feta( (xx + d1)) and P2 == Feta(D(x 82)), again apply the squared loss on Pi and Pa to implement the loss design, as the following:
(xc(x, y) == 1 ÿ2 and 인1 # v) (IlPı sg(p:)) ll22
(8)
where ag(+) denotes the stopping gradient operation.
Overall objective. Combining the two proposed losses, i.e., the Brier loss and anti anti-consistency loss, defines a new regularization objective to add upon any pre-training objective for the denoiser D:
L(D)= = Inanotzer + À (IBrier + a Tua)),
where X, > are hyperparameters. Here, a denotes the relative strength of T:cc over Tarser in its regularization." Remark that increasing a would give more penalty on over-confident samples, which would lead the model to make more abstentions: therefore, this results in increased accuracy particularly in cascaded smoothing (Section 3.2).
In experiments, we simply 1 unless otherwise specified.

---

10
Codee 1010
Cocrretal
Contre scouroo
Conticdd Scouteer
Cedfeed sosurwey
Coneldency
Carlnist (2023)
Coocuding (ours)
Cascading (ours)
Calibration (ours)
82
Radius
Radks
(a) CIFAR-10
(b) ImageNet
Figure Comparison of certi- fied test accuracy on CIFAR-10 at σ E {0.25, 0.50, 1.00].
Figure 2: Comparison of certified accuracy with Carlini et al. [1]] on the effectiveness of cascaded smoothing.
Table Comparison of (a) certified accuracy, (b) empricial accuracy, and (c) average certified radius (ACR) on CIFAR-I0. We set our result bold-faced whenever it achieves the best upon baselines.
CIFAR-1O, VTT-B/160224 224
Empirical accuracy (%)
Certified accuracy (%)
Training
C10
C10-C
C10.1
0.0
0.5
1.0
1.5
2.0
ACR
Gaussian [15]
92.7
88.3
76.7
87.0
55.3
0.479
Consistency [35]
89.5
84.7
67.6
84.9
60.7
0.517
0.25
Carlini et al. [11]
92.8
67.6
89.5
0.527
+ Calibration (ours)
94.5
89.4
69.2
90.3
61.9
0.535
Gaussian [15]
87.4
81.8
71.0
69.9
45.9
24.4
6,0
0.543
Consistency [35]
80.7
75.0
59.6
67.7
49.2
31.2
13.1
0.617
0.50
Carlini et al. [11]
88.8
84.9
64.3
75.3
50.6
31.0
14.3
0.646
Calibration (ours)
88.6
84.2
63.7
75.1
53.3
32.5
15.3
0.664
(0.25-0.50)
+ Cascading (ours)
92.5
89.1
78.5
85.1
53.3
32.9
14.9
0.680
Gaussian [1S]
75.6
72.4
68.3
42.2
27.9
16.9
9.0
4.0
0.426
Consistency [35]
65.8
61.3
53.6
43.1
31.1
22.3
14.6
8.7
0.513
1.00
Carlini et al. [11]
87.3
83,2
73.0
48.6
32.5
21.9
11.8
6.4
0.530
Calibration (ours)
83.3
79.3
60.9
52.6
36.7
24.9
16.1
8.8
0.577
(0.25-1.00 (.25-1.00)
+ Cascading (ours)
90.1
86.9
74.5
79.6
40.5
25.0
16.2
8.8
0.645
Experiments
We verify the effectiveness of our proposed schemes, (a) cascaded smoothing and (b) diffusion calibration, mainly on CIFAR-10 10 [39] and ImageNet [56]: two standard datasets for evaluation of certified f2-robustness. We provide the detailed experimental setups, egg., training, datasets, hyperparameters, computes, elc., in Appendix B.
Baselines. Our evaluation mainly compares with diffusion denoised smoothing [11], the current state-of-the-art methodology in randomized smoothing. We additionally compare with two other training baselines from the literature, by considering models with the same classifier architecture but without the denoising step of Carlini et al. [11]. Specifically, consider (a) Gaussian training [15], which trains a classifier with Gaussian augmentation; and (b) Consistency [35], which additionally Gaussian noise in training. We select the baselines
regularizes the variance of predictions assuming practical scenarios where the training cost of the classifier side is crucial: other existing methods for smoothed classifiers often require much more costs, e.g., 8 times over Gaussian [57, 73].
Setups. We follow Carlini et al. [11] for the choice diffusion models: specifically, we use the 50M-parameter 32 32 diffusion model from Nichol and Dhariwal [52] for CIFAR-10, and the 552M- parameter 256 x 256 unconditional model from Dhariwal and Nichol [18] for ImageNet. For the classifier side, we use ViT-B/16 [63] pre-trained via CLIP [53] throughout our experiments. For uses we fine-tune the model on each of CIFAR- 10 and ImageNet via FT-CLIP [19], resulting in classifiers that achieve 98.1 % and 85.2% in t accuracy CIFAR-10 10 and ImageNet, respectively. Following the prior works, we mainly consider a e (0.25, 0.50, 1.00) for smoothing in our experiments.
Evaluation metrics. We consider two popular metrics in the literature when evaluating certified robustness smoothed classifiers: (a) the approximate certified test accuracy at r: the fraction of the test set which CERTIFY [15] classifies correctly with the radius larger than without abstaining,

---

Table 2: Comparison of (a) certified accuracy, (b) empricial accuracy, and (c) average certified radius
(ACR) on ImageNet. We set our result bold-faced whenever achieves the best upon baselines.
ImageNet, ViT-B/16@224
Empirical accuracy (%)
Certified accuracy at (%)
Truining
IN-IK
IN-R
IN-A
0.0
0.5
1.0
1.5
2.0
ACR
Carlini et al. [11]
0.25
80.9
69.3
35.3
78.8
61.4
0.517
Carlini et al. [11]
0,50
79.5
67.5
32.3
65.8
52.2
38.6
23.6
0.703
Cascading (ours)
(0.25-0.50)
83,5
69.6
41.3
75.0
52.6
39.0
22.8
0.720
Calibration (ours)
(0.25-0.50)
83.8
69.8
41.7
76.6
54.6
39.8
23.0
0.743
Carlini et al. [[]]
1.00
77.2
64.3
32.0
30.6
25.8
20.6
17.0
12.6
0.538
Cascading (ours)
(0.25-1.00)
82.6
69.8
40.5
69.0
42.4
26.6
19.0
14.6
0.752
Calibration (ours)
(0.25-1.00)
83,2
69.5
40.8
72.5
44.0
27.5
19.9
14.1
0.775
and (b) the average certified radius (ACR) [73]: the average of certified radii on the test set D Dreat while assigning incorrect samples as 0: viz., ACR [Pamet Exx.)) D (C,(f,,,x) [xx)=]
where CR()) denotes the certified radius that CERTIFY returns. Throughout our experiments, we use
10, 000 noise samples to certify robustness for both CIFAR-10 and ImageNet. We follow [15] for the other hyperparameters to run CERTIFY, namely by no = 100, and a = 0.001. In addition to certified accuracy, we also compare the empirical accuracy of smoothed classifiers. Here, we define empirical accuracy by the fraction of test samples those are either (a) certifiably correct, or (b) abstained but correct in the clean classifier: which can be a natural alternative especially at denoised smoothing. For this comparison, we use n = 100 to evaluate empirical accuracy.
Unlike the evaluation of Carlini et al. [11]. however, do not compare cascaded smoothing with the envelop accuracy curve over multiple smoothed classifiers at a E {0.25, 0.50, 1.00]: although the envelope curve be a succinct proxy compare methods, it can be somewhat misleading and unfair to compare the curve directly with an individual smoothed classifier. This is because the curve does not really construct concrete classifier its own: it additionally assumes that each test sample has prior knowledge on the value of a E {0.25, 0.5, 1.0) to apply, which is itself challenging to infer. This is indeed what our proposal of cascaded smoothing addresses.
4.1 Results
Cascaded smoothing. Figure 2 visualizes the effect of cascaded smoothing we propose in the plots of certified accuracy, and the detailed results are summarized in Table I and as "+ Cascading". Overall, we observe that the certified robustness that cascaded smoothing offers can be highly desirable over the considered single-scale smoothed classifiers. Compared to the single-scale classifiers at highest a, e.g., 1.0 in Table 1 and 2, our cascaded classifiers across a E (0.25, 0.50, 1.00 absolutely improve the certified accuracy at all the range of & by incorporating more accurate predictions from classifiers, e.g., of lower E {0.25, 0.503. On the opposite side, e.g., compared to Carlini et al. [11] at σ = 0.25, the cascaded classificrs provide competitive certified clean accuracy (E = 0), e.g., 89.5% vs. 85.1% on CIFAR-10. while being capable of offering wider range of robustness certificates. Those considerations are indeed reflected quantitatively in terms of the improvements in ACRs. The existing gaps in the clean accuracy in principle due to the errors in higher-o classifiers: ie., a better calibration, to let them better abstain, could potentially reduce the gaps.
Diffusion calibration. Next, we evaluate the effectiveness of our proposed diffusion fine-tuning scheme: "+ Calibration" Table and 2 report the results on CIFAR- 10 and ImageNet, respectively, and Figure 3 plots the CIFAR-10 results for each of a (0.25, 0.50, 1.00). Overall, on both CIFAR- 10 and ImageNet, we observe that the proposed fine-tuning scheme could uniformly improve certified accuracy across the range considered, even including the clean accuracy. This confirms that the tuning could essentially improve the accuracy-robustnes trade-off rather than simply moving along itself. As provided in Table 3, we remark that simply pursuing only the Brier loss (7) may achieve better ACR in overall, but with a decreased accuracy: it is the role of the anti-consistency loss (8) to balance between the two, consequently to achieve a better trade-off afterwards.
Empirical accuracy. practical scenarios of adopting smoothed classifiers for inference, is up to users to decide how to deal with the "abstained" inputs. Here, we consider a possible candidate of simply outputting the standard prediction instead for such inputs: this way, the output could be noted as an "uncertified" prediction, while possibly being more accurate, e.g., for in mndiitribution
8

---

Table 3: Comparison of ACR and certified accuracy of cascaded smooth classifiers on CIFAR-10 over different training and ablations. We use o E {0.25, 0.5, 1.0) for cascaded smoothing. Bold and underline indicate the best and runner-up, respectively. Model reported in Table is marked as grey.
Cascaded, o c (0.25, 0.5, 1.0)
Certified accuracy at e (%)
Training
ACR
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Gaussian [15]
0.470
70.7
46.9
31.7
23.7
16.8
12.4
9.2
6.5
Consistency [35]
0.548
57.6
42.7
32.3
26.3
22.4
18.1
14.6
11.1
Carlini al. [I1]
0.579
80.5
52.8
37.4
28.0
21.7
16.8
11.8
8.5
Cross-entropy
0.605
77.9
52.8
39.6
29.1
22.2
18.7
13.5
9.5
BBrier loss
0.666
77.6
55.7
41.7
32.7
26.8
21.7
16.3
11.9
AAnti-consist. 1.0)
0.645
79.6
53.5
40.5
31.2
25.0
20.1
16.2
11.4
+
Anti-consist. (a 2.0)
0.625
80.1
54.6
39.2
30.6
24.0
18.5
14.5
9.9
Anti-consist. (a 1.0)
0,566
83.9
55.1
37.8
25.8
19.8
14,5
11.5
8.1
Table 4: Comparison of ACR and certified accuracy (%) CIFAR-10 across two fine-tuning schemes: diffit- sion fine-tuning ("Diff."; ours) and classifier fine-tuning ("Class.") ") [11].
Table 5:
Comparison of model error
rates CIFAR-10 decomposed into (a) over-smoothing (p S po) and (b) over- confidence (p po). also with ACR.
We use o 0.5 for this experiment.
Fine-tuning
Certified accuracy at (%)
Erar rates (St, 4)
Diff.
Class.
ACR
0.00
0.25
0,50
0.75
1.00
1.25
Model (a = =1.))
S pa
>> pa
ACR (1)
x
x
0.639
75.3
62.0
50.6
39.3
31.0
22,9
Carlini et al. [11]
43.8
7.6
0.498
/
x
0.664
75.1
64.1
53.3
42.1
32,5
25.0
Cascading
5.0
14.5
0.579
x
0.664
76.0
64.7
52.7
42.1
32.4
24.4
+ Antii-oonist.
4.7
11.4
0.566
0.673
75.2
64.7
53.5
43.8
32.5
25.8
+ Briier loss
16.8
0.645
inputs. Specifically, in Table 1 and 2, consider a fixed standard classifier of CLIP-finetued VÝT-B/16 on C1FAR-10 (or ImageNet), and compare the empirical accuracy of smoothed classifiers on CIFAR-10, -0-C [27] and - [54] (or ImageNet, -R [28], and -A [30]). Overall, the results show that our smoothed models could consistently outperform even in terms of empirical accuracy while maintaining high certified accuracy, i.e., they abstain only when necessary. We additionally observe in Appendix C.2 that the empirical accuracy of our smoothed model can be further improved by considering an "ensemble" with the clean elassifier: e-g., the ensemble improves the accuracy of our cascaded classifier (a E {0.25, 0.5]) on CIFAR-IO-C by 88.8% + 95.0%, even outperforming the accuracy of the standard classifier of 93.4%.
4.2 Ablation study
In Table 3, we compare our result with other training baselines as well as some ablations for a component-we wise analysis, particularly focusing their performance in cascaded smoothing across E (0.25, 0.50, 1.00) on CIFAR-10. Here, we highlight several remarks from the results, and provide the more detailed study, e.g., the effect of po (5), in Appendix C.3.
Cascading from other training. "Gaussian" and "Consistency" in Table 3 report the certified robustness of cascaded classifiers where each of single-scale models is individually trained by the method. Even while their (certified) clean accuracy of 0.25 models are competitive with those of denoising-based models, their collective accuracy significantly degraded after cascading, and interestingly the drop is much more significant on "Consistency"". although did provide more robustness at larger e. Essentially, for high clean accuracy in cascaded smoothing the individual classifiers should make consistent predictions although their confidence may differ: the results imply that individual training of classifiers, without shared denoiser, may break this consistency. This supports an architectural benefit of denoised smoothing for a use as cascaded smoothing.
Cross-entropy vs. Brier. "+ Cross Cross-entroph entropy" in Table 3 considers an ablation of the Brier loss (7) where the loss is replaced by the standard cross-entropy: although indeed improves ACR compared to Carlini et al. [11] and achieves the similar clean accuracy with "+ Brier loss", its gain in overall certified robustness significantly inferior to that of Brier loss as also reflected in the worse ACR. The superiority of the "squared" loss instead of the "log" loss suggests that it may not be necessary to optimize the confidence of individual denoised image toward value strictly close to which reasonable considering the severity of noise usually considered for randomized smoothing.

---

Anti-consistency loss. The results marked as "+ Anti-consist." in Table 3, on the other hand, validates the effectiveness of the anti-consittecyy loss (8). Compared "+ Brier loss", which is equivalent to the case when a = 0.0 in (9), we observe that adding anti-consistency results in slight decrease in ACR but with an increase in clean accuracy. The increased accuracy of cascaded smoothing indicates that the fine-tuning could successfully reduce over-confidence, and this tendency continues at larger a = 2.0. Even with the decrease in ACR over the Brier loss, the overall ACRs attained are still superior to others, confirming the effectiveness of our proposed loss in total (9).
Classifier fine-tuning. In Table 4, we compare our proposed diffusion fine-tuning (Section 3.3) with another possible scheme of classifier fine-tuning, of which the effectiveness has been shown by Carlini et al. [1]]. Overall, we observe that tinettnnig both classifiers and denoiser can bring their complementary effects to improve denoised smoothing in ACR, and the diffusion fine-tuning itself could obtain a comparable gain to the classifier fine-tuning. The (somewhat counter-intuitive) effectiveness of diffusion fine-tuning confirms that denoising process can be also biased as well as classifiers to make over-confident errors. We further remark two aspects where classifier fine-tuning can be less practical, especially when it comes with the cascaded smoothing pipeline we propose:
(a) To perform cascaded smoothing at multiple noise scales, classifier fine-tuning would require separate runs of training for optimal models per scale, also resulting in multiple different models load which can be less scalable in terms of memory efficiency.
(b) In a wider context of denoised smoothing, the classifier part is often assumed to be model scale, even covering cases when it is "black-box" model, e.g., public APIs. Classifier fine-tuning, in such cases, can become prohibitive or even impossible.
With respect to (a) and (b), denoiser fine-tuning we propose offers a more efficient inference archi- tecture: it can handle multiple noise scales jointly with a single diffusion model, while also being applicable to the extreme scenario when the classifier model is black-box.
Over-smoothing and over-confidence. As proposed in Section 3.1, errors from smoothed classifiers can be decomposed into two: ie., p S Po for over-smoothing, and p > Po over-confidence, respec- tively. In Table 5, we further report a detailed breakdown of the model errors on CIFAR-10, 10, assuming 1.0. Overall, we observe that over-smoothing can be the major source of errors especially at such a high noise level, and our proposed cascading dramatically reduces the error. Although we find cascading could increase over-confidence from accumulating errors through multiple inferences, our diffusion tnn-ttunig could alleviate the errors jointly reducing the mmothhin as well.
5 Conclusion and Discussion
Randomized smoothing has been traditionally viewed as a somewhat less-practical approach, perhaps due to its cost in inference time and impact on accuracy. In another perspective, to our knowledge, it currently one of a few existing approaches that is prominent in pursuing adversarial robusiness at scale, e.g., in paradigm where training cost scales faster than computing power. This work aims to make randomized smoothing more practical, particularly concerning on scalable scenarios of robustifying large pre-trained classifiers. We believe our proposals in this respect, i.e., cascaded smoothing and diffusion calibration, can be a useful step towards building safer Al-based systems.
Limitation. A practical downside of randomized smoothing is in its increased inference cost, mainly from the majority voting procedure per inference. Our method also essentially possesses this practical bottleneck, and the proposed multi-scale smoothing scheme may further increase the cost from taking multiple smoothed inferences. Yet, we note that randomized smoothing itself is equipped with many practical axes to reduce its inference cost by compensating with abstention: for example, one can reduce the number of noise samples, e.g., to n = 100. It would be an important future direction to explore practices for better trade-off between the inference cost and robustness of smoothed classifiers, which could eventually open up a feasible way to obtain adversarial robustness at scale.
Broader impact. Deploying deep learning based systems into the real-world, especially when they are of security-concentn [12, 72], poses risks for both companies and customers, and we researchers are responsible to make this technology more reliable through research towards A/ safery [2, 29]. Adversurial robustness, that we focus on in this work, is one of the central parts of this direction, but one should also recognize that adversarial robustness is still bare minimum requirement for reliable deep learning. The future research should also explore more diverse notions of Al Safety to establish realistic sense of security for practitioners, e.g., monitoring and alignment research, to name a few.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by Defense Acquisition Program Administration (DAPA) and Agency for Defense Development (ADD) (UD230017TD), and by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2019-0-00) Artificial Intelligence Graduate School Program (KAIST)). We are grateful to the Center for Al Safety (CAIS) for generously providing compute resources that supported significant portion of the experiments conducted in this work. We thank Kyuyoung Kim and Sihyun Yu for the proofreading of our manuscript, and Kyungmin Lee for the initial discussions. We also thank the anonymous reviewers for their valuable comments to improve our manuseript.
by
References
[1] Motasem Alfarra, Adel Bibi, Philip HS Torr, and Bernard Ghanem. Data dependent randomized smoothing. In Uncertainty in Artificial Intelligence, pages 64 74. PMLR, 2022.
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems Al safety. arXiv preprint Mxiiv:606.00500 2016.
[3] Anish Athalye. Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 274-283, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:// htttt::/pproceediggs. memiiinngsmrr.... mlr. clr.press/ v80 280/athalyee8a. html.
[4] Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In International Conference on Learning sepresensations, 2020. URL https: htttps://openrevier net/forum? id=SJxSDxrKDr.
[5] Carlo Bonferroni. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze, 8:3 62, 1936.
[6] Glenn W Brier. Verification of forecasts expressed terms of probability. Monthly weather review, 78(1):
1-3, 1950.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Danicl Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not as say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[8] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr.. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[9] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan Goodfellow, and Aleksander Madry. On evaluating adversarial robustness. arXiv preprint arXiv: wwt.::992.0075.2200 2019.
[10] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown. Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium, pages 1233-2650,,- 2021.
[11] Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and Zico Kolter. (Certified!!) adversarial robustness for free! In The Eleventh International Conference on Learning Representations, 2023. URL https::/
[12] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pncumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721-1730, 2015.
[13] Chen Chen, Kezhi Kong, Paihong Yu, Juan Luque, Tom Goldstein, and Furong Huang. Insta-RS: Instance- wise randomized smoothing for improved robustness and accuracy. arXiv preprint arXhv:2103.04455 2021.
[14] Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404 413, 1934.

---

[15] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learming Re- search, pages 1310-1320, Long Beach, Califormia, USA, 09-15 Jun 2019. PMLR. URL http: //proceedings mlr. wwwps/v97.... cttt///cccc99ccccccccccc
[16] Francesco Croce and Matthias Hein. Provable robustness against all adversarial 12- perturbations for p 1. In International Conference on Learning Representations, 2020.
[17] Francesco Croce, Maksym Andriushchenko, Vikash Schwag, Edoardo Dehenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hcin. RobustBench: a standardized adversarial robustness benchmark. arXiv preprint (xXi:2001000967700 2020.
[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Asso- ciates, Inc., 2021. URL https: /pppceee/n//ce////... neur ips. http://cpppp.e.eeeee/pppeeeeeeee fttess/paper/222//E1E2 Paper-pdf.
[19] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Gu Shuyang, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. CLIP itself is strong fine-tuner: Achieving 85.7 %% and 88.0% top-1 2022.
accuracy with ViT-B and ViT-L on ImageNet. arXiv preprint
[20] Francisco Eiras, Motasem Alfarra, M. Pawan Kumar, Philip H. S. Torr, Puneet K. Dokania, Bernard Ghanem, and Adel Bibi. ANCER: Anisotropic certification via sample-wise volume maximization, 2021.
[21] Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial examples that fool both computer vision and time-limited humans. Advances in Neural Information Processing Systems, 31, 2018.
[22] Mare Fischer, Maximilian Baader, and Martin Vechev. Certified defense image transformations via randomized smoothing. Advances Neural information processing systems, 33:8404 8417, 2020.
[23] Timon Gehr, Matthew Mirman, Dana Drachsler- Coohe,, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In IEEE Symposium on Security and Privacy, 2018.
[24] Icc A Goodman. On simultancous confidence intervals for multinomial proportions. Technometrics, 7(2):
247-254, 1965.
[25] Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ucsato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In /EEE/CVF International Conference on Computer Vision, pages 4842-4851, 2019.
[26] Chong Guo, Michael Lee, Guillaume Leclerc, Joel Dapello, Yug Rao, Aleksander Madry, and James Dicarlo. Adversarially trained neural representations are already as robust as biological neural representations. In International Conference on Machine Learning, pages 8072-8081. PMLR, 2022.
[27] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations, 2019. URL https:// openrevies
[28] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributioo generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, October 2021.
[29] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.15:.. 13916, 2021.
[30] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, June 2021.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
12

---

[32] Miklós Z. Horváth, Mark Niklas Mueller, Marc Fischer, and Martin Vechev. Boosting randomized smoothing with variance reduced classifiers. In International Conference on Learning Represenlations, 2022. URL
[33] Miklós Z. Horváth, Mark Niklas Müller, Mare Fischer, and Martin Vechev. Robust and accurate compositional architectures for randomized smoothing. In ICLR 2022 Workshop on Socially Responsible Machine Learning, 2022.
[34] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[35] Jongheon Jeong and Jinwao Shin. Consistency regularization for certified robustness of smoothed classifiers. Advances in Neural Information Processing Systems, 33: 10558-10570, 2020.
[36] Jongheon Jeong, Sejun Park, Minkyu Kim, Heung-Chang Chang Lee, Doguk Kim, and Jinwoo Shin. MixothMix: Training confidence-calbraate smoothed classifiers for certified robustness. In Advances in Neural Information Processing Systems, 2021. URL https://openreviii
EQMVED359.
[37] Jongheon Jeong, Seojin Kim, and Jinwoo Shin. Confidence-awaree training of smoothed classifiers for certified robustness. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 8005-8013, 2023, doi: 10. 16.11669aai.33770 25968. URI.htte: //ojsp:::///....// /aapa//aa///aaaaaaa...aaaaaaaaaa article/vies/2586.
aaai wwai.org//aa.a.aa......aa..a orgg/ndex. ht///iddx..ppp phpppppp//aa AAAT/
[38] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[39] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto, 2009.
[40] Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng- Yu Tsai, I-Chen Wu, and Cho-Jui Hsich. Are AlphaZero- like agents robust to adversarial perturbations? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: // //ppnreeviww.eeeeeeeeee meet/ooummiddyy__.... J1Za0CZv.
[41] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pages 656-672. IEFF, 2019.
[42] Kyungmin Lec. Provable defense by denoised smoothing with learned score function. In ICLR Workshop on Security and Safety in Machine Learning Systems, 2021.
[43] Alexander Levine and Soheil Feizi. Improved, deterministic smoothing for 4 certified robustness. In International Conference on Machine Learning, pages 6254-6264. PMI.R, 2021.
[44] Linyi Li, Xiangyu Qi. Tao Xie, and Bo Li. SoK: Certified robustness for deep neural networks. arXiv preprint arXiv:2009.044 1331,, 2020.
[45] Linyi Li, Jiawei Zhang. Tao Xie, and Li. Double sampling randomized smoothing. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13163-13208. PMLR, 17-23 Jul 2022. URL ststpp://ppoceedinnge mlr. greas/v162/1122a. html.
[46] Chizhou Liu, Yunzhen Feng, Ranran Wang, and Bin Dong. Enhancing certified robustness via smoothed weighted ensembling, 2020.
[47] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. hwt.:hhttp//peeeeeeee
[48] Chengzhi Mao, Scott Geng, Junfeng Yang. Xin Wang. and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In International Conference on Learning Representations, 2023. URL htttpps:/pppereviiie
[49] Matthew Mirman, Timon Gehr, and Martin Vechev. Differenniable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, volume 80, pages 3578-3586 10-15 15 Jul 2018. URL http:://tpoooeoo http:://proceedings. mlr, walr.ppees//ee v80 ht/800//mmmm mirmmn18b. html.
13

---

[50] Jeet Mohapatra, Ching- Yun Ko, Tsui-Wei Weng. Pu Chen, Sijia Liu, and Luca Daniel. Higher- order certification for randomized smoothing. Advances in Neural Information Processing Systems, 33: 4501-45111 2020.
[51] Mark Niklas Mueller, Mislav Balunovic, and Martin Vechev. Certify or predict: Boosting certified robustness with compositional architectures. In International Conference on Learning Representations, 2021. URL https://openneviee
[52] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR, 18-24
Jul 2021. URL https: ptattp://ppoceeddingg mlr. waa...pres//vveee http////////nnccooee html.................mmm...mm...
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8788 PMLR, 2021.
[54] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize CIFAR-10?, 2018.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffuusin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[56] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein. Alexander C. Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3)2111255) 2522 2015. 10.1007/s 223-015-081617)
doi:
[57] Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengehuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep leamingg via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems 32, pages 11289-11300 Curran Associates, Inc., 2019.
[58] Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33:21945-21957, 2020.
[59] Jascha Dickksiis,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2266 PMLR, 2015.
[60] Peter Súkeník, Aleksei Kuvshinov, and Stephan Günnemann. Intriguing properties of input- randomized smoothing. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20697-20743. PMLR, 17-23 Jul 2022. URL https://proceedding mlr. www.pl.e..........m.... saaaaa/vv62/ssueee
k22a. html.
[61] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
[62] Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(1 11951) 1970, 2008 doi: 10. 10.1009/PAML20000 128.
[63] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMIR, 18-24 Jul 2021. URL httpp://pproceedingg mlr, press/ htt////199/toooeeeeee
html.
[64] Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Jörn-Henrik Jacobsen. Fundamen- tradeoff's between invariance and sensitivity to adversarial perturbations. In International Conference on Machine Learning, pages 9561 9571, PMLR, 2020.
[65] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, volume 33, 2020.
14

---

[66] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner. and Aleksander Madry. Robust- ness may be odds with accuracy. In International Conference on Learning Representations, 2019. URL https://openrerie
het/Eooru22ddSSyyb300..77
[67] Lei Wang. Runtian Zhai, Di He, Liwei Wang. and Li Jian. cerrain-to-finetuue adversarial training via sample-wise randomized smoothing, 2021. URL https:: wttp://rrrweeieeeeeeer/oruruuaa ddt.ummmddddeeeeeey2eeaa
[68] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope, International Conference on Machine Learning, volume 80, pages 5286 5295, 10-13 Jul 2018. URL http: pptpp/////ooceengsss.... mlr press/v80/wwrr 2000//ong88 html.
[69] Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, and Dawn Song. DensePure: Understanding diffusion models for adversarial robustness. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ /opnreviee/rr.eee///ee/em het:/forurr1d-pp700006000...
[70] Greg Yang. Tony Duan. J Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. In International Conference Machine Learning, pages 10693-10705. PMIRR, 2020.
[71] Zhuolin Yang. Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robustness ensemble models and beyond. In International Conference on Learning Representations, 2022. URL https::/openrert
net/forum? Fawforrmmnddduaaaaa Ud-tUAARE
[72] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. survey autonomous driving: Common practices and emerging tochnologies. IEEE Access, 8:58443 -544699, 2020.
[73] Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsich, and Liwei Wang. MACER: Attack-free and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2020. URL https: h/ttpp:///opereviiee net/forum? id=rJx1Ma4Forr
[74] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlshy, and Lucas Beyer. Scaling Vision Transformers. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104- 12113, 2022.
[75] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 Proceedings of Machine Learning Re- search, pages 7472-7482, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: //proceedings. mww.eeddiiinmm.........mmm.mm..m Stt.ur.rres///////// /ttpg//a.nng91...
https//////aaaag9..9.9. html.
[76] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho- Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. In International Con- ference Learning Representations, 2020. URL htttp:://ppeneeve/e
net net/ffruum wwt.mmddddSikikaaemmmmmmm
[77] Huan Zhang, Shiqi Wang. Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. General cutting planes for propodpppppaannbbbaeeeeeee neural network verification. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: //oppn/eviee..eeeeeeeeee hett/toruumidd-hhaaJoooo
15

---

A Related Work
In the context of aggregating multiple smoothed classifiers, several works have previously explored on ensembling smoothed classifiers of the same noise scale [46, 32, 711, in attempt to boost their certified robustness. Our proposed multi-scale framework for randomized smoothing considers a different setup of aggregating smoothed classifiers of different noise scales, with novel motivation of addressing the current accuracy-robustnee trade-off in randomized smoothing. More in this respect, Mueller et al. [51] have considered a selective inference scheme between "certification" network (for robustness) and a "core" network (for accuracy), and Horváth et al. [33] have adapted the framework into randomized smoothing. Our approach explores an orthogonal direction, and can be viewed as directly improving the "certification" network here: in this way, we could offer improvements not only in empirical but also in certified accuracy as result.
Independently to our method, there have been approaches to further tighten the certified lower-bound that randomized smoothing guarantees [50, 70, 43, 45, 69]. Mohapatra et al. [50] have shown that nigher-order information of smoothed classifiers, e.g., its gradient on inputs, can tighten the certification of Gaussian-smoothed classifiers, beyond that only utilizes the zeroth-order information of smoothed confidence. In the context of denoised smoothing [58, 11], Xiao et al. [69] have recently proposed multi-step, multi-round scheme for diffusion models to improve accuracy of denoising at randomized smoothing. Such approaches could be also incorporated to our method to further improve certification. Along with the developments of randomized smoothing, there have been also other attempts in certifying deep neural networks against adversarial examples [23, 68, 49, 25, 76, 16, 4, 77], where a more extensive survey on the field can be found in Li al. [44].
B Experimental Details
B.1 Datasets
CIFAR-10 [39] consist of 60,000 images of size 32x32 pixels, 50,000 for training and 10,000 for testing. Each of the images is labeled to one of 10 classes, and the number of data per class is set evenly, i.e., 6,000 images per each class. By default, we do not apply any data augmentation but the input normalization with mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5]. following the standard training configurations of diffusion models. The full dataset can be downloaded at https://wwy cs. toronto. edu/"krizz cifar. html.
CIFAR-10-C [27] collections of 75 replicas of the CIFAR- 10 test datasets (of size 10,000), which consists of 15 different types of common corruptions each of which contains levels of corruption severities. Specifically, the datasets includes the following corruption types: (a) noise: Gaussian, shot, and impulse noise; (b) blur: defocus, glass, motion, zoom; (c) weather: snow, frost, fog, bright; and (d) digital: contrast, elastic, pixel, JPEG compression. In our experiments, we evaluate test errors on CIFAR- for models trained on the "clean" CIFAR-10 10 datasets. where the error values are averaged across different corruption types per severity level. The full datasets can be downloaded at https:// gi thub. com coom/hnndrycc lackks/robbee robustness....
CIFAR-10.1 [54] is reproduction of the CIFAR-10 test set that separately collected from Tiny Images dataset [62]. The dataset consists of 2,000 samples for testing, and designed to minimize distribution shift relative the original CIFAR-10 dataset in their data creation pipelines. The full dataset can be downloaded at https: / /i thub. www.moooommoaaaaeoooooooeooooeee wwwoddsstyaccts/CIAAA. 10 where we use the "v6" version for our experiments among those given in the repository.
ImageNet [56], also known as ILSVRC 2012 classification dataset, consists of 1.2 million high- resolution training images and 50,000 validation images, which are labeled with 1,000 classes. As data pre-processing step, we apply a 256x256 center cropping for both training and testing images after re-scaling the images to have 256 in their shorter edges, making it compatible with the pre- processing of ADM [18], the backbone diffusion model. Similarly to CIFAR- 10, all the images are normalized with mean [0.5. 0.5, 0.5) and standard deviation [0.5, 0.5, 0.5). A link for downloading the full dataset can be found in http: //mmmmaaeeana net org/dosnload.
IetgeNet-R [28] consists of 30,000 images of various artistic renditions for 200 (out of 1,000) ImageNet classes: e.g., art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, video game renditions,
16

---

and so on. To perform an evaluation of ImageNet classifiers on this dataset, we apply masking on classifier logits for the 800 classes those are not ImageNet-R. The full dataset can be downloaded at https: // gi tiub........ com http//hnnnrcdaaaaeeeeeeeeeeeeeee
ImageNet-A [30] consists of 7,500 images of 200 ImageNet classes those are "adversarially" filtered, by collecting natural images that causes wrong predictions from a pre-trained ResNet-50 model. To perform an evaluation of ImageNet classifiers on this dataset, we apply masking on classifier logits for the 800 classes those are not in ImageNet-A. The full dataset can be downloaded at https://githe /ttgggtthhb..... cacluu...oommoae com hennrhendyccs//////e/eco /nttsss/nataaaaaaaaaa adv examples.
B.2 Training
CLIP fine-tuning. We follow the training configuration suggested by FT-CLIP [19] in fine- tuning the CLIP ViT-B/16@ 224 model, where the full script is specified at htttp::/githtn
com/ LightDXY/FT SEXX/FT-CLLI CLIP. The same configuration is applied to all the datasets considered for fine- fine-tuning, namely CIFAR- 10, ImageNet, and caussian-corrupted versions of CIFAR-10 (for baselines such as "Gaussian" and "Consistency"), which can be done by resizing the datasets into 224 x 224 as a data pre-processing. We fine-tune all these tested models for 50 epochs each of the considered datasets.
Diffusion fine-tuning. Similarly to the CLIP fine men-tuiigg we follow the training details given by Nichol and Dhariwal [52] in fine-tuning diffusion models, which are specified in https:// / //ts::///tttte gitt.... com/ com/openai///e oeprovedd- wwweeedddddfffffd difffsion.. More concretely, here we fine-tune each pre-trained diffusion model by "resuming" the training following its configuration, but with an added regularization term we propose upon the original training loss. We use the 50M 10M-amamettrr 32 x 32 diffusion model from Nichol and Dhariwal [52] for CIFAR- 10, 6 and the 552M 552M-paraeeter 256 256 unconditional model from Dhariwal and Nichol [18] for ImageNet' to initialize each fine-tuning run. We fine-ture tune each model for 50K training steps, using batch size 128 and 64 for CIFAR- 10 and ImageNet, respectively.
B.3 Hyperparameters
Unless otherwise noted, we use po 0.5 for cascaded smoothing throughout our experiments. We mainly consider two configurations of cascaded smoothing: (a) E (0.25, 0.50, 1.00]. and (b) E f0.25, 0.50]. each of which denoted as "(0.25-1.00)" and "(0.25-0.50)" in tables, respectively. For diffusion calibration, on the other hand, we use = 1.0 by default. We use X = 0.01 on CIFAR-10 and = 0.005 on ImageNet, unless noted: we halve the regularization strength on ImageNet considering its batch size used in the fine-tuning. i.e., 128 (for CIFAR-10) vs. 64 (for ImageNet). In other words, we follow x = 0.01
Among the baselines considered, "Consistency" [35] requires two hyperparameters: the coefficient for the consistency
term 7 and the entropy term y. Following those considered by Jeong and Shin [35], we fix 7 = 0.5 and r 5.0 throughout our experiments.
B.4 Computing infrastructure
Overall, we conduct our experiments with cluster of NVIDIA V100 32GB GPUs and 8 instances of a single NVIDIA A 1100 80GB GPU. All the 10 experiments are run on a single NVIDIA A100 80GB GPU, including both the diffusion fine-tuning and the smoothed inference procedures. For the ImageNet experiments, we use 8 NVIDIA V100 32GB GPUs per run. In our computing environment, takes e.g., ~10 seconds and 55 minutes per image (we use n 10, 000 for each inference) to perform single pass of smoothed inference on CIFAR- 10 and ImageNet, respectively. For a single run of fine-tuning diffusion models, we observe ~1.5 days of training cost on CIFAR-10 10 with a single NVIDIA A100 80GB GPU, and that of 44 days on ImageNet using cluster of 8 NVIDIA V100 32GB GPUs, to run 50K training steps.
At inference time, our proposed cascaded smoothing may introduce an extra computational overhead as the cost for the increased accuracy, i.e., from taking multiple times of smoothed inferences. Nevertheless, remark that the actual overhead per input in practice can be different depending on the early- stopping result of cascaded smoothing. For example, in our experiments, we observe that
17

---

1.0
Carl-i et (2003)
0.6
(IFES (Elo) mu) 283)
Pg
0.8
(ox10.2
Pg 0.6
Contted ocouney
0.50
Pg 0.7
0.4
0.8
0.9
0.4
02
ABSTAIN 0.25
0.50
1.00
Figure 4: Comparison of certified test accu-
racy with (a) Carlini al. [11] at 0.25, (b) DependDennddnttt Randomized Smoothing
Figure 5: Comparison of histograms across Po
[0.5, 1.0) on the distribution of a's where each of CIFAR-10 test samples halts from cascaded smooth- ing. Each smoothed inferences is performed with 10, 000 noise samples and 0.001.
(IDRS) proposed by Súkeník et al. [60]. For (b), we adopt the per-sample a values of CIFAR-10 test set of Súkeník al. [60].
the average inference time to process the CIFAR-10 test set via cascaded smoothing is only ~ 1.5x compared to standard (single-scate scale) smoothed inferences, even considering a E (0.25, 0.5, 1.0).
C Additional Results
C.1 Comparison with input-dependent smoothing
In sense that the proposed cascaded smoothing results in different smoothing factors per-sample basis, our approach can be also viewed in the context of input-dependent randomized smoothing
[1, 67, 13, 20], line of attempts improve certified robustness by applying different smoothing factor conditioned on input. From being more flexible in assigning a, however, these approaches commonly require more assumptions for their certification, making them hard compare directly upon the standard evaluation protocol, e.g., that we considered in our experiments. For example, Súkeník al. [60] have proved that such a scheme of freely assigning does not provide valid robustness certificates in the standard protocol, and [1, 20] in response have suggested to further assume that all the previous test samples during evaluation can be memorized to predict future samples, which can affect the practical relevance of the overall evaluation protocol.
To address the issue, Súkeník al. [60] have also proposed fix of [1] discard its test-time dependency, by assuming a more strict restriction on the variation of a. Nevertheless, they also report that the benefit of input-dependent then becomes quite limited, supporting the challenging nature of depeddet smoothing. As shown in Figure 4 that additionally compares our method with [60], we indeed confirm that cascaded smoothing offers significantly wider robustness certificates with a better accuracy trade-off. This is unprecedented to the best of our knowledge, which could be achieved from our completely new approach of data-dependent smoothing: iee. by combining multiple smoothed classifiers through abstention, moving away from the previous attempts that are focused on obtaining a single-step smoothing.
C.2 Clean ensemble for empirical accuracy
In this appendix, we demonstrate that predictions from smoothed classifiers can only useful to obtain adversarial robustness, but also to improve out-of-distribution robustness of clean classifiers, which suggests new way utilize randomized smoothing. Specifically, consider a smoothed classifier f. and its smoothed confidences x, say Py(y (y) x) for y ). Next, we consider prediction confidences of "clean" prediction, namely py (v)x). Remark that py (y|x) can be interpreted in terms of logits, namely ( of neural network based model follows:
(r(x) = (x v) Py (x) Eyey P)
p(v) py
(10)
Note that the prior p(y) usually assumed to be the uniform distribution (1)).. Here, we consider variant of this prediction, by replacing the prior p(y) by mixture of p(y) and a smoothed prediction
18

---

Table 6: Comparison of per-corruption empirical accuracy of clean ("FT-CLIP") and smoothed classifiers (Others) on CIFAR-IO-C. For each corruption type, we report accuracy averaged over levels of severity, and for each level we use 1,000 uniform subsamples to compute the accuracy.
Brightnes
Clean
Gaaosian
Sioe
Inpulse
Delocsu
Chass
Molion
Z0n0n
Frontt
Fig
Contaut
Elanke
Pinchat
JPEG
Method
FIFCLIP
98.1
83.7
58.7
96.4
96.8
S5.7
95.0
96.1
96.1
96.0
97.0
97.8
97.5
91.7
91.0
88.5
93.1
Carlini
78.9
87.8
85.8
84.9
Casc. Calla.
92.5
89.0
90.1
91.5
90.7
$7.3
89.6
90.7
89.9
89.4
86.0
91.1
81.2
89.7
90.4
89.3
89.1
Clean prior
98.0
90.9
93.0
96.5
96.8
89.9
95.2
96.3
96.2
96.6
96.8
97.7
97.4
95.3
94.2
91.6
95.0
Table Comparison of empirical accuracy and certified accuracy of cascaded smooth classifiers on CIFAR-10 across ablations on A. We use E {0.25, 0.5, 1.0} for cascaded smoothing.
CIFAR-10
Empirical
Certified accuracy at (%)
Setups
Clean
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
a' 1.0
84.1
79.9
54.4
39.6
30.9
24.2
18.9
13.9
9.3
2.0
84.3
80.7
53.9
38.9
29.7
23.9
18.7
13.0
9.6
À = 0.001
a 4.0
80.9
52.8
39.1
29.5
23.2
17.3
12.4
9.4
a = 8.0
86.1
81.5
53.2
38.5
29.0
22.0
17.1
13.0
8.8
1.0
82,9
79.6
53.5
40.5
31.2
25.0
20.1
16.2
11.4
2.0
84.8
80.1
54.6
39.2
30.6
24.0
18.5
14.5
9.9
= 0.01
4.0
85.1
80.4
53.6
39.1
29.5
22.8
17.8
13.1
9.5
8.0
86.1
81.5
54.5
39.0
27.7
21.4
17.3
12.9
8.8
a 1.0
80.8
77.2
53.9
39.9
31.4
24.8
20.1
15.5
11.6
0
2.0
82.7
78.0
53.9
39.9
30,7
24.1
19.8
15.1
11.1
À = 0.1
4.0
82.1
78.3
52.6
39.1
30.1
22.6
17.4
13.7
10.0
8.0
83.7
78.2
51.1
36.9
28.0
22.3
17.9
13.1
9.1
Table Comparison of empirical accuracy and certified accuracy of cascaded smooth classifiers on CIFAR-10 across ablations on Po- We use a E {0.25, 0.5, 1.0} for cascaded smoothing.
CIFAR-10
Empirical
Certified accuracy at (%)
Setups
Clean
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Po 0.50
84.8
81.1
52.8
37.2
27.9
21.7
16.8
11.8
8.5
Po 0.55
87.9
82.6
53.5
35.6
26.9
19.8
14.0
10.1
7.1
Po 0.60
90.7
83.9
52.7
34.5
23.9
16.9
11.9
8.3
6.4
po 0.65
92.5
83.4
52.1
34.0
22.6
14.4
9.8
7.0
5.5
Po 0l.70
94.1
83.0
52.6
31.6
19.2
12.2
8.2
6.3
4.7
Po 0.75
95.2
81.9
52.2
29.1
16.5
9.5
6.7
5.2
3.7
Po 0.80
96.0
79.5
47.6
25.0
14.0
8.0
5.9
4.6
2.4
Po 0.85
96.6
76.3
45.0
22.8
12.5
7.1
4.7
2.8
2.2
Po 0.90
97.2
72.1
42.5
20.0
11.7
4.6
2.8
1.9
1.0
Po 0.95
97.8
65.5
38.6
16.7
7.2
2.2
1.7
0.7
0.0
p(t) = (xlx)): in this way, one could use f(x) as an additional prior to refine the clean confidence
in cases when the two sources of confidence do not match. More specifically, we consider:
(0gp)(0lx) = 1 (xlp)(>()) + +C, where log (0g1)()1xx) = =(1 (1 11-39 3)- log (y) + p(y).
(11)
(12)
where C' denotes the normalizing constant, and 3 > 0 is hyperparameter which we fix as f 0.1 our experiment. In Table we compare the accuracy of the proposed inference with the accuracy of standard classifiers derived from FT-CLIP [19] fine-tuned on CIFAR-10 overall, we observe that this "ensemble" of clean prediction with eascaded smoothed predictions could improve the corruption accuracy of FT-CLIP on CIFAR-10-C, namely by 93.4% - 95.0%, while maintaining its clean accuracy. The gain can be considered significant in sense that the baseline accuracy of 93.4% already at the state-of-the-ar level, e.g., according to RobustBench [17].
19

---

2.0
1.5
Criiccl
1.0
0.5
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
CIFAR-10 Classes
Figure 6: Comparison of per-class histograms of critical a: the minimum value of for each input where the smoothed confidence at true class falls below 0.5. We use the whole CIFAR-10 training samples for the analysis. Each image smoothed with n = 1, 000 samples of Gaussian noise, while iterating within the range [0.0, 2.0]. Images with critical a larger than 2.0 are truncated.
C.3 Ablation study
Effect of A. In Table 7, we jointly ablate the effect of tuning À E {0.001, 0.01, 0.1} upon the choices of a E {1.0, 2.0, 2.0, 4.0, 8.0), by comparing its certified accuracy of cascaded smoothing as CIFAR-10 test set. Overall, observe that increasing has
well as their empirical aceuracy effect of improving robustness at large radii, with a slight decrease in the clean accuracy as well as the empirical accuracy, where using higher could compensate this at some extent. Although generally observe effectiveness of the proposed regularization for a wide range of X, using higher values of X, e-g-, À 0.1, often affects the base diffusion model and results in degradation in accuracy without a significant gain in its certified robustness.
Effect of Po- Table 8, the other hand, reports the effect of po introduced for cascaded smoothing. Recall that we use Po = 0.5 throughout our experiments by default. We observe that using bit of higher values of Po. e.g., Po 0.6 could be beneficial to improve the clean accuracy of cascaded smoothing, at the cost of robustness at larger radii: for example, it improves the certified accuracy at 0.0 by 81.1 + 83.9. The overall certified robustness starts decrease po further increases, due to its effect of increasing abstention rate. One could still observe the consistent improvement in the empirical accuracy as Po increases, i.e., by also considering abstain prediction, i..., using high values po could be useful in certain scenarios. We think a more detailed choice of Po per important practical consideration.
valid
further boost the performance of cascaded smoothing, which is potentially
Critical a. Although it is not required for cascaded smoothing to guarantee the stability of smoothed confidences across a (e.g., outside of {0.25, 0.5, 1.0} in our experiments), we observe that smoothed confidences from denoised smoothing usually interpolate smoothly between different o's, at least for the diffusion denoised smoothing pipeline. In this respect, we consider a new concept of critical o for each sample, by measuring the threshold of a where its confidence goes below Po 0.5. Figure 6 examines this measure on CIFAR-10 10 training samples and plots its distribution for each class as histograms. Interestingly, "Ship" class among CIFAR-10 10 classes relatively obtains much higher critical a possibly due to its peculiar background of, e.g., ocean.
observe that the distributions
often significantly biased, e-g., the
Per-o distribution from cascading. To further verify that per-sample a's from cascaded smoothing are diverse enough to achieve better robustness-accurac trade-off, we examine in Figure 5 the actual histograms of the resulting a E {ABSTAIN, 0.25, 0.50, 1.00} on CIFAR-10, also varying Po (0.5, 0.6, 0.7, 0.8, 0.9]. Overall, observe that (a) the distributions of o indeed widely cover the total range of a for every Po 's considered, and (b) using higher pa can further improve diversity of the distribution while its chance to be abstained can be also increased.
D Technical Details
D.1 Proof of Theorem 3.1
Our proof is based on the fact that any smoothed classifier f composed with +-1(.) is always Lipschitz-continuuus, which is shown by Salman et al. [57]:
20

---

Lemma D.1 (followed by Salman et al. [57]). Let == = exp (-2s²) ds be the cu-
mulative distribution function of standard Gaussian. For any function f R - [0. 11, the map x & (vj, (x)) is 1-Lipschitz for any O.
Given Lemma I. the robustness certificate by Cohen et al. [15] in (2) (of the main text) can be viewed as computing the region of f. (x ò; y: > 0.5, ie., where it is guaranteed to return y over other (binary) smoothed classifiers pj. (x v/ of t
In general, as in our proposed cascaded smoothing, one can consider a more "strict" region defined by Pj. (x + ô; y) > Po for some Po 2 0.5. Here, the region corresponds to a policy that fo abstains whenever j. (x) S po- Accordingly, the certified radius one obtain from this policy becomes:
(fo: = (pj. (x. v)) - +b (pu))
(13)
With this notation, we restate Theorem 3.1 into tighter form show, which do not consider neither 2 nor p upon the knowledge of Pf.,, and implies Theorem 3.1 as consequence:
Theorem 3.1 (restated). Let Jo,.".. Sox XX- y be smoothed classifiers with O < < < оg. Suppose casc(x; (fo. (v=1] =: 5 y halts at fox for some Then, it holds that
casc(x + 6; ()) = 1 for ||0|| < R, where:
R = min (ou 0). R }. and R = min u= k'sk
{ min {0. - Roo (Sow
(14)
Proof. By the definition of casc(x)), one can ensure casc(x 6) the following holds:
(a) for (x + 8) = with Ple (x+ 8, Po, and
(b) Pj.p (x 6) pu for 6) -. for every W = k K.
Remark that the condition (a) is satisfied by ô within ||ô|| < Reo x Specifically, one can consider the "certifiable" region of 6 at Sry. as follows:
R₂ {8 X |00|| < Ryo (Foz :x,xx,,)1 ())
(15)
For the condition (b), on the other hand, note that the condition equivalent to that:
(b') Pleg. (x + 5; Po for y + j. and for every k' = k 1,..., K.
Fix W' E + KJ. From the Lipschitzedness of x + a --- (x; y)) by Lemma D.1, it is clear to show that 8's inside the following region R2 satisfies (b'):
Ry
n yデğ
x: ||l|| < min {o, Rlx (fo 8)
(16)
X |||||
min ッو
x, { min (0, -Rp-Rpoo (f. x1))
(17)
Lastly, the result is followed by considering the intersection of R2 's which represent certifiable region where casc(x + 5: f.fo. 1 i:
R = R2 n ((w) {6 E X: :18 < R).
(18)
Once we have the restated version of Theorem 3.1 above, the original statement follows from an observation that the new radius by introducing either p or p. say R, always lower-bounds R of the above: i.e., R <. R for any choices of p and thus certifiable for the region as well.
21

---

D.2 Certification and prediction
For practical uses of cascaded smoothing, where the exact values of pfa, (x, y) are not available, one has to estimate the smoothed confidence values as Theorem 3.1, i.e. by properly lower-bound (or upper-bound) the confidence values. More concretely, one is required to obtain (a) Pf., (x) at the
halting stage k; and (b) Pfey (x,y) for k' = k + 1, K and y E y.
For both estimations, perform a majority voting across { + 0:)3 to obtain a histogram on y of n trials:
Monte Carlo algorithm with ii.d Gaussian samples, say {o,), and
(a) To estimate a lower-bound f.gg (x), we follow the statistical procedure by Cohen et al. [15], namely CERTIFY (o,, ar), which additionally takes two hyper-parameter no and a: here, To denotes the number of samples initially make guess to estimate a smoothed prediction, and a is the significance level of the Clopper-Pearson confidence interval [14]. Concretely, it sets Pi. as the lower confidence level of Pf,, with coverage
(b) For fey (x, y). on the other hand, we first obtain voting counts from n. Gaussian trials, say
cx (gw), and adopt Goodman confidence interval to bound multinomial proportions [24] with set Pf... (x. y) for y E b the (1 a-)-pper level of the (multinomial) confidence interval. This can be implemented by using, e.g.. mult wtr.iimiaalpppporrttto
the significance level Specifically,
confint of the statsmodels library.
We make an additional treatment the significance level unless the pipeline halts at the first stage K, cascaded smoothing makes a (K + -ocossecuee testing confidence intervals to make a prediction. In attempt to maintain the overall significance level of this prediction a, one can apply the Bonferroni correction [5] on the significance level of individual testing, specifically by considering (K-k+1j the adjusted significance level (instead of using at the stage k.
E
Other Possible Designs
In this appendix, we present two alternative designs of multi-scale randomized smoothing we have also considered other than cascaded smoothing proposed in the main text. Specifically, here additionally propose two different policies of aggregating multiple smoothed classifiers, each of which dubbed as the (a) max-radius policy, and (b) focal smoothing, respectively. Overall, both design do provide valid certified guarantees combining the accuracy-robustess trade-offs across multiple smoothing factors, often attaining a better robustness when n, the number noise samples, is large enough.
compared to cascaded smoothing
The common drawback in the both approaches here, however, is the necessity to access every smoothed confidence, which can be crucial in practice in both terms of efficiency and accuracy. Specifically, for given K smoothed classifiers, this implies the exact K-times increase in the inference cost, which makes it much costly on average compared to cascaded smoothing. The need to access and compare with K smoothed confidence in practice also demands more accurate estimations of confidence values to avoid abstaining. For example, these approaches can significantly degrade the certified accuracy in practice where cannot be large unlike the certification phase: which again makes our cascaded smoothing more favorable over the approaches presented here.
E.1 Max-radius policy
Consider K smoothed classifiers, say for, X + V, where 0 < <.. < OK- The max-radius policy, say maxR(-), can be another simple policy one consider for given (fa.):
m (fo, K, = (x). where a² == arg max or Φ- (p (x))
(19)
a
Again, given the Lipschitzedness associated with the smoothed confidences pj, (x, v). it is easy to check the following robustness certificate for maxR(x; (ffo, IK)):
22

---

Theorem E.1. Let for,'.. fox - y be smoothed classifiers with 0 < <.. < og. If
maxR(x; (o. M 1j E y, it holds that maxR(x + 8; (fo J ÿ for [1ô]1 R. where:
R 0² 1 (p).
max 0; $-
Pfo, (xx:i)
(20)
1=1, K
Remark that the certified radius given by (20) recovers the guarantee of Cohen al. [15] K
E.2 Focal smoothing
In this method, coined focal smoothing, we take a somewhat different approach to achieve multi-scale smoothing. Here, now we assume two different levels of smoothing factor, say dg oi > and consider inputs that contains noise with spatially different magnitudes. Specifically, we consider binary mask M (0, 1} and an input corrupted by the following:
focal Mo; σ1) = x + (oo (1- to M)
(21)
where @ denotes the element-wiss wise product and e ~ (O, I). gets "hint" recover the original content of x by observing a less-corrupted region marked by M.
In this way, each of corrupted input &
Randomized smoothing over such anisotropic noise been recently studied [22, 20]: nutshell, smoothing over noises specified by (21) could produce an ellipsoid-shuped certified region with two (p)
axes of length u (pp) and σ Φ (p) for its smoothed confidence p. respectively.
In terms of certified radius, however, this implies that one can still certify only the region limited by the shorter axes (of length o1 / >--10(p)) (p)). turns out that such limitation can be avoided by considering an "ensemble" of K multiple orthogonal masks, say M1.-. ... Mr. so that the shorter axes of ellipsoids from the masks do not overlap each other. Specifically, we observe the following:
Theorem E.2. Let MI.... ,M E 13 be K orthogonal masks and ou > 01 >> 0 be smoothing factors, and Pı(x) be the smoothed confidence from inputs corrupted by focal(x, Mai oo o1). given binary classifier fk Rd + [0. 1]. Define p == K Σx Pk. and suppose f(x) 0.5. Then, is certified that p(x + 8) 0.5 for |0||| < Rb., where ba is the solution of the following optimization:
1 Σ
ag
minimize
Rb =
K
Vp +
subject to
Σ b6 1,
(22)
by 2 0, 1, 2,
K,
where ag := - for) 1,,.. K, and ==
c of - 1).
In Algorithm 1, we propose an efficient 1-dimensional grid-search based optimization to solve (22), based observations given as Lemma E.3 and E.4: each of which identifies the condition of by when it attains either the maximum value or zero, respectively. By solving the optimization, one could obtain a certified radius R that the aggregated smoothing can guarantee consistent prediction.
Lemma E.3. Consider the constrained optimization (22) with respect to b (b),... ,bk). Let Ma : {m E [K] am max; ax) and b* be an optimal solution of (22). Then, it holds that by br > for any m, m' E Ma-
Proof. First, we show that there exists at least one m E M, where by is nonzero, i.e., by, > Suppose the contrary, iee., bj 0 for all h E M,, Denote the corresponding objective value as R² Given the condition Σ, by 1, there always exists & M where > 0. Now, consider swapping the allocations of bm and b, for some e M, and denote the corresponding objective value R².
23

---

Algorithm Focal smoothing: A grid-search based optimization of (22)
Require: On > 0, k: 1.2,...., K, tolerance > 0
Ensure: optimal budget assignment b", certified radius R
1: Find the maximum value amn. max dg
2: Identify the index set M such that dg dman for & .M
3: Set a grid of values for bm the interval (0, 1].
4: for each grid value of by do
5: for ls = 1.2,..... K do
6: if l & M then
2/3 ((ex) am
7:
by - if S
else 1
(fhm + +1)
(1 +1///2
8:
end if end for
9:
E by 10:
if
then
11:
R k==l Ve
an
12:
Update bi, Ht bb, R if R²
end if
14:
end for
Then, we have:
R'-
R²
aj
Ok
aj
K
Pb +1
VP
+
bb +1
VBb, +1
1
1
1
1
K
al
/tb,
+1
Vleb
+1
+aj
V1 +1
/tb, +1
1
1
Ok
+ 0k
0,
K
/Pb, +1
+1
VB-bk +1
/eb, +
(23)
given that a% > aj and
1. This means R' < R', contradicting the optimality
128 Pbg
of the initial solution b' with by = 0 for all E Ma. Thus, there must exist at least one 6%, > 0 for some E Ma
Next, we show by b"T for any m, m' E Ma. Suppose there exist such m, m' E M,, where Dm * by. Due to the symmetry, one can construct another optimal allocation b' by swapping the two allocations d and bi,,, as it does not affect the optimality. Now consider another allocation b by averaging the two allocations:
b + b
bm = 6.
(24)
Let the corresponding objective value R. Note that R strictly convex, which implies that R < R', contradicting the optimality of the initial solution b". Thus, it holds by = b for any
m, m' Ma-
Lemma E.4. Consider the constrained optimization (22) with b' as an optimal solution. Denote
B = b" = bb, > for all m. m,m' Ma. Then, bj = 0 and only if og S (LBB+1p/2
A
Proof. Suppose bū, 0. From the Karush-Kuhn-Tuckk (KKT) stationarity condition, have
- X - Pa = 0. Since X, HR 2 0, we obtain À 2 aut² K Now, consider the stationarity
condition for m M: K(vPB+1M + X = 0. Substituting the lower bound for X, we have
At
as E (128-1)77
Conversely, suppose a* & (41577 Then, by > 0 contradict to that bg
1
2/3 ((B+1)-1) < 0, hence bi = 0.
24


Reviews:
[{"review_id": "83b2jIOeAG", "reviewer": "Reviewer_Vnny", "summary": "This work aims to improve robustness-accuracy tradeoff in image classifiers, using the recent smoothing ideas. The authors propose two techniques for improving adversarial robustness without loss of accuracy. One is to fine-tune the prepended denoiser with a regularized loss, to reduce miss-classification of smoothed images. Second is to use a cascade of denoisers trained on different levels of noise to pick the highest level robustness radius. They test their method on CIFAR10 and ImageNet datasets. \n\n", "questions": "1) The idea of cascading denoise and classify models from large to small noise levels is a clever way to find the largest radius of robustness. The final sigma at which the model does not abstain and returns a classification output is taken as the certifies radius. This value however depends on two factors: one is the threshold $p_0$: for higher threshold the radius will be smaller and for lower threshold the radius will be larger. It is not clear in the text how this dependency is addressed. The second factor is the image content and the image class: for example in CIFAR10 for a class with generally distinctive background and content the certified radius will be larger (more smoothed images will be classified with high confidence). On the other hand, if the average background is similar between a few classes then the radius will be smaller. How do you address this variability? \n\n2) For a fixed $p_0$ and a fixed image, do different noise samples (different runs of the method) result in different certified radius and or different predictions? If that is the case, how is this variability addressed? \n\n3) Have you experimented with stability of prediction after the optimal radius is achieved? In other words, one way to quantify the error (or incorrect over confidence in the cascade) is to keep running the cascade for all the smaller noise levels and measure how volatile the prediction is. Assuming that for $\\sigma=0$ the predication is correct (consistent with your oracle assumption on $f_{std}$), you can measure the $\\sigma$ level for which the incorrect prediction flips to the correct one. Will you see a significant difference for different classes for this critical $sigma$?  \n", "limitations": "I found the paper lacking solid theoretical ground. The methods, although clever, are ad hoc and do not address the robustness against adversarial attacks fundamentally. Regardless, due to impressive empirical results my score is borderline accept. I will increase the score if the authors address the comments and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) Focusing on important topic of robustness of classifiers. \n2) Interesting method in cascading models from high noise to low noise. \n3) Good empirical results compared to recent state of the art models ", "weaknesses": "The writing can be improved a lot both in terms of coherence and clarity. I found the paper lacking in a good flow and hard to read, because of redundant repetition, lack of a coarse to fine description of the goal and the result, redundant undefined notation. The authors The same amount of content can be written such that the reader can take the message with much less effort. \n\nWhenever a new notation is introduced, please describe what it is defining. The reader should not have to infer what symbol is refering to what. They should be mentioned explicitly in the text. Examples: $\\mathbb{P}, \\epsilon, x', \\underline{R}, \\underline{p} \n,\\overline{p}, R^-$ ... This makes the paper more time consuming to read which is very unnecessary. As an example Theorem 3.1 is  unreadable due to undefined notation. \n\nI find the use of \"diffusion\" in the title somewhat misleading. I suggest diffusion be replaced with denoising since that is what has been used in the work as opposed to a diffusion process in the context of generative models. \n\nIn line 109 it is mentioned that the ideal denoiser should return the original image for any noisy image. This statement is incorrect and hold only for very small levels of noise. The error in denoising is inherent in ideal denoisers as well and hence the iteration.\n\nIn order to reduce misclassification, the authors suggest fine-tuning the denoiser instead of the classifier to make the training less cumbersome. Although this resulted in better accuracy, it is not clear if it makes sense in theory. The cause of unrobustness is the random or incorrect boundaries set by the classifier. The denoiser is optimal with respect to denoising loss, whereas the classifier is not optimal with respect to classifying smoothed images (it has never been trained to do so). So in theory, the network that should be fine-tuned is the classifier not the denoiser. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions one-by-one in what follows. Please let us know if you have any comments/concerns that we have not addressed up to your satisfaction.\n\n---\n\n**Q1. Suggestions for better clarity - e.g., notation, title, etc.**\n\nMany thanks for your suggestions to improve the clarity of our manuscript. We will carefully revise our manuscript to reflect your comments, e.g., by adding more detailed descriptions of the mathematical notations. \n\n---\n\n**Q2. Line 109: “ideal denoiser”?**\n\nWe appreciate your detailed feedback on the term “ideal denoiser” we mentioned for an intuition of denoised smoothing. We agree that the notation $\\texttt{denoise}(\\mathbf{x} + \\boldsymbol{\\delta}) \\approx \\mathbf{x}$ could be more precise if it was further noted as “semantic” equivalence with respect to $f$, rather than the exact match. We will clarify this point in the final draft.  \n\n---\n\n**Q3. Fine-tuning denoiser vs. classifier**\n\nThank you for your comment. We have addressed this specific comment in the common response, so please refer to the section for a detailed answer.  \n\n---\n\n**Q4. How is the effect of $p_0$ addressed?**\n\nWe simply use a fixed value of $p_0 = 0.5$ in our experiments (as specified in Appendix B.3), but we note here for your information that we did perform an ablation study of $p_0$ in Appendix E.2. Indeed, the choice of threshold $p_0$ affects the certified radius of cascading, as well as its accuracy: higher $p_0$ tends to decrease the overall certified radii, but it may increase the accuracy due to the increased chance to reach lower $\\sigma$. We think the detailed choice of $p_0$ per $\\sigma$ can further boost the results, which would be an interesting practical consideration. We will further clarify this point in the final draft.   \n\n---\n\n**Q5. Variability of $\\sigma$ over classes**\n\nAs you mentioned, the distributions of output $\\sigma$ from cascaded smoothing can be different per class (Q7 also discusses a related point). A possible reason why such a variability exists is that the model itself, i.e., either the denoiser or classifier, is biased towards certain features of a class, and this may affect the performance of denoised smoothing if the features were less generalizable, e.g., from backgrounds. Our proposed denoiser fine-tuning is an attempt to address this point particularly focusing on the denoiser part: specifically, we propose anti-consistency loss to penalize when a denoiser confidently outputs a different class from the input - Figure 1(b) in our manuscript illustrates an example, for your information.\n\n---\n\n**Q6. Consistency of cascading over noise samples**\n\nWe implement our proposed cascaded smoothing to make “statistically consistent” predictions across different noise samples, considering a certain _significance level_ $\\alpha$ (e.g., $\\alpha = 0.001$ in our experiments) - in a similar fashion as Cohen et al. [1]. Roughly speaking, for a given input $\\mathbf{x}$, it makes predictions only when the confidence interval of $p_{\\hat{f}}(\\mathbf{x})$ does not overlap with $p_0$ upon $n$ _i.i.d._ noise samples and $\\alpha$ (otherwise it abstains). The more details on the certification procedure can be found in Appendix C.2 for your information, and we will further highlight this in the final draft, e.g., by also providing a pseudocode. \n\n---\n\n**Q7. “Critical” $\\sigma$, and its distribution**\n\nAlthough it is not required for our cascaded smoothing itself to ensure the stability you mentioned across all $\\sigma$, e.g., outside $\\\\{0.25, 0.5, 1.0\\\\}$, we indeed observe that smoothed confidences observed from denoised smoothing usually interpolate smoothly between different $\\sigma$’s, at least for the diffusion denoised smoothing pipeline - in other words, one can also consider a concept of “critical” $\\sigma$ as you mentioned, by measuring the threshold of $\\sigma$ where its confidence goes below, e.g., 0.5. Figure S1 in the rebuttal PDF examines this measure on CIFAR-10 training samples and plots its distribution for each class as histograms: interestingly, we observe that the distributions are often significantly biased, e.g., the “Ship” class among CIFAR-10 classes relatively obtains much higher critical $\\sigma$ - possibly due to its peculiar background of, e.g., ocean. We appreciate your insightful question, and we will incorporate this result and its respective discussion in the final draft. \n\n---\n\n**References**\n\n[1] Cohen et al., Certified adversarial robustness via randomized smoothing, ICML 2019.\n"}, {"review_id": "uQHugjHUW4", "reviewer": "Reviewer_QpH2", "summary": "This paper builds on diffusion based denoised randomized smoothing. It analyzes two scenarios of model errors which are over-smoothing and over-confidence. To alleviate these issues, the authors proposed deploying cascaded randomized smoothing where samples are first smoothed with large variance that gradually decreases if the model abstains from prediction. Further, the diffusion model (denoiser) is calibrated to overcome the over-confidence problem.", "questions": "Please refer to the weaknesses part.", "limitations": "The authors have addressed the limitations in Appendix A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work has the following strengths:\n\n1- The problem this work is tackling is important. Building reliable and robust models are necessary for deploying such models.\n\n2- The paper reads smoothly in most parts. Section 3.1 introduces two problems in randomized smoothing and Sections 3.2 and 3.3 propose the two components of the proposed approach.\n\n3- The experiments conducted in this work covered the standard datasets and 3 related baselines.\n", "weaknesses": "Despite the strengths of this work, there are few weaknesses that I would like to be resolved before getting this paper accepted:\n\n1- The experimental results are missing important strong baselines such as [1, 36] (from the main paper) and [12] from the appendix. Since the method of this work results in a different smoothing parameters for (potentially) different inputs, a comparison with data-dependent smoothing approaches is necessary.\n\n2- The proposed approach seems to provide very limited performance gains in the small certified accuracy regimes. For instance, comparing the certified accuracy in Table 1 at $\\sigma \\in [0.25, 1.0]$ and $\\epsilon =0.5$: The proposed method attains 40.5% while [8] attains a 59.8 using $\\sigma = 0.25$. Shouldn't a fair comparison be comparing the proposed method to the envelop curve of each baseline across the three $\\sigma$ values?\n\n3- One ablation experiment that can potentially showcases the diversity of the obtained $\\sigma$ via cascaded smoothing is to plot a histogram of $\\sigma_x$ for when $\\sigma \\in [0.25, 1.0]$.\n\n4- Another experimental analysis is to show the number of samples that were suffering from over-smoothing or over-confidence before employing the cascaded approach and after.\n\n5- In terms of writing, I have the following minor comments/suggestions:\n- The preliminary part about diffusion models in lines 116-128 is not used afterwards, especially the notation. Consider moving it to the appendix.\n- $n$ was defined twice in the \"Evaluation metrics\" paragraph (line 285 and line 290) with two different values. Please consider modifying the second one.\n- It is confusing to have more than one bold number per column in each table. Consider highlighting only one (perhaps the combined cascading + calibration).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions one-by-one in what follows. Please let us know if you have any comments/concerns that we have not addressed up to your satisfaction.\n\n---\n\n**Q1. Comparison with previous data-dependent approaches?**\n\nWe first clarify that most of previous data-dependent smoothing approaches, including [1-4], are not directly comparable upon the standard evaluation protocol we considered, due to their additional assumptions to ensure correctness given the theoretical challenge of data-dependent smoothing. For example, [1] and [4] require that all the previous test samples during evaluation can be memorized to predict future samples, which can affect the practical relevance of the overall evaluation protocol. \n\nTo address the issue, Sukenik et al. [5] have recently proposed a fix of [1] to discard its test-time dependency: nevertheless, they also report that the benefit of data-dependent $\\sigma$ becomes quite limited with the fix, supporting its challenging nature. Upon your suggestion, we have further performed an additional comparison of our method with [5], e.g., as in Figure S2 of the attached rebuttal PDF, confirming that our proposed cascaded smoothing offers significantly wider robustness certificates with a better accuracy trade-off - which is unprecedented to the best of our knowledge. This is achieved with our completely new approach of data-dependent smoothing, i.e., by combining multiple smoothed classifiers through abstention, moving away from previous attempts that are focused on obtaining a single-step smoothing. We will incorporate this and further results into the final draft, as well as the respective discussions. \n\n---\n\n**Q2. Limited gains at small $\\varepsilon$’s, e.g., compared to the $\\sigma$-envelope?**\n\nThe focus of our evaluations, e.g., in Table 1 and 2, is to compare smoothed classifiers taking into account their trade-off between certified robustness and accuracy. For example, although the certified accuracy of a smoothed classifier with $\\sigma=0.25$ may perform better at small $\\varepsilon$ compared to cascaded smoothing with $\\sigma=[0.25, 1.0]$, the classifier cannot provide robustness certification at $\\varepsilon \\ge 1.0$. This is why we chose average certified radius (ACR) as a major performance metric in this paper, which is in compliance with the current literature: specifically, ACR assigns 0 for incorrect samples while giving more weights for samples with better robustness.   \n  \nAlthough the envelope curve across multiple $\\sigma$’s can be a succinct proxy to evaluate different methods, it can be somewhat misleading and unfair to compare the curve directly with individual smoothed classifiers. This is because the curve does not really construct a concrete classifier on its own: specifically, it additionally assumes that each test sample has prior knowledge on the value of $\\sigma \\in  \\\\{0.25, 0.5, 1.0\\\\}$ to apply, which is itself challenging to infer - this is indeed what our proposal of cascaded smoothing addresses. We will clarify this point in the final draft. \n\n---\n\n**Q3. Suggestions for ablation study**\n\nThank you for your suggestions to improve our ablation study, viz., (a) the distribution of $\\sigma_x$, and (b) the effect of cascading on over-smoothing, which would strengthen our manuscript. For (a), we have additionally examined the histogram of $\\sigma \\in \\\\{\\mathtt{ABSTAIN}, 0.25, 0.50, 1.00\\\\}$ from cascaded smoothing on CIFAR-10 at varying $p_0$, e.g., as plotted in Figure S3 of the rebuttal PDF: overall, we observe that distributions of $\\sigma_x$ widely cover across the range for all the $p_0$’s tested; For (b), on the other hand, we will further report and compare a more detailed breakdown of model errors with respect to the two categories, i.e., over-smoothing ($p \\le p_0$) and over-confidence ($p > p_0$), e.g., as reported in Table S2 of the rebuttal PDF: here, e.g., we observe that over-smoothing can be the major source of errors especially at high noise level, e.g., $\\sigma=1.0$, and our proposed cascading dramatically reduces the error. We will incorporate these results into the final draft, as well as more results, e.g., a qualitative inspection of samples with different $\\sigma_x$’s.\n\n---\n\n**Q4. Editorial comments**\n\nMany thanks for the careful reading to improve our manuscript. We will incorporate all of your editorial comments and suggestions into the final draft.\n\n---\n\n**References**\n\n[1] Alfarra et al., Data dependent randomized smoothing, UAI 2022.\n\n[2] Wang et al., Pretrain-to-finetune adversarial training via sample-wise randomized smoothing, 2021.\n\n[3] Chen et al., Insta-RS: Instance-wise randomized smoothing for improved robustness and accuracy, 2021.\n\n[4] Eiras et al., ANCER: Anisotropic certification via sample-wise volume maximization, TMLR 2022.\n\n[5] Sukenik et al., Intriguing properties of input-dependent randomized smoothing, ICML 2022.\n"}, {"review_id": "9uXttfe8P9", "reviewer": "Reviewer_b42a", "summary": "This paper identifies the accuracy-robustness trade-off in randomized smoothing and the over-smoothing and over-confidence issues in diffusion denoised smoothing. It then proposes a to finetune diffusion models to mitigate these issues by first passing a datapoint through a cascaded smoothing pipeline to obtain class label and then minimizing Brier loss and Anti-consistency losses. This allows the framework to achieve higher empirical accuracy and robustness.", "questions": "My concerns and questions are written above. ", "limitations": "No limitations of the model are discussed. Inference time increase can be one to further discuss and analyze. Broader societal impacts are adequately addressed as improving adversarial robustness leads to safer AI systems.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and well motivated. \n- The cascaded pipeline is novel and the explanations of the problems at hand are clear.\n- Empirical evidence for improvement is strong.\n- Ablation study and analysis is performed to investigate the effect of each proposed loss.", "weaknesses": "There are several places in the exposition that may require more clarification.\n\n- Why do we not update classifier? Line 218 - 220 state that finetinung classifier is more computationally intensive than finetuning denoiser. Isn't backpropogating through an entire trajectory of denoising process more prohibitively expensive than finetuning classifier, especially when the starting noise-level is high?\n\n- Can the classifier itself cause any of the two over-smoothing or over-confidence issues? In this case, do we wan to finetune classifiers? In so, how do we decide which model to finetune, or both?\n\n- There lack some empirical investigation on the runtime issue with denoising, as is more apparent for such a long sequence of denoising iterations. The cascaded smoothing starts at a high noise-level, so denoising should take longer time. What's the trade-off between the accuracy/robustness increase with runtime?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions one-by-one in what follows. Please let us know if you have any comments/concerns that we have not addressed up to your satisfaction.\n\n---\n\n**Q1. The higher $\\sigma$, the longer runtime?**\n\nThank you for pointing out an important detail we overlooked to specify. Here, we clarify that the runtime of the diffusion denoised smoothing pipeline [1] (that we are based on) does NOT depend on the choice of $\\sigma$. This is because the pipeline is originally proposed to use “single-step” denoising (which is usually available from standard diffusion models) for its denoiser operation at any $\\sigma$, based on the observation that it is beneficial both for efficiency as well as accuracy [1]. In our experiments, we have also followed this design choice. We will clarify this point in the final draft. \n\n---\n\n**Q2. Fine-tuning denoiser vs. classifier**\n\nAs also discussed in Q1, our proposed diffusion fine-tuning scheme does not require backpropagation through multiple denoising iterations, given that the denoiser operation in each smoothing is restricted to single-step. We have also addressed this specific question in the common response, so please refer to the section for a further discussion.  \n\n---\n\n**Q3. Limitations are not discussed, e.g., inference time?**\n \nFor your information, we note that Appendix A does contain a discussion about limitations, indeed covering the increased inference time of our method. In the final draft, we will make this clearer in the main text as well. \n\n---\n\n**References**\n\n[1] Carlini et al., (Certified!!) Adversarial Robustness for Free!, ICLR 2023.\n"}, {"review_id": "Yv1LpxNyYG", "reviewer": "Reviewer_K2e2", "summary": "This paper presents a practical method to address the issues of over-smoothing and over-confidence in randomized smoothing, thereby enhancing our understanding of the trade-off between accuracy and robustness. For over-smoothing, the author introduces a multi-stage approach for computing collective certified robustness without the need to specify noise scales. This novel process provides a more precise evaluation of the certified radius. Additionally, to address over-confidence, this study incorporates two proposed loss functions, namely Brier loss and Anti-consistency loss, to fine-tune the denoiser model. By integrating these two methods into Diffusion Denoised Smoothing, significant improvements in both clean and certified accuracy can be achieved.", "questions": "1. Results shown in your table are different from those shown in the original paper of Carlini et al. For example, in Table 1, the highest clean acc is 78.8 while the original paper shows 88.1. Is that because you use a different classifier? If yes, why you do not choose to use the classifier used in Carlini et al.? They can even reach the better results.\n\n2. In Table 3, if the clean acc of Carlini et al. is higher than your results with Anti-consist, how can you demonstrate that your proposed loss can improve clean accuracy? I think it can only improve the overall clean accuracy with your cascaded strategy.", "limitations": "No limitations about potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Cascaded randomized smoothing is a specifically designed method for more accurate certified radius evaluation for Diffusion Denoised Smoothing because with diffusion models, we do not need to train smoothed classifiers for each noise scale.\n\n2. Evaluate both empirical accuracy and certified accuracy for the robustness of the proposed method.", "weaknesses": "1. I don't think the cascaded randomized smoothing can overcome the trade-off between accuracy and certified robustness. Cascaded randomized smoothing is only a tricky strategy to make a better computation of the certified radius. It cannot improve certified robustness intrinsically.\n\n2. For calibrating diffusion models, I don't see an advantage compared with classifier fine-tuning in Carlini et al. And Table 2 in Carlini et al. shows a better performance than yours with classifier finetuning. Besides, normally speaking, fine-tuning classifiers is much simpler than fine-tuning diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions one-by-one in what follows. Please let us know if you have any comments/concerns that we have not addressed up to your satisfaction.\n\n---\n\n**Q1. “Cascaded smoothing cannot improve certified robustness intrinsically.”**\n\nWe agree that the “intrinsic” approach you mentioned to enhance adversarial robustness, e.g., presumably by training a more robust model representation, should be an important path forward - indeed, this is why we have also explored diffusion fine-tuning as well as cascaded smoothing in our work. \n\nNevertheless, we politely disagree that such an intrinsic approach is the only way to improve the robustness-accuracy trade-off, especially in the context of randomized smoothing where the trade-off is often “unavoidable”. For example, Table 1 of our paper reports that even the current state-of-the-art, e.g., Carlini et al. [1], could only achieve the clean accuracy $<50\\\\%$ on CIFAR-10 to obtain robustness at $\\varepsilon=2.0$: in order to obtain a non-zero robustness at $\\varepsilon = 2.0$ with randomized smoothing, one has to perturb a given input with severe Gaussian noise of $\\sigma = 1.0$ and it often makes some inputs inevitably non-classifiable. \n\nIn this respect, we view our proposal of cascaded smoothing as not only a simple strategy to optimize certified radius, but also a novel attempt to overcome the trade-off fundamentally induced by the data itself. In the aforementioned example, one can now get a chance to reduce the noise level, e.g., from $\\sigma = 1.0$ to $\\sigma = 0.5$, to turn a “non-classifiable” input “classifiable”, while maintaining the robustness of other samples. Knowing the right amounts of per-input adversarial budget $\\varepsilon$ for better generalization has been one of open questions not only in the context of randomized smoothing but also in adversarial machine learning at large, e.g., [2], and we believe our approach could be a useful step to answer the question. \n\n---\n\n**Q2. Fine-tuning denoiser vs. classifier**\n\nThank you for your comment. We have addressed this specific comment in the common response, so please refer to the section for a detailed answer. \n\n---\n\n**Q3. Table 1: Lower baseline results compared to Carlini et al. [1]?**\n\nWe clarify that there might be some confusion regarding the results presented: the clean accuracy rate you mentioned, 78.8%, is actually from our results in Table 2, which is based on ImageNet. The corresponding result for Table 1, i.e., on CIFAR-10, is 89.5%, which is actually better than the 88.1% from the original paper by Carlini et al. [1]. In general, the clean accuracy of denoised smoothing tends to follow that of the backbone classifier, and this could explain why the highest accuracy in Table 1 is better than those reported in Carlini et al.: specifically, our choice of classifier performs better (98.1% on CIFAR-10) compared to theirs (97.9%). \n\n---\n\n**Q4. Table 3: Clean accuracy is degraded?**\n\nWe remark that a comparison of smoothed classifiers is not solely about clean accuracy: it is important to consider the trade-off between certified robustness and accuracy. This is a reason why we have adopted average certified radius (ACR) as a key metric in our comparisons. The Brier loss we propose is primarily to enhance certified robustness, which might occasionally come at the cost of slightly reduced accuracy. Even with the inclusion of anti-consistency, there could be a minor decline in accuracy. Yet, given the substantial improvements in ACR shown in both ablations, we believe any decrease in accuracy can be easily compensated, e.g., by reducing the relative strength of the Brier loss.\n\nAs another note, we also highlight that the decrease in accuracy displayed in Table 3 is not a constant across different datasets: e.g., for the ImageNet results presented in Table 2, we observe that our proposed fine-tuning not only improved overall certified robustness but also clean accuracy.\n\n---\n\n**References**\n\n[1] Carlini et al., (Certified!!) Adversarial Robustness for Free!, ICLR 2023.\n\n[2] Tramer et al., Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations, ICML 2020.\n"}, {"review_id": "2jnT7NdHnu", "reviewer": "Reviewer_rNFT", "summary": "This paper proposed a new technique for certified adversarial robustness based on randomized smoothing. Two certification schemes are proposed in the paper: first is cascaded randomized smoothing where multiple smoothed classifiers of different smoothing factors are aggregated together in an efficient way; the second part is a new fine-tuning method called diffusion calibration to address the overconfidence issue in the original randomized smoothing approach. Experimental results show that the proposed method improved upon the previous baseline approach of Diffusion Denoised Smoothing.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clarity: This paper is well-written. The method is clearly described and introduced. The results are presented in a concise and clear fashion.\n2. Originality: The method is novel in two aspects. First, the proposed scheme of cascading smoothed classifiers of multiple smoothing factors is novel and promising as an approach for combining multiple smoothed classifiers. Second, two losses are proposed for calibrating the diffusion model in Diffusion Denoised Smoothing.\n3. Significance: Randomized smoothing and its variant, for example, diffusion denoised smoothing are known to have practical limitations as they require huge certification samples to compute the certification bound. The proposed method for more efficient certification could pave the way for practical applications of randomized smoothing and diffusion denoised smoothing.", "weaknesses": "1. While the proposed method delivers improvement of certification accuracies upon the standard diffusion denoised smoothing, the paper does not discuss the computational requirements. It would be good to discuss the computational overhead from running a cascading smoothed classifier and fine-tuning the diffusion denoiser, in the main paper. \n2. It seems from the description of cascaded randomized smoothing in section 3.2, the prediction of the cascaded smoothed classifier is defined, different from what the original randomized smoothing proposed. Then i think there should be a subsection or paragraph defining the certification bounds from the cascaded smoothed classifier. Theorem 1 does not seem to point to which one is the new certification bound. I would suggest the authors to add a pseudocode in the main text or appendix describing how the prediction and certification is done for the proposed method, in a similar way to what the original randomized smoothing paper [1] did.\n3. Section 3.1 discussed some of the limitations of randomized smoothing. It is not clear whether this is a contribution of this work or has been observed before. From the way it is described, these limitations seems like well-recognized in the research community. If that’s the case, it would be better to put this part in the background section to avoid confusion about the novelty. \n4. Regarding the specific limitations of randomized smoothing discussed in Section 3.1, is there evidence that supports these arguments? It would be good to show empirical evidence of overconfidence or point to a previous work that has looked into this matter.\n5. The Brier loss and Anti-consistency loss are proposed to address the overconfidence issue in randomized smoothing. The experiments could benefit from adding an evaluation confirming that the overconfidence issue is tackled via the proposed fine-tuning process, thus further supporting the validity and effectiveness of the approach.\n6. The main contributions of the work are two parts: cascaded randomized smoothing and diffusion calibration. However, the title “multi-scale diffusion denoised smoothing” seems to only focus on the first part of the contribution. I would recommend a more proper title to reveal the true contributions of this work, which are not limited to cascaded randomized smoothing.\n\n[1] Certified adversarial robustness via randomized smoothing. Cohen et al, 2019.\n\n==================================================================   \nI have read the rebuttal and i will remain my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions one-by-one in what follows. Please let us know if you have any comments/concerns that we have not addressed up to your satisfaction.\n\n---\n\n**Q1. Computational overhead**\n\nThank you for the suggestion, and we will report the respective discussion in the final draft. As you mentioned, our proposed cascaded smoothing may introduce an inference overhead as the cost for an increased accuracy, i.e., from taking multiple times of smoothed inferences. Nevertheless, we remark that the actual overhead per input depends on the early-stopping in cascaded smoothing in practice: for example, in our experiments, we observe that the average inference time to process the CIFAR-10 test set via cascaded smoothing ($\\sigma\\in \\\\{0.25, 0.5, 1.0\\\\}$) is only ~1.5x compared to the standard one. We will incorporate this and more results in the final draft. \n\n---\n\n**Q2. Detailed certification procedure**\n\nIndeed, our implementation of cascaded smoothing has a treatment to make it statistically correct across _i.i.d._ noise samples, considering a certain _significance level_ $\\alpha$ (e.g., $\\alpha = 0.001$ in our experiments) in a similar fashion as Cohen et al. [1]. This certification procedure is described in Appendix C.2 for your information, and we will further provide more details in the final draft, e.g., by adding a pseudocode of the procedure as per your suggestion.\n\n---\n\n**Q3. Section 3.1: Is this a prior observation, or a contribution?**\n\nWe believe that (a) our re-interpretation of smoothed confidence, especially in the context of diffusion denoised smoothing, and (b) the corresponding decomposition of model errors based on it can be novel contributions to the community. While smoothed confidence has been used to abstain inputs in randomized smoothing, it has primarily been seen as a trick to ensure statistical guarantee of certified radius. It is from our observations to develop methods to address model errors separately based on types, i.e., through cascaded smoothing and diffusion fine-tuning. We will make this point clearer in the final draft.  \n\n---\n\n**Q4. Empirical support of over-confidence and over-smoothing**\n\nFor a given smoothed classifier, one can indeed measure the two types of model errors following their definitions we consider in this work, i.e., $p \\le p_0$ for over-smoothing and $p > p_0$ over-confidence, respectively. Following your suggestion, we will further report and compare this detailed breakdown of model error, e.g., as provided in Table S2 of the rebuttal PDF: overall, we observe that over-smoothing can be the major source of errors especially at high noise level, e.g., $\\sigma=1.0$, and our proposed cascading dramatically reduces the error. Although we find cascading could increase over-confidence from accumulating errors through multiple inferences, our diffusion fine-tuning could alleviate the errors jointly reducing the over-smoothing as well. We will include these and more results in the final draft, as well as the respective discussions. \n\n---\n\n**Q5. Title**\n\nThank you for the suggestion. We agree that there can be a better title which can further emphasize our key contributions: cascaded smoothing and diffusion calibration. We have chosen the current title to succinctly reflect our key aspect, but we will definitely consider your suggestion and revisit our title, while ensuring it continues to be concise.\n\n---\n\n**References**\n\n[1] Cohen et al., Certified adversarial robustness via randomized smoothing, ICML 2019.\n\n\n"}]

Author rebuttal:
Dear reviewers,

We thank all the reviewers’ efforts to improve our manuscript. This common response addresses a concern raised by multiple reviewers, viz., on our diffusion fine-tuning scheme compared to classifier fine-tuning. We also kindly ask you to find the attached rebuttal PDF in this response, which contains figures and tables to support our responses.

---

**[Vnny, b42a, K2e2] Fine-tuning denoiser vs. classifier**

Despite its effectiveness to improve denoised smoothing, as previously verified by Carlini et al. [1], we would like to highlight that classifier fine-tuning itself can be less practical when it comes with the cascaded smoothing pipeline proposed in this paper (in particular, the issue becomes more severe when dealing with larger classifiers):

- (a) To perform cascaded smoothing at multiple noise scales, classifier fine-tuning would require separate runs of training for optimal models per scale, also resulting in multiple different models to load - which can be less scalable in terms of memory efficiency.  
- (b) In a wider context of denoised smoothing, the classifier part is often assumed to be a model at scale, even covering cases when it is a “black-box” model, e.g., public APIs. Classifier fine-tuning, in such cases, can become prohibitive or even impossible.

With respect to (a) and (b), the denoiser fine-tuning we propose offers a more efficient inference architecture: it can handle multiple noise scales jointly with a single diffusion model, while also being applicable to the extreme scenario when the classifier model is black-box. In other words, we argue that our proposed denoiser fine-tuning can be a scalable alternative to the classifier fine-tuning in practical scenarios. 

Nevertheless, if one wishes, fine-tuning both classifiers and denoiser can bring their complementary effects to improve denoised smoothing: e.g., as we further examined in Table S1 of the rebuttal PDF, we observe that additionally fine-tuning classifiers (viz., to tune the classifier on denoised inputs rather than clean ones upon the diffusion fine-tuning) could further improve the overall ACR of the classifier. We also observe that in our experiments, denoiser fine-tuning itself could obtain a comparable gain in ACR to classifier fine-tuning, while the former can be favorable over the latter to use in practice due to (a) and (b).

In essence, the (somewhat counter-intuitive) effectiveness of denoiser fine-tuning we verify confirms that denoising process (e.g., via diffusion models) can be also biased as well as classifiers to make over-confident errors, where our proposed anti-consistency loss could play a role, e.g., as in Table S2 of the rebuttal PDF. We believe this observation can not only advance the performance, but also opens up a new direction of de-biasing diffusion models through the lens of randomized smoothing.  

In the final draft, we will clarify these points and incorporate the additional results with the respective discussions. 

---

Thank you for your consideration,

Authors

---

**References**

[1] Carlini et al., (Certified!!) Adversarial Robustness for Free!, ICLR 2023.


Meta-review: The AC has carefully read the paper, reviews, author response, and the discussions. This work aims to improve the robustness-accuracy tradeoff in image classifiers by proposing two techniques for improving adversarial robustness without loss of accuracy. Their method has been evaluated on CIFAR10 and ImageNet datasets. All reviewers are positive towards this paper, including the clarity of writing, novelty, experimental results, and the importance of the problem. The AC agrees with the reviewers that the current form of the paper passes the bar of NeurIPS and thus recommends acceptance. However, the authors are strongly encouraged to follow reviews to polish the paper, for example, adding more comparisons between diffusion fine-tuning scheme and classifier fine-tuning.
Decision: Accept (poster)