SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2We also discuss several other possible (and more sophisticated) designs in Appendix E.
3In our experiments, we simply use  $p_0 = 0.5$  for all  $\sigma$ 's. See Appendix C.3 for an ablation study with  $p_0$ .
4The proof of Theorem 3.1 is provided in Appendix D.1.
3.3 Calibrating diffusion models through smoothing
Next, we move on to the over-confidence issue in randomized smoothing:  $\text{viz.}$ ,  $\hat{f}_{\sigma}$  often makes errors with a high confidence to wrong classes. We propose to fine-tune a given  $\hat{f}_{\sigma}$  to make rather diverse outputs when it misclassifies, i.e., towards a more "calibrated"  $\hat{f}_{\sigma}$ . By doing so, we aim to cast the issue of over-confidence as that of over-smoothing: which can be easier to handle in practice, e.g., by abstaining. In this work, we particularly focus on fine-tuning only the denoiser model  $D$  in the context of denoised smoothing, i.e.,  $f \coloneqq f_{\mathrm{std}} \circ D$  for a standard classifier  $f_{\mathrm{std}}$ : this offers a more scalable approach to improve certified robustness of pre-trained models, given that fine-tuning the entire classifier model in denoised smoothing can be computationally prohibitive in practice.
Specifically, given a base classifier  $f \coloneqq f_{\mathrm{std}} \circ D$  and training data  $\mathcal{D}$ , we aim to fine-tune  $D$  to improve the certified robustness of  $\hat{f}_{\sigma}$ . To this end, we leverage the confidence information of the backbone classifier  $f_{\mathrm{std}}$  fairly assuming it as an "oracle" - which became somewhat reasonable given the recent off-the-shelf models available - and apply different losses depending on the confidence information per-sample basis. We propose two losses in this matter: (a) Brier loss for either correct or under-confident samples, and (b) anti-consistency loss for incorrect, over-confident samples.
Brier loss. For a given training sample  $(\mathbf{x},y)$ , we adopt the Brier (or "squared") loss [6] to regularize the denoiser function  $D$  to promote the confidence of  $f_{\mathrm{std}}(D(\mathbf{x} + \boldsymbol{\delta}))$  towards  $y$ , which can be beneficial to increase the smoothed confidence  $p_{\hat{f}_{\sigma}}(\mathbf{x})$  that impacts the certified robustness at  $\mathbf{x}$ . Compared to the cross-entropy (or "log") loss, a more widely-used form in such purpose, we observe that the Brier loss can be favorable in such fine-tuning of  $D$  through  $f_{\mathrm{std}}$ , in a sense that the loss is less prone to "over-optimize" the confidence at values closer to 1.
Here, an important detail is that we do not apply the regularization to incorrect-yet-confident samples: i.e., whenever  $f_{\mathrm{std}}(D(\mathbf{x} + \boldsymbol{\delta})) \neq y$  and  $p_{\mathrm{std}}(\boldsymbol{\delta}) \coloneqq \max_c F_{\mathrm{std},c}(D(\mathbf{x} + \boldsymbol{\delta})) > p_0$ , where  $F_{\mathrm{std}}$  is the soft prediction of  $f_{\mathrm{std}}$ . This corresponds to the case when  $D(\mathbf{x} + \boldsymbol{\delta})$  rather outputs a "realistic" off-class sample, which will be handled by the anti-consistency loss we propose. Overall, we have:
$$
L _ {\text {B r i e r}} (\mathbf {x}, y) := \mathbb {E} _ {\boldsymbol {\delta}} [ \mathbf {1} [ \hat {y} _ {\boldsymbol {\delta}} = y \text {o r} p _ {\text {s t d}} (\boldsymbol {\delta}) \leq p _ {0} ] \cdot \| F _ {\text {s t d}} (D (\mathbf {x} + \boldsymbol {\delta})) - \mathbf {e} _ {y} \| ^ {2} ], \tag {7}
$$
where we denote  $\hat{y}_{\delta} \coloneqq f(\mathbf{x} + \boldsymbol{\delta})$  and  $\mathbf{e}_y$  is the  $y$ -th unit vector in  $\mathbb{R}^{|\mathcal{V}|}$ .
Anti-consistency loss. On the other hand, the anti-consistency loss aims to detect whether the sample is over-confident, and penalizes it accordingly. The challenge here is that identifying over-confidence in a smoothed classifier requires checking for  $p_{\hat{f}_{\sigma}}(\mathbf{x}) > p_0$  (4), which can be infeasible during training. We instead propose a simpler condition to this end, which only takes two independent Gaussian noise, say  $\delta_1, \delta_2 \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ . Specifically, we identify  $(\mathbf{x}, y)$  as over-confident whenever (a)  $\hat{y}_1 \coloneqq f(\mathbf{x} + \delta_1)$  and  $\hat{y}_2 \coloneqq f(\mathbf{x} + \delta_2)$  match, while (b) they are incorrect, i.e.,  $\hat{y}_1 \neq y$ . Intuitively, such a case signals that the denoiser  $D$  is often making a complete flip to the semantics of  $\mathbf{x} + \delta$  (see Figure 1(b) for an example), which we aim to penalize. A care should be taken, however, considering the possibility that  $D(\mathbf{x} + \delta)$  indeed generates an in-distribution sample that falls into a different class: in this case, penalizing it may result in a decreased robustness of that sample. In these respects, our design of anti-consistency loss forces the two samples simply to have different predictions, by keeping at least one prediction as the original, while penalizing the counterpart. Denoting  $\mathbf{p}_1 \coloneqq F_{\mathrm{std}}(D(\mathbf{x} + \delta_1))$  and  $\mathbf{p}_2 \coloneqq F_{\mathrm{std}}(D(\mathbf{x} + \delta_2))$ , we again apply the squared loss on  $\mathbf{p}_1$  and  $\mathbf{p}_2$  to implement the loss design, as the following:
$$
L _ {\mathrm {A C}} (\mathbf {x}, y) := \mathbf {1} [ \hat {y} _ {1} = \hat {y} _ {2} \text {a n d} \hat {y} _ {1} \neq y ] \cdot \left(\left\| \mathbf {p} _ {1} - \mathbf {s g} (\mathbf {p} _ {1}) \right\| ^ {2} + \left\| \mathbf {p} _ {2} \right\| ^ {2}\right), \tag {8}
$$
where  $\mathfrak{sg}(\cdot)$  denotes the stopping gradient operation.
Overall objective. Combining the two proposed losses, i.e., the Brier loss and anti-consistency loss, defines a new regularization objective to add upon any pre-training objective for the denoiser  $D$ :
$$
L (D) := L _ {\text {D e n o i s e r}} + \lambda \cdot \left(L _ {\text {B r i e r}} + \alpha \cdot L _ {\mathrm {A C}}\right), \tag {9}
$$
where  $\lambda, \alpha > 0$  are hyperparameters. Here,  $\alpha$  denotes the relative strength of  $L_{\mathrm{AC}}$  over  $L_{\mathrm{Brier}}$  in its regularization. Remark that increasing  $\alpha$  would give more penalty on over-confident samples, which would lead the model to make more abstentions: therefore, this results in an increased accuracy particularly in cascaded smoothing (Section 3.2).
In our experiments, we simply use  $\alpha = 1$  unless otherwise specified.
(b) ImageNet
Figure 2: Comparison of certified test accuracy with Carlini et al. [11] on the effectiveness of cascaded smoothing.
Figure 3: Comparison of certified test accuracy on CIFAR-10 at  $\sigma \in \{0.25, 0.50, 1.00\}$ .
Table 1: Comparison of (a) certified accuracy, (b) empirical accuracy, and (c) average certified radius (ACR) on CIFAR-10. We set our result bold-faced whenever it achieves the best upon baselines.
CIFAR-10, ViT-B/16@224	Empirical accuracy (%)	Certified accuracy at ε (%)
σ	Training	C10	C10-C	C10.1	0.0	0.5	1.0	1.5	2.0	ACR
0.25	Gaussian [15]	92.7	88.3	76.7	87.0	55.3	0.479
Consistency [35]	89.5	84.7	67.6	84.9	60.7	0.517
Carlini et al. [11]	92.8	88.8	67.6	89.5	59.8	0.527
+ Calibration (ours)	94.5	89.4	69.2	90.3	61.9	0.535
0.50	Gaussian [15]	87.4	81.8	71.0	69.9	45.9	24.4	6.0	0.543
Consistency [35]	80.7	75.0	59.6	67.7	49.2	31.2	13.1	0.617
Carlini et al. [11]	88.8	84.9	64.3	75.3	50.6	31.0	14.3	0.646
+ Calibration (ours)	88.6	84.2	63.7	75.1	53.3	32.5	15.3	0.664
(0.25-0.50)	+ Cascading (ours)	92.5	89.1	78.5	85.1	53.3	32.9	14.9	0.680
1.00	Gaussian [15]	75.6	72.4	68.3	42.2	27.9	16.9	9.0	4.0	0.426
Consistency [35]	65.8	61.3	53.6	43.1	31.1	22.3	14.6	8.7	0.513
Carlini et al. [11]	87.3	83.2	73.0	48.6	32.5	21.9	11.8	6.4	0.530
+ Calibration (ours)	83.3	79.3	60.9	52.6	36.7	24.9	16.1	8.8	0.577
(0.25-1.00)	+ Cascading (ours)	90.1	86.9	74.5	79.6	40.5	25.0	16.2	8.8	0.645
We verify the effectiveness of our proposed schemes, (a) cascaded smoothing and (b) diffusion calibration, mainly on CIFAR-10 [39] and ImageNet [56]: two standard datasets for an evaluation of certified  $\ell_2$ -robustness. We provide the detailed experimental setups, e.g., training, datasets, hyperparameters, computes, etc., in Appendix B.
Baselines. Our evaluation mainly compares with diffusion denoised smoothing [11], the current state-of-the-art methodology in randomized smoothing. We additionally compare with two other training baselines from the literature, by considering models with the same classifier architecture but without the denoising step of Carlini et al. [11]. Specifically, we consider (a) Gaussian training [15], which trains a classifier with Gaussian augmentation; and (b) Consistency [35], which additionally regularizes the variance of predictions over Gaussian noise in training. We select the baselines assuming practical scenarios where the training cost of the classifier side is crucial: other existing methods for smoothed classifiers often require much more costs, e.g., 8 times over Gaussian [57, 73].
**Setup.** We follow Carlini et al. [11] for the choice of diffusion models: specifically, we use the 50M-parameter  $32 \times 32$  diffusion model from Nichol and Dhariwal [52] for CIFAR-10, and the 552M-parameter  $256 \times 256$  unconditional model from Dhariwal and Nichol [18] for ImageNet. For the classifier side, we use ViT-B/16 [63] pre-trained via CLIP [53] throughout our experiments. For uses we fine-tune the model on each of CIFAR-10 and ImageNet via FT-CLIP [19], resulting in classifiers that achieve  $98.1\%$  and  $85.2\%$  in top-1 accuracy on CIFAR-10 and ImageNet, respectively. Following the prior works, we mainly consider  $\sigma \in \{0.25, 0.50, 1.00\}$  for smoothing in our experiments.
Evaluation metrics. We consider two popular metrics in the literature when evaluating certified robustness of smoothed classifiers: (a) the approximate certified test accuracy at  $r$ : the fraction of the test set which CERTIFY [15] classifies correctly with the radius larger than  $r$  without abstaining,
Table 2: Comparison of (a) certified accuracy, (b) empirical accuracy, and (c) average certified radius (ACR) on ImageNet. We set our result bold-faced whenever it achieves the best upon baselines.
ImageNet, ViT-B/16@224	Empirical accuracy (%)	Certified accuracy at ε (%)
Training	σ	IN-1K	IN-R	IN-A	0.0	0.5	1.0	1.5	2.0	ACR
Carlini et al. [11]	0.25	80.9	69.3	35.3	78.8	61.4	0.517
Carlini et al. [11]	0.50	79.5	67.5	32.3	65.8	52.2	38.6	23.6	0.703
+ Cascading (ours)	(0.25-0.50)	83.5	69.6	41.3	75.0	52.6	39.0	22.8	0.720
+ Calibration (ours)	(0.25-0.50)	83.8	69.8	41.7	76.6	54.6	39.8	23.0	0.743
Carlini et al. [11]	1.00	77.2	64.3	32.0	30.6	25.8	20.6	17.0	12.6	0.538
+ Cascading (ours)	(0.25-1.00)	82.6	69.8	40.5	69.0	42.4	26.6	19.0	14.6	0.752
+ Calibration (ours)	(0.25-1.00)	83.2	69.5	40.8	72.5	44.0	27.5	19.9	14.1	0.775
and (b) the average certified radius (ACR) [73]: the average of certified radii on the test set  $\mathcal{D}_{\mathrm{test}}$  while assigning incorrect samples as 0: viz., ACR :=  $\frac{1}{|\mathcal{D}_{\mathrm{test}}|} \sum_{(\mathbf{x},y) \in \mathcal{D}_{\mathrm{test}}} [\mathrm{CR}(f, \sigma, \mathbf{x}) \cdot \mathbb{1}_{\hat{f}(\mathbf{x}) = y}]$ , where  $\mathrm{CR}(\cdot)$  denotes the certified radius that CERTIFY returns. Throughout our experiments, we use  $n = 10,000$  noise samples to certify robustness for both CIFAR-10 and ImageNet. We follow [15] for the other hyperparameters to run CERTIFY, namely by  $n_0 = 100$ , and  $\alpha = 0.001$ . In addition to certified accuracy, we also compare the empirical accuracy of smoothed classifiers. Here, we define empirical accuracy by the fraction of test samples those are either (a) certifiably correct, or (b) abstained but correct in the clean classifier: which can be a natural alternative especially at denoised smoothing. For this comparison, we use  $n = 100$  to evaluate empirical accuracy.
Unlike the evaluation of Carlini et al. [11], however, we do not compare cascaded smoothing with the envelop accuracy curve over multiple smoothed classifiers at  $\sigma \in \{0.25, 0.50, 1.00\}$ : although the envelope curve can be a succinct proxy to compare methods, it can be somewhat misleading and unfair to compare the curve directly with an individual smoothed classifier. This is because the curve does not really construct a concrete classifier on its own: it additionally assumes that each test sample has prior knowledge on the value of  $\sigma \in \{0.25, 0.5, 1.0\}$  to apply, which is itself challenging to infer. This is indeed what our proposal of cascaded smoothing addresses.
Cascaded smoothing. Figure 2 visualizes the effect of cascaded smoothing we propose in the plots of certified accuracy, and the detailed results are summarized in Table 1 and 2 as “+ Cascading”. Overall, we observe that the certified robustness that cascaded smoothing offers can be highly desirable over the considered single-scale smoothed classifiers. Compared to the single-scale classifiers at highest  $\sigma$ , e.g.,  $\sigma = 1.0$  in Table 1 and 2, our cascaded classifiers across  $\sigma \in \{0.25, 0.50, 1.00\}$  absolutely improve the certified accuracy at all the range of  $\varepsilon$  by incorporating more accurate predictions from classifiers, e.g., of lower  $\sigma \in \{0.25, 0.50\}$ . On the opposite side, e.g., compared to Carlini et al. [11] at  $\sigma = 0.25$ , the cascaded classifiers provide competitive certified clean accuracy ( $\varepsilon = 0$ ), e.g.,  $89.5\%$  vs.  $85.1\%$  on CIFAR-10, while being capable of offering a wider range of robustness certificates. Those considerations are indeed reflected quantitatively in terms of the improvements in ACRs. The existing gaps in the clean accuracy are in principle due to the errors in higher- $\sigma$  classifiers: i.e., a better calibration, to let them better abstain, could potentially reduce the gaps.
Diffusion calibration. Next, we evaluate the effectiveness of our proposed diffusion fine-tuning scheme: “+ Calibration” in Table 1 and 2 report the results on CIFAR-10 and ImageNet, respectively, and Figure 3 plots the CIFAR-10 results for each of  $\sigma \in \{0.25, 0.50, 1.00\}$ . Overall, on both CIFAR-10 and ImageNet, we observe that the proposed fine-tuning scheme could uniformly improve certified accuracy across the range considered, even including the clean accuracy. This confirms that the tuning could essentially improve the accuracy-robustness trade-off rather than simply moving along itself. As provided in Table 3, we remark that simply pursuing only the Brier loss (7) may achieve a better ACR in overall, but with a decreased accuracy: it is the role of the anti-consistency loss (8) to balance between the two, consequently to achieve a better trade-off afterwards.
Empirical accuracy. In practical scenarios of adopting smoothed classifiers for inference, it is up to users to decide how to deal with the "abstained" inputs. Here, we consider a possible candidate of simply outputting the standard prediction instead for such inputs: in this way, the output could be noted as an "uncertified" prediction, while possibly being more accurate, e.g., for in-distribution
Table 3: Comparison of ACR and certified accuracy of cascaded smooth classifiers on CIFAR-10 over different training and ablations. We use  $\sigma \in \{0.25, 0.5, 1.0\}$  for cascaded smoothing. Bold and underline indicate the best and runner-up, respectively. Model reported in Table 1 is marked as grey.
Cascaded, σ ∈ {0.25, 0.5, 1.0}	Certified accuracy at ε (%)
Training	ACR	0.00	0.25	0.50	0.75	1.00	1.25	1.50	1.75
Gaussian [15]	0.470	70.7	46.9	31.7	23.7	16.8	12.4	9.2	6.5
Consistency [35]	0.548	57.6	42.7	32.3	26.3	22.4	18.1	14.6	11.1
Carlini et al. [11]	0.579	80.5	52.8	37.4	28.0	21.7	16.8	11.8	8.5
+ Cross-entropy	0.605	77.9	52.8	39.6	29.1	22.2	18.7	13.5	9.5
+ Brier loss	0.666	77.6	55.7	41.7	32.7	26.8	21.7	16.3	11.9
+ Anti-consist. (α = 1.0)	0.645	79.6	53.5	40.5	31.2	25.0	20.1	16.2	11.4
+ Anti-consist. (α = 2.0)	0.625	80.1	54.6	39.2	30.6	24.0	18.5	14.5	9.9
+ Anti-consist. (α = 1.0)	0.566	83.9	55.1	37.8	25.8	19.8	14.5	11.5	8.1
Table 4: Comparison of ACR and certified accuracy (\%) on CIFAR-10 across two fine-tuning schemes: diffusion fine-tuning ("Diff."; ours) and classifier fine-tuning ("Class.") [11]. We use  $\sigma = 0.5$  for this experiment.
Fine-tuning	ACR	Certified accuracy at ε (%)
Diff.	Class.	0.00	0.25	0.50	0.75	1.00	1.25
X	X	0.639	75.3	62.0	50.6	39.3	31.0	22.9
✓	X	0.664	75.1	64.1	53.3	42.1	32.5	25.0
X	✓	0.664	76.0	64.7	52.7	42.1	32.4	24.4
✓	✓	0.673	75.2	64.7	53.5	43.8	32.5	25.8
Table 5: Comparison of model error rates on CIFAR-10 decomposed into (a) over-smoothing  $(p\leq p_0)$  and (b) overconfidence  $(p > p_{0})$  , also with ACR.
Model (σ = 1.0)	Error rates (%, ↓)	ACR (↑)
p ≤ p0	p > p0
Carlini et al. [11]	43.8	7.6	0.498
Cascading	5.0	14.5	0.579
+ Anti-consist.	4.7	11.4	0.566
+ Brier loss	3.6	16.8	0.645
inputs. Specifically, in Table 1 and 2, we consider a fixed standard classifier of CLIP-finetuned ViT-B/16 on CIFAR-10 (or ImageNet), and compare the empirical accuracy of smoothed classifiers on CIFAR-10, -10-C [27] and -10.1 [54] (or ImageNet, -R [28], and -A [30]). Overall, the results show that our smoothed models could consistently outperform even in terms of empirical accuracy while maintaining high certified accuracy, i.e., they abstain only when necessary. We additionally observe in Appendix C.2 that the empirical accuracy of our smoothed model can be further improved by considering an "ensemble" with the clean classifier: e.g., the ensemble improves the accuracy of our cascaded classifier  $(\sigma \in \{0.25, 0.5\})$  on CIFAR-10-C by  $88.8\% \to 95.0\%$ , even outperforming the accuracy of the standard classifier of  $93.4\%$ .
In Table 3, we compare our result with other training baselines as well as some ablations for a component-wise analysis, particularly focusing on their performance in cascaded smoothing across  $\sigma \in \{0.25, 0.50, 1.00\}$  on CIFAR-10. Here, we highlight several remarks from the results, and provide the more detailed study, e.g., the effect of  $p_0$  (5), in Appendix C.3.
Cascading from other training. "Gaussian" and "Consistency" in Table 3 report the certified robustness of cascaded classifiers where each of single-scale models is individually trained by the method. Even while their (certified) clean accuracy of  $\sigma = 0.25$  models are competitive with those of denoising-based models, their collective accuracy significantly degraded after cascading, and interestingly the drop is much more significant on "Consistency", although it did provide more robustness at larger  $\varepsilon$ . Essentially, for a high clean accuracy in cascaded smoothing the individual classifiers should make consistent predictions although their confidence may differ: the results imply that individual training of classifiers, without a shared denoiser, may break this consistency. This supports an architectural benefit of denoised smoothing for a use as cascaded smoothing.
Cross-entropy vs. Brier. “+ Cross-entropy” in Table 3 considers an ablation of the Brier loss (7) where the loss is replaced by the standard cross-entropy: although it indeed improves ACR compared to Carlini et al. [11] and achieves the similar clean accuracy with “+ Brier loss”, its gain in overall certified robustness is significantly inferior to that of Brier loss as also reflected in the worse ACR. The superiority of the “squared” loss instead of the “log” loss suggests that it may not be necessary to optimize the confidence of individual denoised image toward a value strictly close to 1, which is reasonable considering the severity of noise usually considered for randomized smoothing.
Anti-consistency loss. The results marked as “+ Anti-consist.” in Table 3, on the other hand, validates the effectiveness of the anti-consistency loss (8). Compared to “+ Brier loss”, which is equivalent to the case when  $\alpha = 0.0$  in (9), we observe that adding anti-consistency results in a slight decrease in ACR but with an increase in clean accuracy. The increased accuracy of cascaded smoothing indicates that the fine-tuning could successfully reduce over-confidence, and this tendency continues at larger  $\alpha = 2.0$ . Even with the decrease in ACR over the Brier loss, the overall ACRs attained are still superior to others, confirming the effectiveness of our proposed loss in total (9).
Classifier fine-tuning. In Table 4, we compare our proposed diffusion fine-tuning (Section 3.3) with another possible scheme of classifier fine-tuning, of which the effectiveness has been shown by Carlini et al. [11]. Overall, we observe that fine-tuning both classifiers and denoiser can bring their complementary effects to improve denoised smoothing in ACR, and the diffusion fine-tuning itself could obtain a comparable gain to the classifier fine-tuning. The (somewhat counter-intuitive) effectiveness of diffusion fine-tuning confirms that denoising process can be also biased as well as classifiers to make over-confident errors. We further remark two aspects where classifier fine-tuning can be less practical, especially when it comes with the cascaded smoothing pipeline we propose:
(a) To perform cascaded smoothing at multiple noise scales, classifier fine-tuning would require separate runs of training for optimal models per scale, also resulting in multiple different models to load - which can be less scalable in terms of memory efficiency.
(b) In a wider context of denoised smoothing, the classifier part is often assumed to be a model at scale, even covering cases when it is a "black-box" model, e.g., public APIs. Classifier fine-tuning, in such cases, can become prohibitive or even impossible.
With respect to (a) and (b), denoiser fine-tuning we propose offers a more efficient inference architecture: it can handle multiple noise scales jointly with a single diffusion model, while also being applicable to the extreme scenario when the classifier model is black-box.
Over-smoothing and over-confidence. As proposed in Section 3.1, errors from smoothed classifiers can be decomposed into two: i.e.,  $p \leq p_0$  for over-smoothing, and  $p > p_0$  over-confidence, respectively. In Table 5, we further report a detailed breakdown of the model errors on CIFAR-10, assuming  $\sigma = 1.0$ . Overall, we observe that over-smoothing can be the major source of errors especially at such a high noise level, and our proposed cascading dramatically reduces the error. Although we find cascading could increase over-confidence from accumulating errors through multiple inferences, our diffusion fine-tuning could alleviate the errors jointly reducing the over-smoothing as well.
Randomized smoothing has been traditionally viewed as a somewhat less-practical approach, perhaps due to its cost in inference time and impact on accuracy. In another perspective, to our knowledge, it is currently one of a few existing approaches that is prominent in pursuing adversarial robustness at scale, e.g., in a paradigm where training cost scales faster than computing power. This work aims to make randomized smoothing more practical, particularly concerning on scalable scenarios of robustifying large pre-trained classifiers. We believe our proposals in this respect, i.e., cascaded smoothing and diffusion calibration, can be a useful step towards building safer AI-based systems.
Limitation. A practical downside of randomized smoothing is in its increased inference cost, mainly from the majority voting procedure per inference. Our method also essentially possesses this practical bottleneck, and the proposed multi-scale smoothing scheme may further increase the cost from taking multiple smoothed inferences. Yet, we note that randomized smoothing itself is equipped with many practical axes to reduce its inference cost by compensating with abstention: for example, one can reduce the number of noise samples, e.g., to  $n = 100$ . It would be an important future direction to explore practices for a better trade-off between the inference cost and robustness of smoothed classifiers, which could eventually open up a feasible way to obtain adversarial robustness at scale.