SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
R := \min  \left\{R _ {p _ {0}} \left(\hat {f} _ {\sigma_ {k}}; \mathbf {x}, \hat {y}\right), R ^ {-} \right\}, \text {a n d} R ^ {-} := \min  _ {\substack {y \neq \hat {y} \\ k ^ {\prime} > k}} \left\{\min  \left\{0, - R _ {p _ {0}} \left(\hat {f} _ {\sigma_ {k ^ {\prime}}} ; \mathbf {x}, y\right) \right\} \right\}. \tag{14}
$$
Proof. By the definition of  $\operatorname{casc}(\mathbf{x})$ , one can ensure  $\operatorname{casc}(\mathbf{x} + \delta) = \hat{y}$  if the following holds:
(a)  $\hat{f}_{\sigma_k}(\mathbf{x} + \boldsymbol{\delta}) = \hat{y}$  with  $p_{\hat{f}_{\sigma_k}}(\mathbf{x} + \boldsymbol{\delta}, \hat{y}) > p_0$ , and
(b)  $p_{\hat{f}_{\sigma_{k^{\prime}}}}(\mathbf{x} + \boldsymbol {\delta}) <   p_0$  or  $\hat{f}_{\sigma_{k'}}(\mathbf{x} + \boldsymbol {\delta}) = \hat{y}$  , for every  $k^{\prime} = k + 1,\dots ,K$
Remark that the condition (a) is satisfied by  $\delta$  within  $\| \delta \| < R_{p_0}(\hat{f}_{\sigma_k};\mathbf{x},\hat{y})$ . Specifically, one can consider the "certifiable" region of  $\delta$  at  $\hat{f}_{\sigma_k}$  as follows:
$$
\mathcal {R} _ {k} = \left\{\boldsymbol {\delta} \in \mathcal {X}: \| \boldsymbol {\delta} \| <   R _ {p _ {0}} \left(\hat {f} _ {\sigma_ {k}}; \mathbf {x}, \hat {y}\right) \right\}. \tag {15}
$$
For the condition (b), on the other hand, note that the condition is equivalent to that:
(b')  $p_{\hat{f}_{\sigma_{k'}}}(\mathbf{x} + \boldsymbol{\delta}; y) < p_0$  for  $y \neq \hat{y}$ , and for every  $k' = k + 1, \dots, K$ .
Fix  $k' \in \{k + 1, \dots, K\}$ . From the Lipschitzedness of  $\mathbf{x} \mapsto \sigma \cdot \Phi^{-1}(p_{\hat{f}_{\sigma}}(\mathbf{x}; y))$  by Lemma D.1, it is clear to show that  $\delta$ 's inside the following region  $\mathcal{R}_{k'}$  satisfies  $(\mathbf{b}')$ :
$$
\begin{array}{l} \mathcal {R} _ {k ^ {\prime}} = \bigcap_ {y \neq \hat {y}} \left\{\boldsymbol {\delta} \in \mathcal {X}: \| \boldsymbol {\delta} \| <   \min  \left\{0, - R _ {p _ {0}} \left(\hat {f} _ {\sigma_ {k ^ {\prime}}}; \mathbf {x}, y\right) \right\} \right\} (16) \\ = \left\{\boldsymbol {\delta} \in \mathcal {X}: \| \boldsymbol {\delta} \| <   \min  _ {y \neq \hat {y}} \left\{\min  \left\{0, - R _ {p _ {0}} \left(\hat {f} _ {\sigma_ {k ^ {\prime}}}; \mathbf {x}, y\right) \right\} \right\} \right\}. (17) \\ \end{array}
$$
Lastly, the result is followed by considering the intersection of  $\mathcal{R}_k$ 's which represent a certifiable region where  $\operatorname{casc}(\mathbf{x} + \boldsymbol{\delta}; \{\hat{f}_{\sigma_i}\}_{i=1}^K) = \hat{y}$ :
$$
\mathcal {R} := \mathcal {R} _ {k} \cap \left(\cap_ {k ^ {\prime} \geq k + 1} \mathcal {R} _ {k ^ {\prime}}\right) = \{\boldsymbol {\delta} \in \mathcal {X}: \| \boldsymbol {\delta} \| <   R \}. \tag {18}
$$
Once we have the restated version of Theorem 3.1 above, the original statement follows from an observation that the new radius by introducing either  $\underline{p}$  or  $\bar{p}$ , say  $\underline{R}$ , always lower-bounds  $R$  of the above: i.e.,  $\underline{R} \leq R$  for any choices of  $\underline{p}$  and  $\bar{p}$ , thus certifiable for the region as well.
D.2 Certification and prediction
For practical uses of cascaded smoothing, where the exact values of  $p_{\hat{f}_{\sigma_k}}(\mathbf{x},y)$  are not available, one has to estimate the smoothed confidence values as per Theorem 3.1, i.e., by properly lower-bound (or upper-bound) the confidence values. More concretely, one is required to obtain (a)  $\underline{p}_{\hat{f}_{\sigma_k}}(\mathbf{x})$  at the halting stage  $k$ , and (b)  $\overline{p}_{\hat{f}_{\sigma_{k'}}}(\mathbf{x},y)$  for  $k' = k + 1,\dots ,K$  and  $y\in \mathcal{V}$ .
For both estimations, we use Monte Carlo algorithm with  $n$  i.i.d. Gaussian samples, say  $\{\delta_i\}$ , and perform a majority voting across  $\{f_{\sigma}(\mathbf{x} + \boldsymbol{\delta}_i)\}$  to obtain a histogram on  $\mathcal{V}$  of  $n$  trials:
(a) To estimate a lower-bound  $\underline{p}_{\hat{f}_{\sigma_k}}(\mathbf{x})$ , we follow the statistical procedure by Cohen et al. [15], namely  $\mathrm{CERTIFY}(n, n_0, \alpha)$ , which additionally takes two hyper-parameters  $n_0$  and  $\alpha$ : here,  $n_0$  denotes the number of samples to initially make a guess to estimate a smoothed prediction, and  $\alpha$  is the significance level of the Clopper-Pearson confidence interval [14]. Concretely, it sets  $\underline{p}_{\hat{f}_{\sigma_k}}$  as the lower confidence level of  $p_{\hat{f}_{\sigma_k}}$  with coverage  $1 - \alpha$ .
(b) For  $\overline{p}_{\hat{f}_{\sigma_{k'}}}(\mathbf{x},y)$ , on the other hand, we first obtain voting counts from  $n$  Gaussian trials, say  $\mathbf{c}_{\mathbf{x}}^{(\sigma_{k'})}$ , and adopt Goodman confidence interval to bound multinomial proportions [24] with the significance level  $\alpha$ . Specifically, one can set  $\overline{p}_{\hat{f}_{\sigma_{k'}}}(\mathbf{x},y)$  for  $y \in \mathcal{V}$  by the  $(1 - \alpha)$ -upper level of the (multinomial) confidence interval. This can be implemented by using, e.g., multinomial_proportions_confint of the statsmodels library.
We make an additional treatment on the significance level  $\alpha$ : unless the pipeline halts at the first stage  $K$ , cascaded smoothing makes a  $(K - k + 1)$ -consecutive testing of confidence intervals to make a prediction. In attempt to maintain the overall significance level of this prediction as  $\alpha$ , one can apply the Bonferroni correction [5] on the significance level of individual testing, specifically by considering  $\frac{\alpha}{(K - k + 1)}$  as the adjusted significance level (instead of using  $\alpha$ ) at the stage  $k$ .
E Other Possible Designs
In this appendix, we present two alternative designs of multi-scale randomized smoothing we have also considered other than cascaded smoothing proposed in the main text. Specifically, here we additionally propose two different policies of aggregating multiple smoothed classifiers, each of which dubbed as the (a) max-radius policy, and (b) focal smoothing, respectively. Overall, both design do provide valid certified guarantees combining the accuracy-robustness trade-offs across multiple smoothing factors, often attaining a better robustness even compared to cascaded smoothing when  $n$ , the number of noise samples, is large enough.
The common drawback in the both approaches here, however, is the necessity to access every smoothed confidence, which can be crucial in practice in both terms of efficiency and accuracy. Specifically, for given  $K$  smoothed classifiers, this implies the exact  $K$ -times increase in the inference cost, which makes it much costly on average compared to cascaded smoothing. The need to access and compare with  $K$  smoothed confidence in practice also demands more accurate estimations of the confidence values to avoid abstaining. For example, these approaches can significantly degrade the certified accuracy in practice where  $n$  cannot be large unlike the certification phase: which again makes our cascaded smoothing more favorable over the approaches presented here.
E.1 Max-radius policy
Consider  $K$  smoothed classifiers, say  $\hat{f}_{\sigma_1}, \dots, \hat{f}_{\sigma_K}: \mathcal{X} \to \mathcal{Y}$ , where  $0 < \sigma_1 < \dots < \sigma_K$ . The max-radius policy, say  $\max \mathbb{R}(\cdot)$ , can be another simple policy one can consider for given  $\{\hat{f}_{\sigma_i}\}$ :
$$
\max  \mathrm {R} (\mathbf {x}; \left\{\hat {f} _ {\sigma_ {i}} \right\} _ {i = 1} ^ {K}) := \hat {f} _ {\sigma^ {*}} (\mathbf {x}), \text {w h e r e} \sigma^ {*} := \underset {\sigma_ {i}} {\arg \max } \sigma_ {i} \cdot \Phi^ {- 1} \left(p _ {\hat {f} _ {\sigma_ {i}}} (\mathbf {x})\right). \tag {19}
$$
Again, given the Lipschitzedness associated with the smoothed confidence  $p_{\hat{f}_{\sigma}}(\mathbf{x},y)$ , it is easy to check the following robustness certificate for  $\max R(\mathbf{x};\{\hat{f}_{\sigma_i}\}_{i = 1}^K)$ :
Theorem E.1. Let  $\hat{f}_{\sigma_1}, \dots, \hat{f}_{\sigma_K}: \mathcal{X} \to \mathcal{Y}$  be smoothed classifiers with  $0 < \sigma_1 < \dots < \sigma_K$ . If  $\max R(\mathbf{x}; \{\hat{f}_{\sigma_i}\}_{i=1}^K) =: \hat{y} \in \mathcal{Y}$ , it holds that  $\max R(\mathbf{x} + \boldsymbol{\delta}; \{\hat{f}_{\sigma_i}\}_{i=1}^K) = \hat{y}$  for  $\|\boldsymbol{\delta}\| < R$ , where:
$$
R := \frac {1}{2} \cdot \left(\sigma^ {*} \cdot \Phi^ {- 1} \left(p _ {\hat {f} _ {\sigma^ {*}}} (\mathbf {x})\right) - \max  _ {i = 1, \dots , K} \sigma_ {i} \cdot \Phi^ {- 1} \left(p _ {\hat {f} _ {\sigma_ {i}}} (\mathbf {x}; y)\right)\right). \tag {20}
$$
Remark that the certified radius given by (20) recovers the guarantee of Cohen et al. [15] if  $K = 1$ .
E.2 Focal smoothing
In this method, coined focal smoothing, we take a somewhat different approach to achieve multi-scale smoothing. Here, now we assume two different levels of smoothing factor, say  $\sigma_0 > \sigma_1 > 0$ , and consider inputs that contains noise with spatially different magnitudes. Specifically, we consider a binary mask  $M \in \{0,1\}^d$  and an input corrupted by the following:
$$
\operatorname {f o c a l} (\mathbf {x}, M; \sigma_ {0}, \sigma_ {1}) := \mathbf {x} + \left(\sigma_ {0} \cdot (1 - M) + \sigma_ {1} \cdot M\right) \odot \boldsymbol {\epsilon}, \tag {21}
$$
where  $\odot$  denotes the element-wise product and  $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ . In this way, each of corrupted input  $\hat{\mathbf{x}}$  gets a "hint" to recover the original content of  $\mathbf{x}$  by observing a less-corrupted region marked by  $M$ .
Randomized smoothing over such anisotropic noise has been recently studied [22, 20]: in a nutshell, smoothing over noises specified by (21) could produce an ellipsoid-shaped certified region with two axes of length  $\sigma_0\Phi^{-1}(p)$  and  $\sigma_{1}\Phi^{-1}(p)$  for its smoothed confidence  $p$ , respectively.
In terms of certified radius, however, this implies that one can still certify only the region limited by the shorter axes (of length  $\sigma_{1}\Phi^{-1}(p)$ ). It turns out that such a limitation can be avoided by considering an "ensemble" of  $K$  multiple orthogonal masks, say  $M_1,\dots ,M_K$ , so that the shorter axes of ellipsoids from the masks do not overlap each other. Specifically, we observe the following:
Theorem E.2. Let  $M_1, \dots, M_K \in \{0, 1\}^d$  be  $K$  orthogonal masks and  $\sigma_0 > \sigma_1 > 0$  be smoothing factors, and  $p_k(\mathbf{x})$  be the smoothed confidence from inputs corrupted by  $\text{focal}(\mathbf{x}, M_k; \sigma_0, \sigma_1)$ , for a given binary classifier  $f_k: \mathbb{R}^d \to [0, 1]$ . Define  $\hat{p} := \frac{1}{K} \sum_{k} p_k$ , and suppose  $\hat{p}(\mathbf{x}) > 0.5$ . Then, it is certified that  $\hat{p}(\mathbf{x} + \boldsymbol{\delta}) > 0.5$  for  $\| \boldsymbol{\delta} \| < R_{\mathbf{b}^*}$ , where  $\mathbf{b}^*$  is the solution of the following optimization:
$$
\underset {\mathbf {b}} {\text {m i n i m i z e}} \quad R _ {\mathbf {b}} := \frac {1}{K} \sum_ {k} \frac {a _ {k}}{\sqrt {t ^ {2} b _ {k} + 1}}
$$
$$
\text {s u b j e c t} \quad \sum_ {k} b _ {k} = 1, \tag {22}
$$
$$
b _ {k} \geq 0, k = 1, 2, \dots , K,
$$
where  $a_{k} \coloneqq \sigma_{0}\Phi^{-1}(p_{k})$  for  $k = 1, \dots, K$ , and  $t^2 \coloneqq \left(\frac{\sigma_0^2}{\sigma_1^2} - 1\right)$ .
In Algorithm 1, we propose an efficient 1-dimensional grid-search based optimization to solve (22), based on observations given as Lemma E.3 and E.4: each of which identifies the condition of  $b_{k}$  when it attains either the maximum value or zero, respectively. By solving the optimization, one could obtain a certified radius  $R$  that the aggregated smoothing can guarantee consistent prediction.
Lemma E.3. Consider the constrained optimization (22) with respect to  $\mathbf{b} = (b_{1},\dots ,b_{K})$ . Let  $M_{\mathbf{a}}\coloneqq \{m\in [K]:a_m = \max_ka_k\}$  and  $\mathbf{b}^*$  be an optimal solution of (22). Then, it holds that  $b_{m}^{*} = b_{m^{\prime}}^{*} > 0$  for any  $m,m^{\prime}\in M_{\mathbf{a}}$ .
Proof. First, we show that there exists at least one  $m \in M_{\mathbf{a}}$  where  $b_m^*$  is nonzero, i.e.,  $b_m^* > 0$ . Suppose the contrary, i.e.,  $b_k^* = 0$  for all  $k \in M_{\mathbf{a}}$ . Denote the corresponding objective value as  $R^*$ . Given the condition  $\sum_{k} b_k = 1$ , there always exists  $i \notin M$  where  $b_i > 0$ . Now, consider swapping the allocations of  $b_m$  and  $b_i$  for some  $m \in M_{\mathbf{a}}$  and denote the corresponding objective value  $R'$ .
Algorithm 1 Focal smoothing: A grid-search based optimization of (22)
Require:  $a_{k} > 0, k = 1,2,\ldots ,K$  tolerance  $\varepsilon >0$    
Ensure: optimal budget assignment  $\mathbf{b}^*$  , certified radius  $R^{*}$    
1: Find the maximum value  $a_{\mathrm{max}} = \mathrm{max}_k a_k$    
2: Identify the index set  $M$  such that  $a_{k} = a_{\mathrm{max}}$  for  $k\in M$    
3: Set a grid of values for  $b_{m}$  in the interval (0,1].   
4: for each grid value of  $b_{m}$  do   
5: for  $k = 1,2,\dots ,K$  do   
6: if  $k\notin M$  then   
7:  $b_{k}\gets 0$  if  $a_{k}\leq \frac{a_{m}}{(t^{2}b_{m} + 1)^{3 / 2}}$  else  $\frac{1}{t^2}\left(\left(\frac{a_k}{a_m}\right)^{2 / 3}(t^2 b_m + 1) - 1\right)$    
8: end if   
9: end for   
10: if  $\begin{array}{r}\mid \sum_{k = 1}^{K}b_{k} - 1\mid \leq \varepsilon \end{array}$  then   
11:  $R\gets \frac{1}{K}\sum_{k = 1}^{K}\frac{a_k}{\sqrt{t^2b_k + 1}}$    
12: Update  $\mathbf{b}_k^*,R^*\gets b_k,R$  if  $R < R^{*}$    
13: end if   
14: end for
$$
\begin{array}{l} R ^ {\prime} - R ^ {*} = \frac {1}{K} \left(\frac {a _ {k}}{\sqrt {t ^ {2} b _ {j} + 1}} + \frac {a _ {j}}{\sqrt {t ^ {2} b _ {k} + 1}} - \frac {a _ {k}}{\sqrt {t ^ {2} b _ {k} + 1}} - \frac {a _ {j}}{\sqrt {t ^ {2} b _ {j} + 1}}\right) \\ = \frac {1}{K} \left(a _ {k} \left(\frac {1}{\sqrt {t ^ {2} b _ {j} + 1}} - \frac {1}{\sqrt {t ^ {2} b _ {k} + 1}}\right) + a _ {j} \left(\frac {1}{\sqrt {t ^ {2} b _ {k} + 1}} - \frac {1}{\sqrt {t ^ {2} b _ {j} + 1}}\right)\right) \\ <   \frac {1}{K} \left(a _ {k} \left(\frac {1}{\sqrt {t ^ {2} b _ {j} + 1}} - \frac {1}{\sqrt {t ^ {2} b _ {k} + 1}}\right) + a _ {k} \left(\frac {1}{\sqrt {t ^ {2} b _ {k} + 1}} - \frac {1}{\sqrt {t ^ {2} b _ {j} + 1}}\right)\right) = 0, \tag {23} \\ \end{array}
$$
given that  $a_{k} > a_{j}$  and  $\frac{1}{\sqrt{t^2b_j + 1}} < \frac{1}{\sqrt{t^2b_k + 1}} = 1$ . This means  $R' < R^{*}$ , contradicting the optimality of the initial solution  $\mathbf{b}^{*}$  with  $b_{k} = 0$  for all  $k \in M_{\mathbf{a}}$ . Thus, there must exist at least one  $b_{m}^{*} > 0$  for some  $m \in M_{\mathbf{a}}$ .
Next, we show  $b_{m}^{*} = b_{m^{\prime}}^{*}$  for any  $m, m^{\prime} \in M_{\mathbf{a}}$ . Suppose there exist such  $m, m^{\prime} \in M_{\mathbf{a}}$  where  $b_{m}^{*} \neq b_{m^{\prime}}^{*}$ . Due to the symmetry, one can construct another optimal allocation  $\mathbf{b}^{\prime}$  by swapping the two allocations  $b_{m}^{*}$  and  $b_{m^{\prime}}^{*}$ , as it does not affect the optimality. Now consider another allocation  $\bar{\mathbf{b}}$  by averaging the two allocations:
$$
\bar {b} _ {m} = \bar {b} _ {m ^ {\prime}} = \frac {b _ {m} ^ {*} + b _ {m ^ {\prime}} ^ {*}}{2}. \tag {24}
$$
Let the corresponding objective value  $\bar{R}$ . Note that  $R$  is strictly convex, which it implies that  $\bar{R} < R^{*}$ , contradicting the optimality of the initial solution  $\mathbf{b}^{*}$ . Thus, it holds  $b_{m}^{*} = b_{m^{\prime}}^{*}$  for any  $m, m^{\prime} \in M_{\mathbf{a}}$ .
Lemma E.4. Consider the constrained optimization (22) with  $\mathbf{b}^*$  as an optimal solution. Denote  $B := b_m^* = b_{m'}^* > 0$  for all  $m, m' \in M_{\mathbf{a}}$ . Then,  $b_k^* = 0$  if and only if  $a_k \leq \frac{A}{(t^2 B + 1)^{3/2}}$ .
Proof. Suppose  $b_{k}^{*} = 0$ . From the Karush-Kuhn-Tucker (KKT) stationarity condition, we have  $\frac{-a_k t^2}{K} + \lambda - \mu_k = 0$ . Since  $\lambda, \mu_k \geq 0$ , we obtain  $\lambda \geq \frac{a_k t^2}{K}$ . Now, consider the stationarity condition for  $m \in M$ :  $-\frac{A t^2}{K(\sqrt{t^2 B + 1})^3} + \lambda = 0$ . Substituting the lower bound for  $\lambda$ , we have  $a_k \leq \frac{A}{(t^2 B + 1)^{3/2}}$ . Conversely, suppose  $a_k \leq \frac{A}{(t^2 B + 1)^{3/2}}$ . Then,  $b_{k}^{*} > 0$  contradict to that  $b_{k}^{*} = \frac{1}{t^2}\left(\left(\frac{a_k}{A}\right)^{2/3}(t^2 B + 1) - 1\right) \leq 0$ , hence  $b_{k}^{*} = 0$ .