SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Binary-class Robustness Certificates for Hierarchical Randomized Smoothing
Input:  $f, X, p$ , Lower-level smoothing distribution  $\mu$ , Radius  $(r, \epsilon)$ ,  $n_0, n_1, 1 - \alpha$   
counts0  $\leftarrow$  SampleUnderNoise  $(f, X, p, \mu, n_0)$ ;  
counts1  $\leftarrow$  SampleUnderNoise  $(f, X, p, \mu, n_1)$ ;  
 $y_A \leftarrow$  top index in counts0;  
 $\underline{p_A} \leftarrow$  ConfidenceBound(counts1  $[y_A]$ ,  $n_1$ ,  $1 - \alpha$ );  
 $\Delta \leftarrow 1 - p^r$ ;  
 $\tilde{p}_A \leftarrow$  LowerLevelWorstCaseLowerBound  $(\mu, \epsilon, \frac{p_A - \Delta}{1 - \Delta})$ ;  
if  $\tilde{p}_A \cdot (1 - \Delta) > \frac{1}{2}$  then  
| Return:  $y_A$  certified  
end  
Return: ABSTAIN
Here, LowerLevel Worst Case Lower Bound  $(\mu, \epsilon, \underline{p_A})$  computes the existing certificate, specifically the smallest  $p_{\tilde{X},y_A}$  over the entire threat model, e.g.  $\Phi\left(\Phi^{-1}(\underline{p_A}) - \frac{\epsilon}{\sigma}\right)$  for Gaussian smoothing. The remaining certification procedure is analogous to the certification in (Cohen et al., 2019).
C.1 Proofs of Section 5
First, define the space  $\mathcal{Z}^{N\times (D + 1)}\triangleq \mathcal{X}^{N\times D}\times \{0,1\} ^N$  and recall the definition of the reachable set  $\mathcal{A}\triangleq \mathcal{A}_1\cup \mathcal{A}_2$  with
$$
\mathcal {A} _ {1} \triangleq \left\{\boldsymbol {Z} \in \mathcal {Z} ^ {N \times (D + 1)} \mid \forall i: (\boldsymbol {\tau} _ {i} = 0) \Rightarrow (\boldsymbol {z} _ {i} = \boldsymbol {x} _ {i} | 0) \right\},
$$
$$
\mathcal {A} _ {2} \triangleq \left\{\boldsymbol {Z} \in \mathcal {Z} ^ {N \times (D + 1)} \mid \forall i: (\boldsymbol {\tau} _ {i} = 0) \Rightarrow (\boldsymbol {z} _ {i} = \tilde {\boldsymbol {x}} _ {i} | 0) \right\}.
$$
Intuitively,  $\mathcal{A}$  contains only those  $Z$  that can be sampled from either  $\Psi_{X}$  or  $\Psi_{\tilde{X}}$  (given the lower-level smoothing distribution has infinite support). Note that these two sets are not necessarily disjoint, so we cannot apply the Neyman-Pearson Lemma for discrete random variables (Tocher, 1950) on the upper-level smoothing distribution  $\phi$  yet. The following Proposition therefore provides a partitioning of the space  $\mathcal{Z}^{N\times (D + 1)}$ :
$\mathcal{R}_0 \triangleq \mathcal{Z}^{N \times (D + 1)} \setminus \mathcal{A}$
-  $\mathcal{R}_3 \triangleq \{Z \in \mathcal{A} \mid \exists i \in \mathcal{C}: \pmb{\tau}_i = 0, \pmb{z}_i = \tilde{\pmb{x}}_i | 0\}$
Proof. We want to show that the four sets above (1) are pairwise disjoint, and (2) partition  $\mathcal{Z}^{N\times (D + 1)}$ , that is  $\mathcal{R}_0\uplus \mathcal{R}_1\uplus \mathcal{R}_2\uplus \mathcal{R}_3 = \mathcal{Z}^{N\times (D + 1)}$ .
Clearly we have  $\mathcal{Z}^{N\times (D + 1)} = \mathcal{R}_0\uplus \mathcal{A}$  due to the definition of the region  $\mathcal{R}_0$ . Thus we only have to show  $\mathcal{R}_1\uplus \mathcal{R}_2\uplus \mathcal{R}_3 = \mathcal{A}$ :
(1) We show  $\mathcal{R}_1, \mathcal{R}_2, \mathcal{R}_3$  are pairwise disjoint:
We have  $\mathcal{R}_2\cap \mathcal{R}_1 = \emptyset$  and  $\mathcal{R}_2\cap \mathcal{R}_3 = \emptyset$  since in region  $\mathcal{R}_2$  we have  $\pmb {\tau}_i = 1$  for all  $i\in \mathcal{C}$  which does not hold for  $\mathcal{R}_2$  or  $\mathcal{R}_3$
Moreover,  $\mathcal{R}_1\cap \mathcal{R}_3 = \emptyset$  because if  $Z\in \mathcal{R}_1$  then  $Z\notin \mathcal{R}_3$  (and vice versa) due to the definition of  $\mathcal{A}$  and  $\mathcal{R}_1\subseteq \mathcal{A}_1$ $\mathcal{R}_3\subseteq \mathcal{A}_3$
(2) We show  $\mathcal{R}_1\cup \mathcal{R}_2\cup \mathcal{R}_3 = \mathcal{A}$
"  $\subseteq$  " Clearly  $\mathcal{R}_1\subseteq \mathcal{A},\mathcal{R}_2\subseteq \mathcal{A}$  , and  $\mathcal{R}_3\subseteq \mathcal{A}$  by definition of the regions.
"≧": Consider any  $Z \in \mathcal{A} = \mathcal{A}_1 \cup \mathcal{A}_2$ . Either (1) all perturbed rows are selected and  $Z \in \mathcal{R}_2$  or (2) at least one perturbed row is not selected ( $\exists i: \tau_i = 0$ ), in which case we either have  $Z \in \mathcal{R}_1$  or  $Z \in \mathcal{R}_3$ , depending on whether  $Z \in \mathcal{A}_1$ ,  $Z \in \mathcal{A}_3$ , respectively.
We further have to derive the probability to sample  $Z$  of each region from both  $\Psi_{X}$  and  $\Psi_{\bar{X}}$  for applying the Neyman-Pearson Lemma (Tocher, 1950) for discrete random variables on the upper-level smoothing distribution  $\phi$  that selects rows before adding noise.
Proposition 2. Given  $\Psi_{\pmb{X}}$  and  $\Psi_{\widetilde{X}}$ , the regions of Proposition 1 and  $\Delta \triangleq 1 - p^{|C|}$  we have
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {1} ] = \Delta \quad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {2} ] = 1 - \Delta \quad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {3} ] = 0
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {1} ] = 0 \qquad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {2} ] = 1 - \Delta \qquad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {3} ] = \Delta
$$
Proof. The probability to select all  $|\mathcal{C}|$  perturbed rows is  $p^r$  since we sample each  $\pmb{\tau}_i$  independently from a Bernoulli:  $\tau_i \stackrel{iid}{\sim} \operatorname{Ber}(p)$ . Consequently the probability to not select at least one perturbed row is  $1 - p^r = \Delta$ . The remaining follows from the fact that matrices from region  $\mathcal{R}_1$  cannot be sampled from  $\Psi_{\tilde{X}}$  and matrices from  $\mathcal{R}_3$  cannot be sampled from  $\Psi_X$ .
Note that we need region  $\mathcal{R}_0$  just for a complete partitioning of  $\mathcal{Z}^{N\times (D + 1)}$ . This region contains essentially all matrices we cannot sample neither from  $\Psi_{X}$  nor  $\Psi_{\tilde{X}}$ . For example, matrices with a row that differs from the same row in  $X$  and  $\tilde{X}$  without a previous selection of that row cannot be sampled from the two distributions - simply because we first select rows and then we only add noise to the rows that we selected. Thus we have  $\operatorname*{Pr}_{Z\sim \Psi_X}[Z\in \mathcal{R}_0] = 0$  and  $\operatorname*{Pr}_{Z\sim \Psi_{\tilde{X}}}[Z\in \mathcal{R}_0] = 0$ .
Before we derive the lower bound on  $p_{\tilde{X},y}$  we first restate the Neyman-Pearson Lemma as presented in Section 3:
$$
\underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \underset {\boldsymbol {W} \sim \mu_ {\tilde {\boldsymbol {X}}}} {\operatorname * {P r}} [ \boldsymbol {W} \in S _ {\kappa} ] \quad w i t h \kappa \in \mathbb {R} _ {+} s. t. \quad \underset {\boldsymbol {W} \sim \mu_ {\boldsymbol {X}}} {\operatorname * {P r}} [ \boldsymbol {W} \in S _ {\kappa} ] = p _ {\boldsymbol {X}, y}
$$
There are different views on Lemma 1. In the context of statistical hypothesis testing, Lemma 1 states that the likelihood ratio test is the uniformly most powerful test when testing the simple hypothesis  $\mu_{\mathbf{X}}(\mathbf{W})$  against  $\mu_{\tilde{\mathbf{X}}}(\mathbf{W})$ . For such views we refer to Cohen et al. (2019).
Now we prove our main theorem:
Theorem 1 (Neyman-Pearson lower bound for hierarchical smoothing). Given fixed  $\pmb{X}$ ,  $\tilde{\pmb{X}} \in \mathcal{X}^{N\times D}$  let  $\mu_{\pmb{X}}$  denote a smoothing distribution that operates independently on matrix elements, and  $\Psi_{\pmb{X}}$  the induced hierarchical distribution over  $\mathcal{Z}^{N\times (D + 1)}$ . Given label  $y \in \mathcal{V}$  and the probability  $p_{\pmb{X},y}$  to classify  $\pmb{X}$  as  $y$  under  $\Psi$ . Define  $\hat{S}_{\kappa} \triangleq \left\{\pmb{W} \in \mathcal{X}^{|C| \times D}: \mu_{\tilde{\pmb{X}}_C}(\pmb{W}) \leq \kappa \cdot \mu_{\pmb{X}_C}(\pmb{W})\right\}$ . We have:
Proof. We are solving the following optimization problem:
$$
\underline {{\underline {{p}}}} _ {\tilde {\boldsymbol {X}}, y} \triangleq \min  _ {h \in \mathbb {H}} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ h (\boldsymbol {Z}) = y ] \quad s. t. \quad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ h (\boldsymbol {Z}) = y ] = p _ {\boldsymbol {X}, y}
$$
We can use the partitioning of Proposition 1 and apply the law of total probability:
$$
\begin{array}{l} \underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \min  _ {h \in \mathbb {H}} \sum_ {i = 0} ^ {3} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {i} ] \cdot \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {i} ] \\ \text {s . t .} \quad \sum_ {i = 0} ^ {3} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {i} ] \cdot \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {i} ] = p _ {\boldsymbol {X}, y} \\ \end{array}
$$
We apply Proposition 2 and cancel out all zero probabilities:
$$
\begin{array}{l} \underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \min  _ {h \in \mathbb {H}} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\hat {\boldsymbol {X}}}} {\operatorname * {P r}} \left[ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} \right] \cdot (1 - \Delta) + \underset {\boldsymbol {Z} \sim \Psi_ {\hat {\boldsymbol {X}}}} {\operatorname * {P r}} \left[ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {3} \right] \cdot \Delta \\ \mathrm {s . t .} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) + \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {1} ] \cdot \Delta = p _ {\boldsymbol {X}, y} \\ \end{array}
$$
This is a minimization problem and to minimize we choose  $\operatorname{Pr}_{\mathbf{Z} \sim \Psi_{\tilde{\mathbf{X}}}}[h(\mathbf{Z}) = y \mid \mathbf{Z} \in \mathcal{R}_3] = 0$  and  $\operatorname{Pr}_{\mathbf{Z} \sim \Psi_{\mathbf{X}}}[h(\mathbf{Z}) = y \mid \mathbf{Z} \in \mathcal{R}_1] = 1$ . In other words, in the worst-case all matrices in  $\mathcal{R}_1$  that we never observe around  $\tilde{\mathbf{X}}$  are correctly classified and the worst-case classifier first uses the budget  $p_{\mathbf{X},y}$  on region  $\mathcal{R}_1$ .
We obtain:
$$
\begin{array}{l} \underline {{\underline {{p}} _ {\tilde {\boldsymbol {X}} , y}}} = \min _ {h \in \mathbb {H}} \quad \underset {\boldsymbol {Z} \sim \boldsymbol {\Psi} _ {\tilde {\boldsymbol {X}}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) \\ \text {s . t .} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) = p _ {\boldsymbol {X}, y} - \Delta \\ \end{array}
$$
Here we can see that for  $p = 0$  (i.e.  $\Delta = 1$ ) we have just  $\underline{p_{\tilde{X},y}} = 0$ . We can assume  $p > 0$  in the following.
We can further rearrange the terms:
$$
\begin{array}{l} \underline {{p _ {\boldsymbol {\tilde {X}} , y}}} = (1 - \Delta) \cdot \min _ {h \in \mathbb {H}} \quad \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {\tilde {X}}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \\ \text {s . t .} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] = \frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta} \\ \end{array}
$$
Now we can apply Lemma 1:
$$
\underline {{p _ {\tilde {X} , y}}} = \underset {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} {\operatorname * {P r}} \left[ \boldsymbol {Z} \in S _ {\kappa} \mid \boldsymbol {Z} \in \mathcal {R} _ {2} \right] \cdot (1 - \Delta)
$$
$$
\text {w i t h} \kappa \in \mathbb {R} _ {+} \text {s . t .} \tag {2}
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in S _ {\kappa} \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] = \frac {p _ {\boldsymbol {X} , y} - \Delta}{(1 - \Delta)}
$$
for  $S_{\kappa} \triangleq \{Z \in \mathcal{Z}^{N \times (D + 1)} : \Psi_{\tilde{X}}(Z) \leq \kappa \cdot \Psi_X(Z)\}$ . Note the difference between  $S_{\kappa}$  and  $\hat{S}_{\kappa}$ .
We need to realize that due to independence we have:
$$
\begin{array}{l} \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \Psi_ {\vec {X}} (\boldsymbol {Z}) \leq \kappa \cdot \Psi_ {\boldsymbol {X}} (\boldsymbol {Z}) \right\} \\ = \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \Psi_ {\hat {\boldsymbol {X}}} (\boldsymbol {W}, \tau) \leq \kappa \cdot \Psi_ {\boldsymbol {X}} (\boldsymbol {W}, \tau) \right\} \\ = \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \mu_ {\boldsymbol {X}} (\boldsymbol {W} | \tau) \phi (\tau) \leq \kappa \cdot \mu_ {\boldsymbol {X}} (\boldsymbol {W} | \tau) \phi (\tau) \right\} \\ = \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \prod_ {i = 1} ^ {N} \mu_ {\tilde {\boldsymbol {x}} _ {i}} \left(\boldsymbol {w} _ {i} \mid \boldsymbol {\tau} _ {i}\right) \leq \kappa \cdot \prod_ {i = 1} ^ {N} \mu_ {\boldsymbol {x} _ {i}} \left(\boldsymbol {w} _ {i} \mid \boldsymbol {\tau} _ {i}\right) \right\} \tag {3} \\ \stackrel {(*)} {=} \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \prod_ {i \in \mathcal {C}} \mu_ {\tilde {\boldsymbol {x}} _ {i}} (\boldsymbol {w} _ {i}) \leq \kappa \cdot \prod_ {i \in \mathcal {C}} \mu_ {\boldsymbol {x} _ {i}} (\boldsymbol {w} _ {i}) \right\} \\ = \left\{\boldsymbol {Z} \in \mathcal {R} _ {2}: \mu_ {\tilde {\boldsymbol {X}} _ {\mathcal {C}}} (\boldsymbol {W} _ {\mathcal {C}}) \leq \kappa \cdot \mu_ {\boldsymbol {X} _ {\mathcal {C}}} (\boldsymbol {W} _ {\mathcal {C}}) \right\} \\ \end{array}
$$
where  $(^{*})$  is due to  $\pmb {x}_i = \tilde{\pmb{x}}_i$  for all  $i\notin \mathcal{C}$  and  $\pmb {\tau}_i = 1$  for  $i\in \mathcal{C}$
In other words this means that if  $Z \in S_{\kappa} \cap \mathcal{R}_2$  then all  $Z' \in \mathcal{R}_2$  that differ only in rows  $i \notin \mathcal{C}$  from  $Z$  are also in  $S_{\kappa} \cap \mathcal{R}_2$  and integrate out (the "likelihood ratio" is only determined by rows  $i \in \mathcal{C}$ ).
It follows that for fixed  $\kappa \in \mathbb{R}_{+}$  we have
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} \left[ \boldsymbol {Z} \in S _ {\kappa} \mid \boldsymbol {Z} \in \mathcal {R} _ {2} \right] = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\tilde {\boldsymbol {X}} _ {\mathcal {C}}}} \left[ \boldsymbol {W} \in \hat {S} _ {\kappa} \right]
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in S _ {\kappa} \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X} _ {\mathcal {C}}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ]
$$
for  $\hat{S}_{\kappa} \triangleq \left\{\mathbf{W} \in \mathcal{X}^{|\mathcal{C}| \times D}: \mu_{\tilde{\mathbf{X}}_{\mathcal{C}}}(\mathbf{W}) \leq \kappa \cdot \mu_{\mathbf{X}_{\mathcal{C}}}(\mathbf{W})\right\}$ . All together we obtain:
$$
\underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {\tilde {X}} _ {\mathcal {C}}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] \cdot (1 - \Delta) \quad \mathrm {w i t h} \kappa \in \mathbb {R} _ {+} \mathrm {s . t .} \quad \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X} _ {\mathcal {C}}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] = \frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}
$$
D Multi-class certificates for hierarchical randomized smoothing
In the main section of our paper we only derive so-called binary-class certificates: If  $p_{\tilde{X},y^*} > 0.5$ , then the smoothed classifier  $g$  is certifiably robust. We can derive a tighter certificate by guaranteeing  $p_{\tilde{X},y^*} > \max_{y \neq y^*} \overline{p_{\tilde{X},y}}$ . To this end, we derive an upper bound  $p_{\tilde{X},y} \leq \overline{p_{\tilde{X},y}}$  in the following:
$$
\overline {{p _ {\bar {\boldsymbol {X}} , y}}} \triangleq \max _ {h \in \mathbb {H}} \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\bar {\boldsymbol {X}}}} [ h (\boldsymbol {W}) = y ] s. t. \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X}}} [ h (\boldsymbol {W}) = y ] = p _ {\boldsymbol {X}, y} (4)
$$
Cohen et al. (2019) show that the maximum of Equation 4 can be obtained by using following lemma:
Lemma 3 (Neyman-Pearson upper bound). Given  $\mathbf{X}$ ,  $\tilde{\mathbf{X}} \in \mathcal{X}^{N\times D}$ , distributions  $\mu_{\mathbf{X}}, \mu_{\tilde{\mathbf{X}}}$ , class label  $y \in \mathcal{Y}$ , probability  $p_{\mathbf{X},y}$  and the set  $S_{\kappa} \triangleq \left\{\mathbf{W} \in \mathcal{X}^{N\times D}: \mu_{\tilde{\mathbf{X}}}(\mathbf{W}) \geq \kappa \cdot \mu_{\mathbf{X}}(\mathbf{W})\right\}$  we have
$$
\overline {{p _ {\boldsymbol {\tilde {X}} , y}}} = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {\tilde {X}}}} \left[ \boldsymbol {W} \in S _ {\kappa} \right] \quad \text {w i t h} \kappa \in \mathbb {R} _ {+} \text {s . t .} \quad \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X}}} \left[ \boldsymbol {W} \in S _ {\kappa} \right] = p _ {\boldsymbol {X}, y}
$$
Proof. See (Neyman and Pearson, 1933; Cohen et al., 2019).
We can now use Lemma 3 to derive an upper bound under hierarchical smoothing:
Theorem 2 (Neyman-Pearson upper bound for hierarchical smoothing). Given fixed  $X$ ,  $\tilde{X} \in \mathcal{X}^{N \times D}$ , let  $\mu_{X}$  denote a smoothing distribution that operates independently on matrix elements, and  $\Psi_{X}$  the induced hierarchical distribution over  $\mathcal{Z}^{N \times (D + 1)}$ . Given label  $y \in \mathcal{Y}$  and the probability  $p_{X,y}$  to classify  $X$  as  $y$  under  $\Psi$ . Define  $\hat{S}_{\kappa} \triangleq \left\{\boldsymbol{W} \in \mathcal{X}^{r \times D}: \mu_{\tilde{\boldsymbol{X}}_C}(\boldsymbol{W}) \geq \kappa \cdot \mu_{\boldsymbol{X}_C}(\boldsymbol{W})\right\}$ . We have:
$$
\overline {{p _ {\boldsymbol {X} , y}}} = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {\bar {X}} _ {\mathcal {C}}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] \cdot (1 - \Delta) + \Delta \quad w i t h \kappa \in \mathbb {R} _ {+} s. t. \quad \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X} _ {\mathcal {C}}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] = \frac {p _ {\boldsymbol {X} , y}}{1 - \Delta}
$$
Proof. We are solving the following maximization problem:
$$
\overline {{p _ {\tilde {X} , y}}} \triangleq \max _ {h \in \mathbb {H}} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ h (\boldsymbol {Z}) = y ] s. t. \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ h (\boldsymbol {Z}) = y ] = p _ {\boldsymbol {X}, y}
$$
$$
\begin{array}{l} \overline {{p _ {\bar {\boldsymbol {X}} , y}}} = \max  _ {h \in \mathbb {H}} \sum_ {i = 0} ^ {3} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\bar {\boldsymbol {X}}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {i} ] \cdot \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\bar {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {i} ] \\ \text {s . t .} \quad \sum_ {i = 0} ^ {3} \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {i} ] \cdot \operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {i} ] = p _ {\boldsymbol {X}, y} \\ \end{array}
$$
$$
\begin{array}{l} \overline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \max _ {h \in \mathbb {H}} \underset {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y | \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) + \underset {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y | \boldsymbol {Z} \in \mathcal {R} _ {3} ] \cdot \Delta \\ \mathrm {s . t .} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y | \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) + \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y | \boldsymbol {Z} \in \mathcal {R} _ {1} ] \cdot \Delta = p _ {\boldsymbol {X}, y} \\ \end{array}
$$
This is a maximization problem and to maximize we choose  $\operatorname*{Pr}_{\mathbf{Z}\sim \Psi_{\tilde{\mathbf{X}}}}[h(\mathbf{Z}) = y\mid \mathbf{Z}\in \mathcal{R}_3] = 1$  and  $\operatorname*{Pr}_{\mathbf{Z}\sim \Psi_{\mathbf{X}}}[h(\mathbf{Z}) = y\mid \mathbf{Z}\in \mathcal{R}_1] = 0$ . In other words, all matrices in  $\mathcal{R}_3$  that we never observe around  $\mathbf{X}$  are correctly classified and the worst-case classifier does not use any budget  $p_{\mathbf{X},y}$  on this region  $\mathcal{R}_3$ .
$$
\begin{array}{l} \overline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \max  _ {h \in \mathbb {H}} \quad \Pr_ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) + \Delta \\ \text {s . t .} \quad \underset {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} {\operatorname * {P r}} [ h (\boldsymbol {Z}) = y \mid \boldsymbol {Z} \in \mathcal {R} _ {2} ] \cdot (1 - \Delta) = p _ {\boldsymbol {X}, y} \\ \end{array}
$$
The remaining statement follows analogous to the proof in Appendix C.
In this section we derived multi-class robustness certificates for randomized smoothing and showed the correctness of the following certification procedure:
Algorithm 2: Multi-class Robustness Certificates for Hierarchical Randomized Smoothing
Input:  $f, X, p$ , Lower-level smoothing distribution  $\mu$ , Radius  $(r, \epsilon)$ ,  $n_0, n_1, 1 - \alpha$   
counts0  $\leftarrow$  SampleUnderNoise  $(f, X, p, \mu, n_0)$ ;  
counts1  $\leftarrow$  SampleUnderNoise  $(f, X, p, \mu, n_1)$ ;  
 $y_A, y_B \leftarrow$  top two indices in counts0;  
 $p_A, \overline{p_B} \leftarrow$  ConfidenceBounds(counts1[yA], counts1[yB], n1, 1 -  $\alpha$ );  
 $\Delta \leftarrow 1 - p^r$ ;  
 $\overline{p_B} \leftarrow$  LowerLevelWorstCaseUpperBound( $\mu$ ,  $\epsilon$ ,  $\frac{p_B}{1 - \Delta}$ );  
 $\tilde{p}_A \leftarrow$  LowerLevelWorstCaseLowerBound( $\mu$ ,  $\epsilon$ ,  $\frac{p_A - \Delta}{1 - \Delta}$ );  
if  $\tilde{p}_A \cdot (1 - \Delta) > \overline{\tilde{p}_B} \cdot (1 - \Delta) + \Delta$  then  
| Return:  $y_A$  certified  
end  
Return: ABSTAIN
Here, LowerLevel Worst Case Upper Bound  $(\mu ,\epsilon ,\overline{p_B})$  computes the existing certificate, specifically the largest  $\overline{p_{\tilde{\mathbf{X}},y_B}}$  over the entire threat model, e.g.  $\Phi \left(\Phi^{-1}(\overline{p_B}) + \frac{\epsilon}{\sigma}\right)$  for Gaussian smoothing. The remaining certification procedure is analogous to the certification in (Cohen et al., 2019).
E Hierarchical randomized smoothing using Gaussian isotropic smoothing (Proofs of Section 6)
Having derived our general certification framework for hierarchical randomized smoothing, we can instantiate it with specific smoothing distributions to certify specific threat models. In the following we show the robustness guarantees for initializing hierarchical randomized smoothing with Gaussian isotropic smoothing, as already discussed in Section 6. We will also derive the corresponding multi-class robustness certificates.