SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 3. Given continuous  $\mathcal{X} \triangleq \mathbb{R}$ , consider the threat model  $\mathcal{B}_{2,\epsilon}^r(\boldsymbol{X})$  for fixed  $\boldsymbol{X} \in \mathcal{X}^{N \times D}$ . Initialize hierarchical smoothing with the isotropic Gaussian smoothing distribution defined as  $\mu_{\boldsymbol{X}}(\boldsymbol{W}) \triangleq \prod_{i=1}^{N} \mathcal{N}(\boldsymbol{w}_i | \boldsymbol{x}_i, \sigma^2 \boldsymbol{I})$ . Let  $y \in \mathcal{Y}$  denote a label and  $p_{\boldsymbol{X},y}$  the probability to classify  $\boldsymbol{X}$  as  $y$  under  $\Psi$ . Define  $\Delta \triangleq 1 - p^r$ . Then we have:
$$
\min _ {\tilde {\boldsymbol {X}} \in \mathcal {B} _ {2, \epsilon} ^ {r} (\boldsymbol {X})} \underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \Phi \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}\right) - \frac {\epsilon}{\sigma}\right) (1 - \Delta)
$$
Proof. As outlined in Subsection 3.3, Cohen et al. (2019) show that the optimal value of Equation 1 under Gaussian smoothing is  $\Phi\left(\Phi^{-1}(p_{x,y}) - \frac{||\delta||_2}{\sigma}\right)$ , where  $\Phi$  denotes the CDF of the normal distribution and  $\delta = x - \tilde{x}$ . Plugging this optimal value into Theorem 1 yields for any  $\tilde{\boldsymbol{X}} \in \mathcal{B}_{2,\epsilon}^r(\boldsymbol{X})$ :
$$
\underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \Phi \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}\right) - \frac {| | \boldsymbol {\delta} | | _ {2}}{\sigma}\right) (1 - \Delta)
$$
where  $\Delta = 1 - p^{|\mathcal{C}|}$  and  $\delta = \text{vec}(\boldsymbol{X} - \tilde{\boldsymbol{X}})$ . The remaining statements about the minimum over the entire threat model follow from the fact that  $\underline{p_{\tilde{X},y}}$  is monotonously decreasing in  $||\delta||_2$ , and monotonously decreasing in  $r$  (see also Proposition 3 in Section 5). Note that for fixed  $\tilde{X}$  we write  $\Delta = 1 - p^{|\mathcal{C}|}$  but for the entire threat model we have to use the largest  $\Delta = 1 - p^r$ .
Corollary 1. Given continuous  $\mathcal{X} \triangleq \mathbb{R}$ , consider the threat model  $\mathcal{B}_{2,\epsilon}^r(\boldsymbol{X})$  for fixed  $\boldsymbol{X} \in \mathcal{X}^{N \times D}$ . Initialize the hierarchical smoothing distribution  $\Psi$  using the isotropic Gaussian distribution  $\mu_{\boldsymbol{X}}(\boldsymbol{W}) \triangleq \prod_{i=1}^{N} \mathcal{N}(\boldsymbol{w}_i | \boldsymbol{x}_i, \sigma^2 \boldsymbol{I})$  that applies noise independently on each matrix element. Let  $y^* \in \mathcal{Y}$  denote the majority class and  $p_{\boldsymbol{X}, y^*}$  the probability to classify  $\boldsymbol{X}$  as  $y^*$  under  $\Psi$ . Then the smoothed classifier  $g$  is certifiably robust  $g(\boldsymbol{X}) = g(\tilde{\boldsymbol{X}})$  for any  $\tilde{\boldsymbol{X}} \in \mathcal{B}_{2,\epsilon}^r(\boldsymbol{X})$  if
Proof. For the binary-class certificate, the smoothed classifier  $g$  is certifiably robust for fixed input  $X$  if  $\min_{\tilde{X} \in \mathcal{B}_{2,\epsilon}^r(X)} p_{\tilde{X},y^*} > 0.5$ . By using Corollary 3 we have
$$
\begin{array}{l} \min _ {\tilde {\boldsymbol {X}} \in \mathcal {B} _ {2, \epsilon} ^ {r} (\boldsymbol {X})} \frac {p _ {\tilde {\boldsymbol {X}} , \boldsymbol {y} ^ {*}}}{\mathrm {一}} > \frac {1}{2} \\ \Leftrightarrow \Phi \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y ^ {*}} - \Delta}{1 - \Delta}\right) - \frac {\epsilon}{\sigma}\right) (1 - \Delta) > \frac {1}{2} \\ \Leftrightarrow \Phi^ {- 1} \left(\frac {p _ {\mathbf {X} , y ^ {*}} - \Delta}{1 - \Delta}\right) - \frac {\epsilon}{\sigma} > \Phi^ {- 1} \left(\frac {1}{2 (1 - \Delta)}\right) \\ \Leftrightarrow \epsilon <   \sigma \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y ^ {*} - \Delta}}{1 - \Delta}\right) - \Phi^ {- 1} \left(\frac {1}{2 (1 - \Delta)}\right)\right) \\ \end{array}
$$
In the following we additionally derive the corresponding multi-class robustness certificates:
Corollary 4. Given continuous  $\mathcal{X} \triangleq \mathbb{R}$ , consider the threat model  $\mathcal{B}_{2,\epsilon}^r(\boldsymbol{X})$  for fixed  $\boldsymbol{X} \in \mathcal{X}^{N \times D}$ . Initialize hierarchical smoothing with the isotropic Gaussian smoothing distribution defined as  $\mu_{\boldsymbol{X}}(\boldsymbol{W}) \triangleq \prod_{i=1}^{N} \mathcal{N}(\boldsymbol{w}_i | \boldsymbol{x}_i, \sigma^2 \boldsymbol{I})$ . Let  $y \in \mathcal{Y}$  denote a label and  $p_{\boldsymbol{X},y}$  the probability to classify  $\boldsymbol{X}$  as  $y$  under  $\Psi$ . Define  $\Delta \triangleq 1 - p^r$ . Then we have:
$$
\max _ {\tilde {\boldsymbol {X}} \in \mathcal {B} _ {2, \epsilon} ^ {r} (\boldsymbol {X})} \overline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \Phi \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y}}{1 - \Delta}\right) + \frac {\epsilon}{\sigma}\right) (1 - \Delta) + \Delta
$$
Proof. Cohen et al. (2019) show that the optimal value of Equation 4 under Gaussian smoothing is  $\Phi \left(\Phi^{-1}(p_{x,y}) + \frac{\epsilon}{\sigma}\right)$ , where  $\Phi$  denotes the CDF of the normal distribution. The statement follows analogously to Corollary 3 by directly plugging this optimal value into Theorem 2.
Corollary 5 (Multi-class certificates). Given continuous  $\mathcal{X} \triangleq \mathbb{R}$ , consider the threat model  $\mathcal{B}_{2,\epsilon}^{r}(\pmb{X})$  for fixed  $\pmb{X} \in \mathcal{X}^{N \times D}$ . Initialize the hierarchical smoothing distribution  $\Psi$  using the isotropic Gaussian distribution  $\mu_{\pmb{X}}(\pmb{W}) \triangleq \prod_{i=1}^{N} \mathcal{N}(\pmb{w}_i | \pmb{x}_i, \sigma^2 \pmb{I})$  that applies noise independently on each matrix element. Given majority class  $y^* \in \mathcal{Y}$  and the probability  $p_{\pmb{X}, y^*}$  to classify  $\pmb{X}$  as  $y^*$  under  $\Psi$ . Then the smoothed classifier  $g$  is certifiably robust  $g(\pmb{X}) = g(\tilde{\pmb{X}})$  for any  $\tilde{\pmb{X}} \in \mathcal{B}_{2,\epsilon}^{r}(\pmb{X})$  if
$$
\Phi \left(\Phi^ {- 1} \left(\frac {p _ {\mathbf {X} , y ^ {*}} - \Delta}{1 - \Delta}\right) - \frac {\epsilon}{\sigma}\right) > \max  _ {y \neq y ^ {*}} \Phi \left(\Phi^ {- 1} \left(\frac {p _ {\mathbf {X} , y}}{1 - \Delta}\right) + \frac {\epsilon}{\sigma}\right) + \frac {\Delta}{1 - \Delta}
$$
Proof. For the tighter multi-class certificate we have to guarantee  $\underline{p_{\tilde{\mathbf{X}},y^{*}}} > \max_{y \neq y^{*}} \overline{p_{\tilde{\mathbf{X}},y}}$ . The statement follows directly from Corollary 3 and Corollary 4.
F Hierarchical smoothing for discrete lower-level smoothing distributions
Threat model details for discrete hierarchical smoothing experiments. In our discrete experiments in Section 7 we model adversaries that perturb multiple nodes in the graph by deleting  $r_d$  ones (flip  $1 \to 0$ ) and adding  $r_a$  ones (flip  $0 \to 1$ ) in the feature matrix  $X$ , that is we consider the ball  $\mathcal{B}_{r_a, r_d}^r(X)$ . Here we define this threat model more formally:
$$
\begin{array}{l} \mathcal {B} _ {r _ {a}, r _ {d}} ^ {r} (\boldsymbol {X}) \triangleq \left\{\tilde {\mathbf {X}} \in \mathcal {X} ^ {N \times D} \mid \sum_ {i = 1} ^ {N} \mathbb {1} \left[ \mathbf {x} _ {i} \neq \tilde {\mathbf {x}} _ {i} \right] \leq r \right. \\ \sum_ {i} \sum_ {j} \mathbb {1} \left(\tilde {\boldsymbol {X}} _ {i j} = \boldsymbol {X} _ {i j} + 1\right) \leq r _ {a} \tag {5} \\ \sum_ {i} \sum_ {j} \mathbb {1} (\tilde {\boldsymbol {X}} _ {i j} = \boldsymbol {X} _ {i j} - 1) \leq r _ {d} \} \\ \end{array}
$$
where  $\mathbb{1}$  is an indicator function,  $r$  the number of controlled nodes,  $r_a$  the number of adversarial insertions  $(0\to 1)$ , and  $r_d$  the number of adversarial deletions  $(1\to 0)$ .
We expand the result of Theorem 1 to certificates for discrete lower-level smoothing distributions  $\mu$ :
Theorem 3 (Hierarchical lower bound for discrete lower-level smoothing distributions). Given discrete  $\mathcal{X}$  and a smoothing distribution  $\mu_{\pmb{x}}$  for discrete data, assume that the corresponding discrete certificate partitions the input space  $\mathcal{X}^{rD}$  into disjoint regions  $\mathcal{H}_1,\ldots ,\mathcal{H}_I$  of constant likelihood ratio:  $\mu_{\pmb{x}}(\pmb {w}) / \mu_{\tilde{\pmb{x}}}(\pmb {w}) = c_{i}$  for all  $\pmb {w}\in \mathcal{H}_i$ . Then we can compute a lower bound  $p_{\tilde{\mathbf{X}},y}\leq p_{\tilde{\mathbf{X}},y}$  by solving the following Linear Program (LP):
$$
\underline {{p _ {\tilde {X} , y}}} = \min  _ {\boldsymbol {h}} \quad \boldsymbol {h} ^ {T} \tilde {\boldsymbol {\nu}} \cdot (1 - \Delta) \quad s. t. \quad \boldsymbol {h} ^ {T} \boldsymbol {\nu} = \frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}, \quad 0 \leq \boldsymbol {h} _ {i} \leq 1
$$
where  $\tilde{\pmb{\nu}}_q\triangleq \mathrm{Pr}_{\pmb {W}\sim \mu_{\tilde{\pmb{X}}_{\mathcal{C}}}}[vec(\pmb {W})\in \mathcal{H}_q]$  and  $\pmb {\nu}_q\triangleq \operatorname *{Pr}_{\pmb {W}\sim \mu_{\pmb {X}_{\mathcal{C}}}}[vec(\pmb {W})\in \mathcal{H}_q]$  for all  $q\in \{1,\ldots ,I\}$ .
Proof. This follows from applying the Neyman-Pearson-Lemma for discrete random variables twice (see Lemma 2). While  $\mathcal{R}_1, \mathcal{R}_2, \mathcal{R}_3$  represent "super regions" for the upper-level smoothing distribution  $\phi, \mathcal{H}_1, \ldots, \mathcal{H}_I$  further subdivide  $\mathcal{R}_2$  into smaller regions of constant likelihood ratio, which is required for the lower-level discrete smoothing distribution  $\mu$ .
Notably, Theorem 3 implies that we can compute the LP of the original discrete certificate for the adjusted probability of  $(p_{X,y} - \Delta) / (1 - \Delta)$ , and then multiply the resulting optimal value by the constant  $1 - \Delta$ . In particular, the feasibility of the Linear Program and of computing  $\boldsymbol{v}$ ,  $\tilde{\boldsymbol{v}}$  can be delegated to the certificate for the underlying  $\mu$ .
Analogously for the upper bound for discrete lower-level  $\mu$  we need:
Lemma 4 (Discrete Neyman-Pearson upper bound). Partition the input space  $\mathcal{X}^D$  into disjoint regions  $\mathcal{R}_1, \ldots, \mathcal{R}_I$  of constant likelihood ratio  $\mu_{\boldsymbol{x}}(\boldsymbol{w}) / \mu_{\tilde{\boldsymbol{x}}}(\boldsymbol{w}) = c_i \in \mathbb{R} \cup \{\infty\}$  for all  $\boldsymbol{w} \in \mathcal{R}_i$ . Define  $\tilde{\boldsymbol{r}}_i \triangleq \operatorname*{Pr}_{\boldsymbol{w} \sim \mu_{\tilde{\boldsymbol{x}}}}(\boldsymbol{w} \in \mathcal{R}_i)$  and  $\boldsymbol{r}_i \triangleq \operatorname*{Pr}_{\boldsymbol{w} \sim \mu_{\boldsymbol{x}}}(\boldsymbol{w} \in \mathcal{R}_i)$ . Then Equation 4 is equivalent to the linear program  $\overline{p_{\tilde{\boldsymbol{x}},y}} = \max_{\boldsymbol{h} \in [0,1]^I} \boldsymbol{h}^T \tilde{\boldsymbol{r}}$  s.t.  $\boldsymbol{h}^T \boldsymbol{r} = p_{\boldsymbol{x},y}$ , where  $\boldsymbol{h}$  represents the worst-case classifier.
Then we can show:
Theorem 4 (Hierarchical upper bound for discrete lower-level smoothing distributions). Given discrete  $\mathcal{X}$  and a smoothing distribution  $\mu_{\mathbf{x}}$  for discrete data, assume that the corresponding discrete certificate partitions the input space  $\mathcal{X}^{rD}$  into disjoint regions  $\mathcal{H}_1,\ldots ,\mathcal{H}_I$  of constant likelihood ratio:  $\mu_{\mathbf{x}}(\mathbf{w}) / \mu_{\tilde{\mathbf{x}}}(\mathbf{w}) = c_i$  for all  $\mathbf{w}\in \mathcal{H}_i$ . Then we can compute an upper bound  $\overline{p_{\tilde{\mathbf{X}},y}}\geq p_{\tilde{\mathbf{X}},y}$  by solving the following Linear Program (LP):
$$
\overline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \max  _ {\boldsymbol {h}} \quad \boldsymbol {h} ^ {T} \tilde {\boldsymbol {\nu}} \cdot (1 - \Delta) + \Delta \quad s. t. \quad \boldsymbol {h} ^ {T} \boldsymbol {\nu} = \frac {p _ {\boldsymbol {X} , y}}{1 - \Delta}, \quad 0 \leq \boldsymbol {h} _ {i} \leq 1
$$
where  $\tilde{\pmb{\nu}}_q\triangleq \mathrm{Pr}_{\pmb {W}\sim \mu_{\bar{\pmb{X}}_{\mathcal{C}}}}[vec(\pmb {W})\in \mathcal{H}_q]$  and  $\pmb {\nu}_q\triangleq \mathrm{Pr}_{\pmb {W}\sim \mu_{\pmb {X}_{\mathcal{C}}}}[vec(\pmb {W})\in \mathcal{H}_q]$  for all  $q\in \{1,\ldots ,I\}$ .
Proof. This follows analogously to Theorem 3, specifically by applying the Neyman-Pearson-Lemma upper bound for discrete random variables (Lemma 4) twice (first using the "super regions" and then the "lower-level" regions).  $\square$
G Hierarchical randomized smoothing using ablation smoothing
There are different types randomized ablation certificates: In image classification, Levine and Feizi (2020b) randomly ablate pixels by masking them. In point cloud classification, Liu et al. (2021) ablate points by deleting them to certify robustness against adversarial point insertion, deletion and modification. In sequence classification, Huang et al. (2023) randomly delete sequence elements to certify robustness against edit distance threat models. In node/graph classification, Bojchevski et al. (2020) experiment with ablation smoothing for individual attributes but show that their additive method is superior. Scholten et al. (2022) directly mask out entire nodes, yielding strong certificates against adversaries that arbitrarily perturb node attributes.
Ablation smoothing differences. The ablation baseline (Levine and Feizi, 2020b) draws  $k$  pixels to ablate from a uniform distribution  $\mathcal{U}(k,N)$  over all possible subsets with  $k$  out of  $N$  pixels. Levine and Feizi (2020b) derive  $\Delta^L = 1 - \frac{\binom{N - r}{k}}{\binom{N}{k}}$ . This is in contrast to our approach, which selects pixels to add noise to with a probability of  $p$ , for which we have  $\Delta^S = 1 - p^r$ . In other words, we choose the pixel selection probability of  $p = 1 - \frac{k}{N}$  (i.e. we select  $k$  pixels in expectation). This way, increasing  $k$  corresponds to decreasing  $p$  and vice versa. Interestingly, this type of ablation smoothing is tighter for small  $N$  since  $\Delta^S < \Delta^L$  for  $k > 0$  (see Scholten et al. (2022) (Appendix B, Proposition 5) for a detailed discussion; later also verified by Huang et al. (2023)).
Ablation and additive noise hybrid. Notably, our method generalizes ablation and additive noise smoothing into a common framework: The additional indicator indicates which entity will be ablated, but instead of abating entities we only add noise to them. If we ablate the selected rows instead of adding noise, we can restore certificates for ablation smoothing as a special case of our framework:
Proof. We can first apply Theorem 3 and obtain:
$$
\underline {{p _ {\boldsymbol {X} , y}}} = \min  _ {\boldsymbol {h}} \quad \boldsymbol {h} ^ {T} \tilde {\boldsymbol {\nu}} \cdot (1 - \Delta) \quad \text {s . t .} \quad \boldsymbol {h} ^ {T} \boldsymbol {\nu} = \frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}, \quad 0 \leq \boldsymbol {h} _ {i} \leq 1
$$
for  $\tilde{\pmb{v}} = 1$  and  $\pmb {v} = 1$  (since we always ablate selected rows there is only a single lower-level region  $\mathcal{H}$ ). Solving this LP yields
$$
\underline {{p _ {\tilde {X} , y}}} = \frac {p _ {X , y} - \Delta}{1 - \Delta} (1 - \Delta) = p _ {X, y} - \Delta
$$
Note one can analogously compute the upper bound  $\overline{p_{\bar{X},y}}$  by applying the corresponding Theorem 4:
$$
\overline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \frac {p _ {\boldsymbol {X} , y}}{1 - \Delta} (1 - \Delta) + \Delta = p _ {\boldsymbol {X}, y} + \Delta
$$
Altogether in this special case we obtain for the multi-class certificate  $\underline{p_{\tilde{X},y^*}} > \max_{y \neq y^*} \overline{p_{\tilde{X},y}}$ :
$$
p _ {\boldsymbol {X}, y ^ {*}} - \Delta > p _ {\boldsymbol {X}, \bar {y}} + \Delta
$$
where  $y^{*}$  is the most likely class and  $\tilde{y}$  the follow-up class. That is in this special case with ablation smoothing we again obtain the multi-class guarantees derived by Levine and Feizi (2020b).
Related work. In the context of randomized ablation (a special case of our framework), Jia et al. (2021) also define three regions and build an ablation certificate using the Neyman-Pearson Lemma. Their regions are, however, different to ours, and their certificates are designed for poisoning threat models (whereas we consider evasion settings). Moreover, Liu et al. (2021) define three regions similar to ours, but their certificate is only for point cloud classifiers and suffers from the same limitations of ablation certificates that we overcome: Our key idea is to further process the region  $\mathcal{R}_2$  and to utilize the budget  $p_{X,y} - \Delta$  by plugging it into the certificate for the lower-level smoothing distribution, which yields stronger robustness-accuracy trade-offs under our threat model.
Discussion. In general, the difference between ablation certificates and additive noise certificates is that ablation smoothing does not evaluate the base classifier on the entire input space. Specifically, the supported sample spaces around  $\mathbf{X}$  and  $\tilde{\mathbf{X}}$  are overlapping but not the identical. In this context, additive noise certificates are technically tighter for threat models where adversarial perturbations are bounded. Since our certificates generalize ablation smoothing and additive noise smoothing we inherit advantages and disadvantages of both methods: While the certificate for the lower-level smoothing distribution may be tight, the upper-level smoothing distribution prevents us from evaluating the base classifier on the entire matrix space. We leave the development of tight and efficient certificates for hierarchical randomized smoothing to future work.
Gray-box certificates. Scholten et al. (2022) derive gray-box certificates by exploiting the underlying message-passing principles of graph neural networks. In detail, they derive a tighter gray-box  $\Delta$  that accounts for the fact that nodes may be disconnected from a target node when randomly deleting edges in a graph (and disconnected nodes do not affect the prediction for the target). Interestingly, we can directly plug their gray-box  $\Delta$  into our framework to obtain gray-box robustness certificates for hierarchical randomized smoothing.
H Extended discussion
Motivation for the indicator variable as additional column. We certify robustness of classifiers operating on the higher-dimensional matrix space, which allows our certificates to be efficient and highly flexible. In the following we motivate this by considering the scenario of certifying robustness on the original data space  $\mathcal{X}^{N\times D}$ . The main problem is that it is technically challenging to obtain an efficient certificate that depends only on the radius (and not on fixed  $X$  and  $\tilde{X}$ ):
Consider a discrete lower-level smoothing distribution, fixed  $\mathbf{X}$ ,  $\tilde{\mathbf{X}}$ , smoothed matrix  $\mathbf{W}$  and row  $i \in \mathcal{C}$  such that  $\boldsymbol{x}_i \neq \tilde{\boldsymbol{x}}_i$  but  $\boldsymbol{x}_i = \boldsymbol{w}_i$ . To certify robustness on the original space we have to consider the likelihood ratio for the marginal distribution  $\Psi_{\mathbf{X}}(\mathbf{W}) \triangleq \sum_{\tau} \Psi_{\mathbf{X}}(\mathbf{W}, \tau)$ :
$$
\frac {\Psi_ {X} (W)}{\Psi_ {\tilde {X}} (W)} = \frac {\sum_ {\tau} \Psi_ {X} (W , \tau)}{\sum_ {\tau} \Psi_ {\tilde {X}} (W , \tau)}.
$$
Let  $\mathcal{C}$  describe all perturbed and  $\tilde{\mathcal{C}}$  all clean rows. Similar due to independence we have:
$$
\frac {\Psi_ {\mathbf {X}} (\mathbf {W})}{\Psi_ {\tilde {\mathbf {X}}} (\mathbf {W})} \stackrel {(1)} {=} \frac {\prod_ {i = 1} ^ {n} \Psi_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i})}{\prod_ {i = 1} ^ {n} \Psi_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i})} = \frac {\prod_ {i \in \mathcal {C}} \Psi_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i}) \prod_ {i \in \tilde {\mathcal {C}}} \Psi_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i})}{\prod_ {i \in \mathcal {C}} \Psi_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i}) \prod_ {i \in \tilde {\mathcal {C}}} \Psi_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i})} \stackrel {(2)} {=} \frac {\prod_ {i \in \mathcal {C}} \Psi_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i})}{\prod_ {i \in \mathcal {C}} \Psi_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i})}
$$
where (1) is due independence and (2) due to  $\pmb{x}_i = \tilde{\pmb{x}}_i$  for clean rows  $i \in \tilde{C}$ . Now we can consider the perturbed row  $i \in C$  that we chose above and consider the likelihood ratio:
$$
\frac {\Psi_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i})}{\Psi_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i})} = \frac {\sum_ {\boldsymbol {\tau} _ {i}} \mu_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i} | \boldsymbol {\tau} _ {i}) \phi (\boldsymbol {\tau} _ {i})}{\sum_ {\boldsymbol {\tau} _ {i}} \mu_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i} | \boldsymbol {\tau} _ {i}) \phi (\boldsymbol {\tau} _ {i})} \stackrel {(3)} {=} \frac {\mu_ {\mathbf {x} _ {i}} (\mathbf {w} _ {i}) p + (1 - p)}{p \mu_ {\tilde {\mathbf {x}} _ {i}} (\mathbf {w} _ {i})}
$$
where (3) is due to  $\pmb{w}_i = \pmb{x}_i$  but  $\pmb{w}_i \neq \tilde{\pmb{x}}_i$ .
Due to the mixture, there are two ways sampling  $\pmb{w}_i$  from  $\Psi_{\pmb{x}_i}$ : First, we may not "select" row  $i$  (and then we do not smooth the row). Second, we may "select" row  $i$  but not add any noise on the selected row. Due to the mixture, we cannot simply cancel out dimensions in which the rows  $\pmb{x}_i$  and  $\tilde{\pmb{x}}_i$  agree as in (Bojchevski et al., 2020), leading to the problem that the likelihood ratio and subsequently the point-wise certificate depends on  $\tilde{\pmb{X}}$ . Consequently, certifying robustness against an entire ball of perturbations is technically challenging since the likelihood ratio will change for every  $\tilde{\pmb{X}}$ .
We can resolve this problem by realizing that for reasonable smoothing parameters, the probability  $\mu_{\tilde{\mathbf{x}}_i}(\mathbf{w}_i) = \mu_{\tilde{\mathbf{x}}_i}(\mathbf{x}_i)$  is close to zero (probability for flipping exactly and only the dimensions where  $\tilde{\pmb{x}}_i$  disagrees with  $\pmb{x}_i$  is close to zero for small radii and high dimensional data). Specifically, the likelihood ratio for such elements is approaching  $\infty$  (e.g. for sparse smoothing with fixed radius  $r_a$ ,  $r_d$  we have  $\mu_{\tilde{\mathbf{x}}_i}(\mathbf{w}_i)\to 0$  for  $D\to \infty$ ). Note that if we choose a  $\pmb{W}$  and row  $i$  such that  $\pmb {x}_i\neq \tilde{\pmb{x}}_i$  but  $\tilde{\pmb{x}}_i = \pmb {w}_i$ , we can derive the same but the other way around and find the likelihood ratio approaching 0. In other words, we cannot avoid regions of 0, and  $\infty$  likelihood ratio for such smoothing distributions where we can flip "entire rows" from  $\pmb {x}_i$  to  $\tilde{\pmb{x}}_i$ . Certifying robustness on the higher-dimensional matrix space instead is efficient, highly flexible and tight for higher dimensional rows as for example in graphs where rows represent node features.
Efficiency of hierarchical randomized smoothing. We discuss (1) the efficiency of the smoothing process, and (2) the efficiency of the certification. First, regarding the efficiency of the smoothing process, note that hierarchical smoothing only adds noise on a subset of all entities. Technically this can be more efficient, although drawing from a bernoulli before adding noise may require more time in practice than directly drawing (e.g. gaussian) noise for all matrix elements simultaneously. Second, regarding certification efficiency, our method only requires the additional computation of the term  $\Delta$  which takes constant time. Yet, closed-form solutions for the largest certifiable radius may not be available, depending on the lower-level smoothing distribution. For example we may not be able to derive a closed-form radius for the multi-class certificate under hierarchical Gaussian smoothing (see Appendix E). Still, the certification process can be implemented efficiently even without closed-form solutions, e.g. via bisection or just by incrementally increasing  $\epsilon$  to find the largest certifiable radius.