SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Certifying the smoothed classifier on this higher-dimensional space simplifies the certification: By appending  $\tau$  to the data the supports of both distributions  $\Psi_{\mathbf{X}}$  and  $\Psi_{\tilde{\mathbf{X}}}$  differ, and they intersect only for those  $Z$  for which all perturbed rows are selected by  $\tau$  (see Figure 3). This allows the upper-level certificate to separate clean from perturbed rows and to delegate robustness guarantees for perturbed rows to an existing certificate for the lower-level  $\mu$ . We further elaborate on why this concatenation is necessary in Appendix H. In the following we show robustness certification under this hierarchical smoothing mainly involves (1) transforming the observed label probability  $p_{\mathbf{X},y}$ , (2) computing an existing certificate for the transformed  $p_{\mathbf{X},y}$ , and (3) transforming the resulting lower bound  $p_{\tilde{\mathbf{X}},y}$ . We provide pseudo-code for the entire certification in Appendix C.
Figure 3: Overview of the disjoint regions and probabilities to sample  $Z$  of each region. The absence of arrows into specific regions indicates a probability of zero. Only  $Z$  in region  $\mathcal{R}_2$  can be sampled from both distributions  $\Psi_{\pmb{X}}$  and  $\Psi_{\tilde{\pmb{X}}}$ . The likelihood ratios visualize the proof of Theorem 1.
5.1 Point-wise robustness certificates for hierarchical randomized smoothing
Before providing certificates against the entire threat model we first derive point-wise certificates, i.e. we consider an arbitrary but fixed  $\tilde{\pmb{X}}\in \mathcal{B}_{p,\epsilon}^{r}(\mathbf{X})$  and show robustness  $g(\pmb {X}) = g(\tilde{\pmb{X}})$  by deriving a lower bound  $p_{\tilde{\pmb{X}},y}\leq p_{\tilde{\pmb{X}},y}$ . The smoothed classifier  $g$  is certifiably robust w.r.t.  $\tilde{\pmb{X}}$  if  $p_{\tilde{\pmb{X}},y} > 0.5$ . We derive a lower bound by using the discrete NP-Lemma on the upper-level  $\phi$ , which allows us to delegate robustness guarantees to the lower-level  $\mu$  on the perturbed rows only.
Let  $\mathcal{C} \triangleq \{i : x_i \neq \tilde{x}_i\}$  denote the rows in which  $X$  and  $\tilde{X}$  differ. To apply the discrete NP-Lemma we have to partition the space  $\mathcal{Z}^{N \times (D + 1)}$  into disjoint regions (Lemma 2). The upper-level distribution allows us to define the following four regions (Figure 3): In region  $\mathcal{R}_2$  all perturbed rows are selected ( $\forall i \in \mathcal{C} : \tau_i = 1$ ). In regions  $\mathcal{R}_1$  and  $\mathcal{R}_3$  at least one perturbed row is not selected when sampling from  $\Psi_X$  and  $\Psi_{\tilde{X}}$ , respectively. Region  $\mathcal{R}_0$  contains  $Z$  that cannot be sampled from either distribution.
Example for matrices. Consider the following example for matrix data to see the existence of four different regions. Define  $\mathbf{X} \triangleq (\mathbf{x}_1 | \mathbf{x}_2 | \mathbf{x}_3)^T$  and  $\tilde{\mathbf{X}} \triangleq (\mathbf{x}_1 | \tilde{\mathbf{x}}_2 | \mathbf{x}_3)^T$  (second row in  $\tilde{\mathbf{X}}$  is perturbed). Now consider a smoothed row  $\mathbf{w}$  with  $\mathbf{w} \neq \mathbf{x}_2$  and  $\mathbf{w} \neq \tilde{\mathbf{x}}_2$  that we can sample from both lower-level distributions  $\mu_{\mathbf{x}_2}(\mathbf{w}) > 0$  and  $\mu_{\tilde{\mathbf{x}}_2}(\mathbf{w}) > 0$ . Then we can define four examples, one for each region:
$$
\left( \begin{array}{c c} \mathbf {Z} _ {0} \in \mathcal {R} _ {0} \\ - \mathbf {x} _ {1} - & 0 \\ - \mathbf {w} - & 0 \\ - \mathbf {x} _ {3} - & 0 \end{array} \right)
$$
$$
\Psi_ {X} \left(\mathbf {Z} _ {0}\right) = 0, \Psi_ {\hat {X}} \left(\mathbf {Z} _ {0}\right) = 0
$$
$$
\begin{array}{c} \boldsymbol {Z} _ {1} = \boldsymbol {X}   | 0 \in \mathcal {R} _ {1} \\ \left( \begin{array}{c c} - \boldsymbol {x} _ {1} - & 0 \\ - \boldsymbol {x} _ {2} - & 0 \\ - \boldsymbol {x} _ {3} - & 0 \end{array} \right) \end{array}
$$
$$
\Psi_ {\boldsymbol {X}} (\boldsymbol {Z} _ {1}) > 0, \Psi_ {\tilde {\boldsymbol {X}}} (\boldsymbol {Z} _ {1}) = 0
$$
$$
\begin{array}{c} \mathbf {Z} _ {2} \in \mathcal {R} _ {2} \\ \left( \begin{array}{c c} - \mathbf {x} _ {1} - & 0 \\ - \mathbf {w} - & \mathbf {1} \\ - \mathbf {x} _ {3} - & 0 \end{array} \right) \end{array}
$$
$$
\Psi_ {X} \left(\mathbf {Z} _ {2}\right) > 0, \Psi_ {\tilde {X}} \left(\mathbf {Z} _ {2}\right) > 0
$$
$$
\begin{array}{c} \boldsymbol {Z} _ {3} = \tilde {\boldsymbol {X}} | 0 \in \mathcal {R} _ {3} \\ \left( \begin{array}{c c} - \boldsymbol {x} _ {1} - & 0 \\ - \tilde {\boldsymbol {x}} _ {2} - & 0 \\ - \boldsymbol {x} _ {3} - & 0 \end{array} \right) \end{array}
$$
$$
\Psi_ {\boldsymbol {X}} \left(\boldsymbol {Z} _ {3}\right) = 0, \Psi_ {\tilde {\boldsymbol {X}}} \left(\boldsymbol {Z} _ {3}\right) > 0
$$
Note that we can sample the row  $\boldsymbol{w}$  by adding noise to the rows  $x_{2}$  or  $\tilde{x}_{2}$  only if the upper-level smoothing distribution  $\phi$  first selects the perturbed row  $(\tau_{2} = 1)$ . In general, we can only sample the same matrix  $Z$  from both distributions  $\Psi_{\mathbf{X}}$  and  $\Psi_{\tilde{\mathbf{X}}}$  if the upper-level smoothing distribution first selects all perturbed rows  $i \in \mathcal{C}$ . We group all those  $Z$  in the region  $\mathcal{R}_2$ .
Partitioning into regions. By appending the indicator  $\pmb{\tau}$  to the data we obtain four disjoint regions. To formally characterize these regions, we define the reachable set as  $\mathcal{A} \triangleq \mathcal{A}_1 \cup \mathcal{A}_2 \subseteq \mathcal{Z}^{N \times (D + 1)}$  with  $\mathcal{A}_1 \triangleq \{Z \mid \forall i : (\tau_i = 0) \Rightarrow (z_i = x_i|0)\}$ , and  $\mathcal{A}_2 \triangleq \{Z \mid \forall i : (\tau_i = 0) \Rightarrow (z_i = \tilde{x}_i|0)\}$ , where  $x_i|0$  denotes concatenation. Intuitively,  $\mathcal{A}$  only contains matrices that can be sampled from  $\Psi$ . Now we can define the regions and derive the required probabilities to sample  $Z$  of each region:
Proposition 1. The regions  $\mathcal{R}_0, \mathcal{R}_1, \mathcal{R}_2$  and  $\mathcal{R}_3$  are disjoint and partition the space  $\mathcal{Z}^{N \times (D + 1)}$ :
-  $\mathcal{R}_0\triangleq \mathcal{Z}^{N\times (D + 1)}\setminus \mathcal{A}$
-  $\mathcal{R}_1 \triangleq \{\pmb{Z} \in \mathcal{A} \mid \exists i \in \mathcal{C}: \pmb{\tau}_i = 0, \pmb{z}_i = \pmb{x}_i | 0\}$
-  $\mathcal{R}_2 \triangleq \{Z \in \mathcal{A} \mid \forall i \in \mathcal{C}: \tau_i = 1\}$
-  $\mathcal{R}_3 \triangleq \{\pmb{Z} \in \mathcal{A} \mid \exists i \in \mathcal{C}: \pmb{\tau}_i = 0, \pmb{z}_i = \tilde{\pmb{x}}_i | 0\}$
Proposition 2. Given  $\Psi_{\mathbf{X}}$  and  $\Psi_{\tilde{\mathbf{X}}}$ , the regions of Proposition 1 and  $\Delta \triangleq 1 - p^{|C|}$  we have
$$
\operatorname *{Pr}_{\substack{\boldsymbol {Z}\sim \Psi_{\boldsymbol{X}}}}[ \boldsymbol {Z}\in \mathcal{R}_{1}] = \Delta
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {2} ] = 1 - \Delta
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \boldsymbol {\Psi} _ {\boldsymbol {X}}} [ \boldsymbol {Z} \in \mathcal {R} _ {3} ] = 0
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\bar {X}}} \left[ \boldsymbol {Z} \in \mathcal {R} _ {1} \right] = 0
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\tilde {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {2} ] = 1 - \Delta
$$
$$
\operatorname * {P r} _ {\boldsymbol {Z} \sim \Psi_ {\hat {\boldsymbol {X}}}} [ \boldsymbol {Z} \in \mathcal {R} _ {3} ] = \Delta
$$
Proofs in Appendix C. Intuitively,  $1 - \Delta = p^{|C|}$  is the probability that all perturbed rows  $i\in \mathcal{C}$  are selected  $(\tau_{i} = 1)$ , and  $\Delta$  that at least one row  $i\in \mathcal{C}$  is not selected by  $\phi$ . Lastly, the probability to sample  $\pmb{Z}\in \mathcal{R}_0$  is just 0. We visualize Proposition 1 and Proposition 2 in Figure 3.
Hierarchical smoothing certificates. Having derived a partitioning of  $\mathcal{Z}^{N\times (D + 1)}$  as above, we can apply the NP-Lemma for discrete random variables (Lemma 2) on the upper-level distribution  $\phi$ . Notably, we show that the problem of certification under hierarchical smoothing can be reduced to proving robustness for the lower-level distribution  $\mu$  on the adversarially perturbed rows  $\mathcal{C}$  only:
Theorem 1 (Neyman-Pearson lower bound for hierarchical smoothing). Given fixed  $\mathbf{X}$ ,  $\tilde{\mathbf{X}} \in \mathcal{X}^{N\times D}$ , let  $\mu_{\mathbf{X}}$  denote a smoothing distribution that operates independently on matrix elements, and  $\Psi_{\mathbf{X}}$  the induced hierarchical distribution over  $\mathcal{Z}^{N\times (D + 1)}$ . Given label  $y \in \mathcal{Y}$  and the probability  $p_{\mathbf{X},y}$  to classify  $\mathbf{X}$  as  $y$  under  $\Psi$ . Define  $\hat{S}_{\kappa} \triangleq \left\{\mathbf{W} \in \mathcal{X}^{|C| \times D}: \mu_{\tilde{\mathbf{X}}_C}(\mathbf{W}) \leq \kappa \cdot \mu_{\mathbf{X}_C}(\mathbf{W})\right\}$ . We have:
$$
\underline {{p _ {\tilde {\boldsymbol {X}} , y}}} = \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {\tilde {X}} _ {C}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] \cdot (1 - \Delta) w i t h \kappa \in \mathbb {R} _ {+} s. t. \quad \operatorname * {P r} _ {\boldsymbol {W} \sim \mu_ {\boldsymbol {X} _ {C}}} [ \boldsymbol {W} \in \hat {S} _ {\kappa} ] = \frac {p _ {\boldsymbol {X} , y} - \Delta}{1 - \Delta}
$$
where  $X_{\mathcal{C}}$  and  $\tilde{X}_{\mathcal{C}}$  denote those rows  $\pmb{x}_i$  of  $\pmb{X}$  and  $\tilde{\pmb{x}}_i$  of  $\tilde{\pmb{X}}$  with  $i\in \mathcal{C}$ , that is  $\pmb{x}_i\neq \tilde{\pmb{x}}_i$ .
Proof sketch (Full proof in Appendix C). In the worst-case all matrices in  $\mathcal{R}_1$  are correctly classified. Note that this is the worst case since we cannot obtain any matrix of region  $\mathcal{R}_1$  by sampling from the distribution  $\Psi_{\tilde{\pmb{X}}}$ . Therefore the worst-case classifier first uses the budget  $p_{\pmb{X},y}$  on region  $\mathcal{R}_1$  and we can subtract the probability  $\mathrm{Pr}_{\pmb{Z} \sim \Psi_{\pmb{X}}}[\pmb{Z} \in \mathcal{R}_1] = \Delta$  from the label probability  $p_{\pmb{X},y}$ .
Since we never sample matrices of region  $\mathcal{R}_3$  from the distribution  $\Psi_{\pmb{X}}$ , the remaining correctly classified matrices must be in region  $\mathcal{R}_2$ , which one reaches with probability  $1 - \Delta$  from both distributions  $\Psi_{\pmb{X}}$  and  $\Psi_{\tilde{\pmb{X}}}$ . In region  $\mathcal{R}_2$ , however, we can simplify the problem to computing the certificate for  $\mu$  on the perturbed rows to distribute the remaining probability. Since one never samples matrices of region  $\mathcal{R}_0$  we do not have to consider it. The remaining statement follows from the Neyman-Pearson-Lemma (Neyman and Pearson, 1933; Tocher, 1950).
Note that we also derive the counterpart for discrete lower-level  $\mu$  in Appendix F.
Implications. Notably, Theorem 1 implies that we can delegate robustness guarantees to the lower-level smoothing distribution  $\mu$ , compute the optimal value of Equation 1 under  $\mu$  given the transformed probability  $(p_{X,y} - \Delta) / (1 - \Delta)$  and multiply the result with  $(1 - \Delta)$ . This way we obtain the optimal value of Equation 1 under the hierarchical smoothing distribution  $\Psi$  on the extended matrix space and thus robustness guarantees for hierarchical smoothing. This means our certificates are highly flexible and can integrate the whole suite of existing smoothing distributions with independent noise per dimension (Lecuyer et al., 2019; Cohen et al., 2019; Lee et al., 2019; Yang et al., 2020a). This is in contrast to all existing approaches where one has to come up with novel smoothing distributions and derive certificates from scratch once new threat models are introduced.
Special cases. We discuss two special cases of the probability  $p$  for smoothing rows: First, if the probability to select rows for smoothing is  $p = 1$  (intuitively, we always smooth the entire matrix), then we have  $\Delta = 1 - p^{|C|} = 0$  for any number of perturbed rows  $|\mathcal{C}|$  and with Theorem 1 we get  $p_{\tilde{\mathbf{X}},\underline{y}}(\Psi) = p_{\tilde{\mathbf{X}},\underline{y}}(\mu)$ . That is in this special case we obtain the original certificate for the lower-level smoothing distribution  $\mu$ . Note that in this case the certificate ignores  $r$ . Second, if the probability to select rows is  $p = 0$  (intuitively, we never sample smoothed matrices), we have  $\Delta = 1 - p^{|C|} = 1$  for any  $|\mathcal{C}| \geq 1$ , and with Theorem 1 we get  $p_{\tilde{\mathbf{X}},\underline{y}} = 0$ , that is we do not obtain any certificates.
5.2 Regional robustness certificates for hierarchical randomized smoothing
So far we can only guarantee that the prediction is robust for a specific perturbed input  $\tilde{\pmb{X}}\in \mathcal{B}_{p,\epsilon}^{r}(\pmb {X})$  To ensure that the smoothed classifier is robust to the entire treat model  $\mathcal{B}_{p,\epsilon}^{r}(\pmb {X})$  we have to guarantee  $\min_{\tilde{\pmb{X}}\in \mathcal{B}_{p,\epsilon}^{r}(\mathbf{X})}\underline{p_{\tilde{\pmb{X}},y^{*}}} >0.5.$  In the following we show that it is sufficient to compute Theorem 1 for  $\Delta = 1 - p^r$  with the largest radius  $r$  . In fact, the final certificate is independent of which rows are perturbed, as long the two matrices differ in exactly  $r$  rows the certificate is the same.
Proposition 3. Given clean  $\mathbf{X} \in \mathcal{X}^{N \times D}$ . Consider the set  $\mathcal{Q}_{p,\epsilon}^r(\mathbf{X})$  of inputs at a fixed distance  $r$  (i.e.  $|\mathcal{C}| = r$  for all  $\tilde{\mathbf{X}} \in \mathcal{Q}_{p,\epsilon}^r(\mathbf{X})$ ). Then we have  $\min_{\tilde{\mathbf{X}} \in \mathcal{B}_{p,\epsilon}^r(\mathbf{X})} \underline{p_{\tilde{\mathbf{X}},y}} = \min_{\tilde{\mathbf{X}} \in \mathcal{Q}_{p,\epsilon}^r(\mathbf{X})} \underline{p_{\tilde{\mathbf{X}},y}}$ .
Proof. The probability  $1 - \Delta = p^{|C|}$  is monotonously decreasing in the number of perturbed rows  $|\mathcal{C}|$ . Thus the lower bound  $p_{\bar{\mathbf{X}},y}$  is also monotonously decreasing in  $|\mathcal{C}|$  (see Theorem 1). It follows for fixed  $\epsilon$  that  $p_{\bar{\mathbf{X}},y}$  is minimal at  $|\mathcal{C}| = r$ , i.e. the largest possible radius under our threat model  $\mathcal{B}_{p,\epsilon}^{r}(\mathbf{X})$ .
6 Initializing hierarchical randomized smoothing
In the previous section we derive robustness certificates for hierarchical randomized smoothing for any lower-level smoothing distribution  $\mu$  by deriving a general lower bound on the probability  $p_{\tilde{X},y}$  in Theorem 1. In this section we instantiate our hierarchical smoothing framework with specific lower-level smoothing distributions for discrete and continuous data.
6.1 Hierarchical randomized smoothing using Gaussian isotropic smoothing
Concerning continuous data  $\mathcal{X} = \mathbb{R}$ , consider the threat model  $\mathcal{B}_{p,\epsilon}^{r}(\mathbf{X})$  for  $p = 2$ . We initialize the lower-level smoothing distribution of hierarchical smoothing with the Gaussian distribution (Cohen et al., 2019) (as introduced in Section 3), since it is specifically designed and tight for the well-studied  $\ell_2$ -norm threat model. For our purposes we apply the Gaussian noise independently on the matrix entries. In the following we present the binary-class certificates for hierarchical randomized smoothing induced by isotropic Gaussian smoothing:
Corollary 1. Given continuous  $\mathcal{X} \triangleq \mathbb{R}$ , consider the threat model  $\mathcal{B}_{2,\epsilon}^{r}(\pmb{X})$  for fixed  $\pmb{X} \in \mathcal{X}^{N \times D}$ . Initialize the hierarchical smoothing distribution  $\Psi$  using the isotropic Gaussian distribution  $\mu_{\pmb{X}}(\pmb{W}) \triangleq \prod_{i=1}^{N} \mathcal{N}(\pmb{w}_i | \pmb{x}_i, \sigma^2 \pmb{I})$  that applies noise independently on each matrix element. Let  $y^* \in \mathcal{Y}$  denote the majority class and  $p_{\pmb{X}, y^*}$  the probability to classify  $\pmb{X}$  as  $y^*$  under  $\Psi$ . Then the smoothed classifier  $g$  is certifiably robust  $g(\pmb{X}) = g(\tilde{\pmb{X}})$  for any  $\tilde{\pmb{X}} \in \mathcal{B}_{2,\epsilon}^{r}(\pmb{X})$  if
$$
\epsilon <   \sigma \left(\Phi^ {- 1} \left(\frac {p _ {\boldsymbol {X} , y ^ {*} - \Delta}}{1 - \Delta}\right) - \Phi^ {- 1} \left(\frac {1}{2 (1 - \Delta)}\right)\right)
$$
where  $\Phi^{-1}$  denotes the inverse CDF of the normal distribution and  $\Delta \triangleq 1 - p^r$ .
Proof in Appendix E. We also derive the corresponding multi-class certificates in Appendix E.
6.2 Hierarchical randomized smoothing using sparse smoothing
To demonstrate our certificates in discrete domains we consider binary data  $\mathcal{X} = \{0,1\}$  and model adversaries that delete  $r_d$  ones (flip  $1\rightarrow 0$ ) and add  $r_a$  ones (flip  $0\rightarrow 1$ ), that is we consider the ball  $\mathcal{B}_{r_a,r_d}^r (\boldsymbol {X})$  (see Appendix B for a formal introduction). We initialize the lower-level smoothing distribution of hierarchical smoothing with the sparse smoothing distribution proposed by Bojchevski et al. (2020):  $p(\mu (\pmb {x})_i\neq \pmb {x}_i) = p_+^{1 - \pmb{x}_i}p_-^{\pmb{x}_i}$ , introducing two different noise probabilities to flip  $0\rightarrow 1$  with probability  $p_{+}$  and  $1\to 0$  with probability  $p_{-}$ . The main idea is that the different flipping probabilities allow to preserve sparsity of the underlying data, making this approach particularly useful for graph-structured data. Note that we can consider the discrete certificate of Lee et al. (2019) as a special case (Bojchevski et al., 2020). We derive the corresponding certificates in Appendix F.
6.3 Hierarchical randomized smoothing using ablation smoothing
Lastly, randomized ablation (Levine and Feizi, 2020b) is a smoothing distribution where the input is not randomly smoothed but masked, e.g. by replacing the input with a special ablation token  $t \notin \mathcal{X}^D$  that does not exist in the original space. There are different variations of ablation and we carefully discuss such differences in Appendix G. By choosing a smoothing distribution  $\mu$  that ablates all selected rows we can prove the following connection between hierarchical and ablation smoothing:
Corollary 2. Initialize the hierarchical smoothing distribution  $\Psi$  with the ablation smoothing distribution  $\mu_{\pmb{x}}(\pmb{t}) = 1$  for ablation token  $\pmb{t} \notin \mathcal{X}^D$  and  $\mu_{\pmb{x}}(\pmb{w}) = 0$  for  $\pmb{w} \in \mathcal{X}^D$  otherwise (i.e. we always ablate selected rows). Define  $\Delta \triangleq 1 - p^r$ . Then the smoothed classifier  $g$  is certifiably robust  $g(\pmb{X}) = g(\tilde{\pmb{X}})$  for any  $\tilde{\pmb{X}} \in \mathcal{B}_{2,\epsilon}^r(\pmb{X})$  if  $p_{\pmb{X},y} - \Delta > 0.5$ .
Proof in Appendix G. Interestingly, Corollary 1 and Corollary 2 show that hierarchical smoothing is a generalization of additive noise and ablation certificates. The additional column indicates which rows will be ablated, but instead of abating we only add noise to them. In contrast to ablation smoothing (which just checks if  $p_{X,y} - \Delta > 0.5$ ) we further "utilize" the budget  $p_{X,y} - \Delta$  by plugging it into the lower-level certificate. Notably, we are the first to generalize beyond ablation and additive noise certificates, and our certificates are in fact orthogonal to all existing robustness certificates that are based on randomized smoothing.
Figure 4: Hierarchical smoothing significantly expands the Pareto-front w.r.t. robustness and accuracy in node and image classification. Left: Discrete hierarchical smoothing for node classification, smoothed GAT on Cora-ML ( $r = 1$ ,  $r_d = 40$ ,  $r_a = 0$ ). Right: Continuous hierarchical smoothing for image classification, smoothed ResNet50 on CIFAR10 ( $r = 3$ ,  $\epsilon = 0.35$ ). Non-smoothed GAT achieves  $80\% \pm 2\%$  clean accuracy on Cora-ML, ResNet50  $94\%$  on CIFAR10. Large circles and stars are dominating points for each certificate. Dashed lines connect dominating points across methods.
7 Experimental evaluation
In this section we highlight the importance of hierarchical smoothing in image and node classification, instantiating our framework with two well-established smoothing distributions for continuous (Cohen et al., 2019) and discrete data (Bojchevski et al., 2020). In the following we present experiments supporting our main findings. We refer to Appendix A for more experimental results and to Appendix B for the full experimental setup and instructions to reproduce our results.
Threat models. For images we model adversaries that perturb at most  $r$  pixels of an image, where the perturbation over all channels is bounded by  $\epsilon$  under the  $\ell_2$ -norm. For graphs we model adversaries that perturb binary features of at most  $r$  nodes by inserting at most  $r_a$  and deleting at most  $r_d$  ones.
Datasets and models. For image classification we train ResNet50 (He et al., 2016) on CIFAR10 (Krizhevsky et al., 2009) consisting of images with three channels of size  $32 \times 32$  categorized into 10 classes. We certify the models on a random but fixed subset of 1,000 test images. For node classification we train graph attention networks (GATs) (Velickovic et al., 2018) with two layers on Cora-ML (McCallum et al., 2000; Bojchevski and Gunnemann, 2018) consisting of 2,810 nodes with 2,879 binary features, 7,981 edges and 7 classes. We train and certify GNNs in an inductive learning setting (Scholten et al., 2022). We defer results for more models and datasets to Appendix A.
Experimental setup. During training, we sample one smoothed matrix each epoch to train our models on smoothed data. Note that for hierarchical smoothing we also train our models on the higher-dimensional matrices. At test time, we use significance level  $\alpha = 0.01$ ,  $n_0 = 1,000$  samples for estimating the majority class and  $n_1 = 10,000$  samples for certification. We report the classification accuracy of the smoothed classifier on the test set (clean accuracy), and the certified accuracy, that is the number of test samples that are correctly classified and certifiably robust for a given radius. For node classification we report results averaged over five random graph splits and model initializations.
Baseline certificates. We compare hierarchical smoothing to additive noise and ablation certificates since our method generalizes both into a novel framework. To this end, we implement the two additive noise certificates of Cohen et al. (2019) and Bojchevski et al. (2020) for continuous and discrete data, respectively. Concerning ablation certificates, we implement the certificate of Levine and Feizi (2020b) for image classification, which ablates entire pixels. For node classification we implement the generalization to GNNs of Scholten et al. (2022) that ablates all node features of entire nodes. We conduct exhaustive experiments to explore the space of smoothing parameters for each method (see details in Appendix B). To compare the three different approaches we fix the threat model and investigate the robustness-accuracy trade-off by comparing certified accuracy against clean accuracy.
Hierarchical smoothing significantly expands the Pareto-front. Notably, hierarchical smoothing is clearly expanding the Pareto-front when optimization for both - certified accuracy and clean accuracy (see Figure 4). Especially when the number of adversarial nodes or pixels is small but the feature perturbation is large, hierarchical smoothing is significantly dominating both baselines.