SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 26: Randomized smoothing robustness guarantees for APPNP under attribute perturbations on Cora-ML. The Smoothing parameters are set to  $p_{\mathbf{X}}^{+} = 0.001$ ,  $p_{\mathbf{X}}^{-} = 0.8$ ,  $p_{\mathbf{A}}^{+} = 0$ ,  $p_{\mathbf{A}}^{-} = 0$ .
Figure 27: Randomized smoothing robustness guarantees for APPNP under adjacency perturbations on Cora-ML. The smoothing parameters are set to  $p_{\mathbf{X}}^{+} = 0, p_{\mathbf{X}}^{-} = 0, p_{\mathbf{A}}^{+} = 0.001, p_{\mathbf{A}}^{-} = 0.8$ .
Figure 28: Randomized smoothing robustness guarantees for APPNP under attribute perturbations on CiteSeer. The Smoothing parameters are set to  $p_{X}^{+} = 0.001$ ,  $p_{X}^{-} = 0.8$ ,  $p_{A}^{+} = 0$ ,  $p_{A}^{-} = 0$ .
Figure 29: Randomized smoothing robustness guarantees for APPNP under adjacency perturbations on CiteSeer. The smoothing parameters are set to  $p_{\mathbf{X}}^{+} = 0, p_{\mathbf{X}}^{-} = 0, p_{\mathbf{A}}^{+} = 0.001, p_{\mathbf{A}}^{-} = 0.8$ .
A.4 Adversarial attacks
As discussed in Section 5, any traditional adversarial attack is a valid adversarial attack under our proposed notion of adversarial robustness for equivariant tasks. This holds no matter if the equivariances of task  $y$  and model  $f$  match or not. As an example we conduct adversarial attacks on a PointNet classifier and a graph convolutional network. We use the same datasets, models and hyperparameters as for the experiments shown in Fig. 3 (without randomized smoothing) and Fig. 10. To attack the PointNet classifier, we use a single gradient step w.r.t. cross-entropy loss, which we scale to have an  $\ell_2$  norm of  $\epsilon$ . To attack the graph convolutional network, we use the method from Section 4.4 of [10], which is directly derived from their certification procedure. As can be seen, future work is needed to further improve the robustness of equivariant models to our proposed perturbation model.
(a) PointNet
(b) GCN
Figure 30: Adversarial attacks on (a) PointNet using a single gradient step with  $\ell_2$  norm  $\epsilon$  and (b) GCN (with perturbation costs  $c_{X}^{+} = c_{X}^{-} = 1$ ) using the method from Section 4.4 of [10].
B Full experimental setup
B.1 Point cloud classification
For our experiments on point cloud classification, we replicate the experimental setup from [26], where randomized smoothing was also applied to point cloud classification. We just use different base models, which are not rotation and translation invariant. Each experiment (including training) is repeated 5 times using different random seeds. We report the mean and standard deviation in certificate strength across all seeds.
B.1.1 Data
All experiments are performed on ModelNet[116] which consists of 9843 training samples and 2468 test samples from 40 classes. We apply the same preprocessing steps as in [114] to transform the CAD models into point clouds consisting of 1024 points, i.e. random sampling on the surface of the objects and normalization into the unit sphere. We use  $20\%$  of the training data for validation. All certificates are evaluated on the test data.
We use two different models, PointNet [114] and DGCNN [115].
For PointNet, we use three linear layers (64, 128, 1024 neurons) before pooling and three layers (512, 256, 40 neurons) after pooling. Before feeding input point clouds into the model, we multiply them with a matrix predicted by a T-Net, i.e. another PointNet with three linear layers (64, 128, 1024 neurons) before pooling and three linear layers (512, 256, 3·3 neurons) after pooling. We use batch normalization ( $\epsilon = 1e - 5$ , momentum  $= 0.1$ ) for all layers, except the output layer. We use dropout ( $p = 0.4$ ) for the second-to-last layer.
For DGCNN, we use four dynamic graph convolution layers (64, 64, 64, 128 neurons) with k-nearest neighbor graph construction ( $k = 20$ ). The outputs of all four layers are concatenated, passed through a linear layer (1024 neurons), then max-pooled across the node dimension and finally passed through three more linear layers (512, 256, 40 neurons). We use batch normalization for all layers ( $\epsilon = 1e - 5$ , momentum  $= 0.1$ ). We use dropout ( $p = 0.4$ ) for the second-to-last and third-to-last layer. Before feeding input point clouds into the model, we multiply them with a  $3 \times 3$  matrix predicted by the same T-Net architecture we described for PointNet.
B.1.3 Training
We use the same training parameters as in [26]. We train for 200 (PointNet) or 400 (DGCNN) epochs using Adam  $(\beta_{1} = 0.9, \beta_{2} = 0.99, \epsilon = 1e^{-8}$ , weight Decay  $= 1e^{-4}$ ) with a learning rate of 0.001 and exponential weight decay with factor 0.7 every 20 (PointNet) or 40 (DGCNN) epochs. We use a batch size of 128 for PointNet and 32 for DGCNN. We add the T-Net loss proposed by [114] to the cross entropy loss with a weight of 0.0001. The data is randomly augmented via scaling by a factor uniformly sampled from [0.8, 1.25]. We additionally add isotropic Gaussian noise with the same standard deviation we use for randomized smoothing (0.05, 0.1, 0.15, 0.2 or 0.25).
B.1.4 Certification parameters
We perform randomized smoothing with isotropic Gaussian noise ( $\sigma \in \{0.05, 0.1, 0.15, 0.2$  or  $0.25\}$ ) and majority voting [20]. We use 1,000 Monte Carlo samples for prediction and 10,000 Monte Carlo samples for certification. We compute the Clopper-Pearson confidence intervals for class probabilities with  $\alpha = 0.001$ , i.e. all guarantees hold with high probability  $99.9\%$ .
B.1.5 Computational resources
All experiments involving PointNet were performed on a Xeon E5-2630 v4 CPU @ 2.20 GHz with a NVIDIA GTX 1080TI GPU. The average time for training a model was  $21.1\mathrm{min}$ . The average time for certifying the robustness of a single model on the entire test set for 50 different budgets  $\epsilon$  was  $81.7\mathrm{min}$ .
All experiments involving DGCNN were performed on an AMD EPYC 7543 CPU @ 2.80GHz with a NVIDIA A100 (40 GB) GPU. The average time for training a model was  $161.9\mathrm{min}$ . The average time for certifying the robustness of a single model on the entire test set for 50 different budgets  $\epsilon$  was  $601.7\mathrm{min}$ .
B.2 Molecular force prediction
Like with point cloud classification, we train and certify 5 different models using different random seeds on a common benchmark dataset. We report the mean and standard deviation in certificate strength across all seeds.
B.2.1 Data
We use the original (not the revised) MD17 [118] collection of datasets. It consists of 8 different datasets, each consisting of a large number (between 133 770 and 993 237) of spatial configurations of a specific molecule. For each dataset, we use 1000 randomly chosen configurations for training, 1000 for validation and 1000 for evaluating the robustness guarantees. We use the same data split for all experiments.
We use DimeNet++ [117], SchNet [119] and SphereNet [120] as our base model. All models predict atomic energies. Force are calculated via automatic differentiation w.r.t. the input coordinates.
For DimeNet++, we use the default model parameters for MD17 force predictions specified in [117]. There are 4 layers with 128 hidden channels. The triplet embedding size is 64. The basis embedding size is 8. The output embedding size is 256. The number of basis functions is 8 (bilinear), 7 (spherical), 6 (radial). The cutoff radius for graph construction is  $5\AA$ . The number of residual layers before the skip connection is 1. The number of residual layers after the skip connection is 2.
For SchNet, we use 128 hidden channels and 128 filters. We set the number of interaction blocks to 6 and use 50 Gaussians. The cutoff distance is  $10\AA$ , with a maximum number of 32 neighbors. We use addition as our global readout function.
For SphereNet, we use 4 layers with 128 hidden channels. The triplet embedding size is 64. The basis embedding size is 8 for distance, angle and torsion. The output embedding size is 256. The number of basis functions is 8 (bilinear), 7 (spherical), 6 (radial). The cutoff radius for graph construction is  $5\mathring{\mathrm{A}}$ . We use swish activation functions. The number of residual layers before the skip connection is 1. The number of residual layers after the skip connection is 2.
B.2.3 Training parameters
To train DimeNet++ and Schnet, we use ADAM with a learning rate of 0.001. We use linear warmup with exponential decay as our learning rate scheduler (1000 warmup steps, 400,000 decay steps, decay rate of 0.01) and train until convergence with a patience of 50 epochs and a convergence threshold of  $10^{-4}$ .
To train SphereNet, we use ADAM with a learning rate of 0.001. We use "reduce on plateau" as our learning rate scheduler (decay factor of 0.8, patience of 80 epochs, convergence threshold of  $10^{-4}$ , cooldown of 10 epochs) and train until convergence with a patience of 50 epochs and a convergence threshold of  $10^{-4}$ .
We do not add randomized smoothing noise during the training process.
B.2.4 Certification parameters
We perform randomized smoothing with isotropic Gaussian noise  $(\sigma \in \{1\mathrm{fm}, 10\mathrm{fm}, 100\mathrm{fm}\})$  and center smoothing [93] with default parameters  $\alpha_{1} = 0.005, \alpha_{2} = 0.005, \Delta = 0.05$ , i.e. all guarantees hold with high probability  $99\%$ . We use 10,000 samples for prediction, 10,000 samples to test for abstention and 10,000 samples for certification. In our experiments the smoothed model never abstained.
B.2.5 Computational resources
All experiments for molecular force prediction were performed on a Xeon E5-2630 v4 CPU @ 2.20 GHz with a NVIDIA GTX 1080TI GPU. We trained 4 models simultaneously. The average time for training a DimeNet++ model was  $28.9\mathrm{h}$ . The average time for certifying the robustness of a single DimeNet++ model on 1000 molecule configurations for 1000 different budgets  $\epsilon$  was  $10.49\mathrm{h}$ .
B.3 Node and graph classification
In Section 5.2 we generalized six approaches for proving the robustness of graph neural networks with respect to graph edit distances. While the approaches differ, we mostly use the same datasets and models. As before, we perform each experiment (including training) with 5 different random seeds and report the mean and standard deviation in certificate strength across all seeds.
B.3.1 Data
We use two standard node classification datasets, Cora-ML [121, 122] (2,708 nodes, 10,556 edges, 1,433 features, 7 classes) and CiteSeer [124](3,327 nodes, 9,104 edges, 3,703 features, 6 classes), and two graph classification sets, PROTEINS (1,113 graphs, 3 features, 2 classes) and MUTAG (188 graphs, 7 features, 2 classes), which are part of the TUDataset [125]. These datasets were also used in the papers that proposed the original certification procedures. With Cora-ML and CiteSeer, we follow the procedure from [14] and use 20 nodes per class for training, 20 nodes per class for validation and the remainder for evaluating the certificates. With PROTEINS and MUTAG, we use  $30\%$  of graphs for training,  $20\%$  for validation and the remainder for evaluating the certificates.
B.3.2 Models
The main model we use for node classification is a standard 2-layer graph convolutional network with ReLU nonlinearities and 32 hidden features. We insert self-loops and perform degree normalization via  $D^{-1/2}AD^{-1/2}$ .
For node classification, we use the architecture described in [15], i.e. a single layer GCN with 64 neurons, followed by a linear layer and max-pooling. We also insert self-loops, but perform degree normalization via  $D^{-1}A$ .
With sparsity-aware randomized smoothing, we additionally use graph attention networks [128] and APPNP [122]. We implement all GNNs with two-layers and 32 hidden dimensions, except GAT for which we implement 8 hidden dimensions and 8 attention heads. For APPNP we further set  $\mathrm{k\_hops} = 10$  and teleport probability  $\alpha = 0.15$ .
For the experiments with certificates for the  $\pi$ -PPNP architecture we set teleport probability  $\alpha = 0.15$  and use a hidden dimension of 64.
B.3.3 Training parameters
We train models for a maximum of 1,000 ( $\pi$ -PPNP), 3,000 (other node classifiers) or 200 (graph classifiers) epochs with Adam ( $lr = 1e - 3$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.99$ ,  $\epsilon = 1e - 8$ ) and cross entropy loss. For node classification we additionally use weight decay  $5e - 4$  and early stopping after 50 epochs. For node-classification we perform full-batch gradient descent. For graph classification we use minibatches of size 20. For the sparsity-ware randomized smoothing experiments we use a dropout of 0.5 on the hidden node representations after the first graph convolution for GCN and GAT, and additionally on the attention coefficients for GAT. For randomized smoothing we additionally add sparsity-aware noise to the training data. We use the same flip probabilities as for certification.
B.3.4 Certification parameters
For interval bound propagation and the convex outer adversarial polytope method, we set the nodewise local budgets  $\rho_{1},\ldots ,\rho_{N}$  to  $1 \%$  of input features, just like in [10].
For our generalization of the graph classification certificate from, we set the local budget  $\rho_{n}$  of node  $n$  with degree  $d_{n}$  to  $\min(0, d_{n} - \max_{m} d_{m} + 3)$ , just like in [15]. We additionally enforce symmetry
of the adjacency matrix via Lagrange dualization (see Appendix A.3) and perform 200 alternating optimization steps using the default parameters from [15].
We perform randomized smoothing with sparse noise  $(p_{\mathbf{X}}^{+} = 0.001, p_{\mathbf{X}}^{-} = 0.8, p_{\mathbf{A}}^{+} = 0, p_{\mathbf{A}}^{-} = 0$  or  $p_{\mathbf{X}}^{+} = 0, p_{\mathbf{X}}^{-} = 0, p_{\mathbf{A}}^{+} = 0.001, p_{\mathbf{A}}^{-} = 0.8)$  and majority voting [14]. For node classification we use 1,000 Monte-Carlo samples for prediction and 1,000,000 Monte-Carlo samples for certification. For graph classification we use 1,000 Monte-Carlo samples for prediction and 10,000 Monte-Carlo samples for certification. We compute the Clopper-Pearson confidence intervals for class probabilities with  $\alpha = 0.01$ , i.e. all guarantees hold with high probability  $99\%$ .
B.3.5 Computational resources
All experiments were performed on a Xeon E5-2630 v4 CPU @ 2.20 GHz with a NVIDIA GTX 1080TI GPU. With interval bound propagation, the average time of training and certifying a model for 100 budgets  $\epsilon$  was  $1.2\mathrm{min}$  on Cora-ML and  $3.0\mathrm{min}$  on CiteSeer. With the convex outer adversarial polytope method, the average time of training and certifying a model for 100 budgets  $\epsilon$  was  $9.7\mathrm{min}$  on Cora-ML and  $14.4\mathrm{min}$  on CiteSeer. With the linearization and dualization method for graph classifiers, the average time of the average time of training and certifying a model for 100 budgets  $\epsilon$  on the entire test set was  $1.2\mathrm{h}$  on MUTAG and  $48.1\mathrm{h}$  on PROTEINS. With the policy iteration method for  $\pi$ -PPNP, the average time for training and certifying a model for 10 attack strengths was  $126.1\mathrm{min}$ . With sparsity-aware randomized smoothing, the average time for training and certifying a model for 1000 budgets  $\epsilon$  was  $267.7\mathrm{min}$ .
B.4 Third-party assets
Since we extend various existing methods for proving the robustness of graph neural networks, we naturally built upon their reference implementations. For interval bound propagation and convex outer adversarial polytopes, we extend the reference implementation from [10] (https://github.com/danielzuegner/robust-gcn). For the linearization and dualization method, we extend the reference implementation from [15] (https://github.com/RobustGraph/RoboGraph). For the policy iteration method, we extend the reference implementation from [11] (https://github.com/abojchevski/graph_cert). For sparsity-aware randomized smoothing on graphs, we use the reference implementation from [14] (https://github.com/abojchevski/sparse_smoothing).
For equivariance-preserving randomized smoothing on point clouds, we use the training and sampling procedure from [26] (https://github.com/jan-schuchardt/invariance-smoothing).
All of the above are available under MIT license.
To train models for molecular force prediction, we use a pre-release version of code from [129] (https://github.com/wolfschl/uncertainty_for_molecules), which we will include in our reference implementation.
In the following, we show that the action-induced distance is the only function  $\hat{d}_{\mathrm{in}}:\mathbb{X}\times \mathbb{R}\to \mathbb{R}_+$  that fulfills all three desiderata defined in Section 4.1. These were
-  $\forall x, x' \in \mathbb{X}, g \in \mathbb{G}: \hat{d}_{\mathrm{in}}(x, g \bullet x') = \hat{d}_{\mathrm{in}}(x, x')$ ,
-  $\forall x, x': \hat{d}_{\mathrm{in}}(x, x') \leq d_{\mathrm{in}}(x, x')$
-  $\hat{d}_{\mathrm{in}}(x,x^{\prime}) = \max_{\gamma \in \mathbb{D}}\gamma (x,x^{\prime}),$
where  $\mathbb{D}$  is the set of functions from  $\mathbb{X} \times \mathbb{X}$  to  $\mathbb{R}_+$  that fulfill the first two desiderata
Proof. Consider a specific pair  $x, x' \in \mathbb{X}'$ . The first desideratum states that we must have  $\hat{d}_{\mathrm{in}}(x, x') = \hat{d}_{\mathrm{in}}(x, g \bullet x')$  for all  $g \in \mathbb{G}$ . The second desideratum states that we must have  $\hat{d}_{\mathrm{in}}(x, g \bullet x') \leq d_{\mathrm{in}}(x, g \bullet x')$  for all  $g \in \mathbb{G}$ . Thus, by the transitive property, fulfilling both desiderata simultaneously for our specific  $x$  and  $x'$  is equivalent to  $\forall g \in \mathbb{G}: \hat{d}_{\mathrm{in}}(x, x') \leq d_{\mathrm{in}}(x, g \bullet x')$ . This is equivalent to  $\hat{d}_{\mathrm{in}}(x, x') \leq \min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, g \bullet x')$ . The left side of the equality is naturally maximized to fulfill the third desideratum when we have a strict equality, i.e.  $\hat{d}_{\mathrm{in}}(x, x') = \min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, g \bullet x')$ .
D Combining smoothing measures and smoothing schemes
Most randomized smoothing literature can be divided into works that focus on investigating properties of different smoothing measures (e.g. [14, 20, 60, 64, 65, 94, 95] and works that propose new smoothing schemes (e.g. [20, 75, 77, 93]). Measure-focused works usually only consider classification tasks, while scheme-focused works usually only consider Gaussian measures. But, as we shall detail in the following, any of the smoothing schemes discussed in Section 5.1 can be used with arbitrary smoothing measures to enable certification for various input distances  $d_{\mathrm{in}}$  and output distances  $d_{\mathrm{out}}$ .
D.1 Smoothing measures
Measure-focused works usually consider a base classifier  $h: \mathbb{X} \to \mathbb{Y}$  with  $\mathbb{Y} = \{1, \dots, K\}$ . They define a family of smoothing measures  $(\mu_x)_{x \in \mathbb{X}}$  to construct the smoothed classifier  $f(x) = \max_{k \in \mathbb{Y}} \operatorname*{Pr}_{z \sim \mu_x}[h(z) = k]$ . To certify the predictions of such a smoothed classifier, they derive lower and upper bounds on the probability of classifying a perturbed input  $x' \in \mathbb{X}$  as a specific class  $k \in \{1, \dots, k\}$ , i.e.  $p_{h,x',k} \leq \operatorname*{Pr}_{z \sim \mu_{x'}}[h(z) = k] \leq p_{h,x',k}$ . These bounds are usually obtained by finding the least / most robust model that has the same clean prediction probability as base classifier  $h$ , i.e.
$$
\underline {{p _ {h , x ^ {\prime} , k}}} = \min  _ {\tilde {h}: \mathbb {X} \to \mathbb {Y}} \Pr_ {z \sim \mu_ {x ^ {\prime}}} \left[ \tilde {h} (z) = k \right] \text {s . t .} \Pr_ {z \sim \mu_ {x}} \left[ \tilde {h} (z) = k \right] = \Pr_ {z \sim \mu_ {x}} \left[ h (z) = k \right],
$$
$$
\overline {{p _ {h , x ^ {\prime} , k}}} = \max  _ {\tilde {h}: \mathbb {X} \to \mathbb {Y}} \Pr_ {z \sim \mu_ {x ^ {\prime}}} \left[ \tilde {h} (z) = k \right] \text {s . t .} \Pr_ {z \sim \mu_ {x}} \left[ \tilde {h} (z) = k \right] = \Pr_ {z \sim \mu_ {x}} \left[ h (z) = k \right]
$$
As long as  $\mu_x$  and  $\mu_{x'}$  have a density, these problems can always be solved exactly via the Neyman-Pearson lemma [130].
For appropriately chosen smoothing measures  $(\mu_x)_{x\in \mathbb{X}}$ , one can then identify that the perturbed probability bounds  $\underline{p_{h,x',k}}$  and  $\overline{p_{h,x',k}}$  only depend on a certain distance function  $d_{\mathrm{in}}:\mathbb{X}\times \mathbb{X}\to \mathbb{R}_{+}$ . For example:
- For  $\mathbb{X} = \mathbb{R}^N$ , and  $\mu_{\boldsymbol{x}} = \mathcal{N}(\boldsymbol{x}, \sigma \cdot \boldsymbol{I})$ , the bounds  $\underline{p_{h, x', k}}$  and  $\overline{p_{h, x', k}}$  only depend on  $d_{\mathrm{in}}(\boldsymbol{x}, \boldsymbol{x}') = ||\boldsymbol{x} - \boldsymbol{x}'||_2$  and are monotonically decreasing / increasing [20].
- For  $\mathbb{X} = \mathbb{R}^N$ , and  $\mu_{\boldsymbol{x}} = \operatorname{Laplace}(\boldsymbol{x}, \sigma \cdot \boldsymbol{I})$ , the bounds  $\overline{p_{h, x', k}}$  and  $\overline{p_{h, x', k}}$  only depend on  $d_{\mathrm{in}}(\boldsymbol{x}, \boldsymbol{x}') = ||\boldsymbol{x} - \boldsymbol{x}'||_1$  and are monotonically decreasing / increasing [95].
- For  $\mathbb{X} = \{0,1\}$ , and  $\mu_{\pmb{x}}$  being the measure associated with i.i.d. flipping of input bits, the bounds  $\overline{p_{h,x',k}}$  and  $\overline{p_{h,x',k}}$  only depend on  $d_{\mathrm{in}}(\pmb{x},\pmb{x}') = ||\pmb{x} - \pmb{x}'||_0$  and are monotonically decreasing / increasing [94].
For an overview of various additive smoothing schemes and their associated input distances, see [95]. Note that works on non-additive smoothing measures, like ablation smoothing [131] or derandomized smoothing [132], also provide bounds  $\underline{p_{h,x',k}}$  and  $\overline{p_{h,x',k}}$  as a function of some input distance  $d_{\mathrm{in}}$ , such as the number of perturbed pixels or the size of an adversarial image patch.
D.2 Smoothing schemes
Scheme-focused works include majority voting [20], expected value smoothing [75], median smoothing [77] and center smoothing [93]. These works only consider Gaussian smoothing, i.e.  $\mathbb{X} = \mathbb{R}^N$  and  $\mu_{\boldsymbol{x}} = \mathcal{N}(\boldsymbol{x}, \sigma \cdot \boldsymbol{I})$ . Each one proposes a different prediction procedure and certification procedure. The prediction procedures provide a sampling-based approximations of the smoothed prediction  $\xi(\mu \circ h^{-1})$  and does not depend on the choice of smoothing measure. The certification procedures can be adapted to arbitrary smoothing measures as follows.
Majority voting. Majority voting assumes a base classifier  $h: \mathbb{X} \to \mathbb{Y}$  with  $\mathbb{Y} = \{1, \dots, K\}$ . It constructs a smoothed classifier  $f(x) = \max_{k \in \mathbb{Y}} \operatorname{Pr}_{z \sim \mu_x}[h(z) = k]$ . The prediction  $f(x) = k^*$  of this smoothed classifier is robust if  $\underline{p_{h,x',k^*}} > \max_{k \neq k^*} \underline{p_{h,x',k}}$  or if  $\underline{p_{h,x',k^*}} > 0.5$ . This certification procedure can be adapted to any other smoothing measure by inserting the corresponding lower and
3 or continuous relaxations thereof, if any constant likelihood region has non-zero measure
upper bounds. It can be generalized to multi-output classification tasks with  $\mathbb{Y} = \{1,\dots ,K\} ^M$  (e.g. segmentation) by applying the certification procedure independently to each output dimension.