SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Special cases. If the task is not equivariant, i.e.  $\mathbb{G} = \{e\}$ , we recover the traditional notion of adversarial robustness because  $e$  acts via identity. For  $\epsilon = 0$ , constant  $\delta$  is a bound on the equivariance error, a common evaluation metric in geometric machine learning (see, e.g. [86-90]).
Other invariant distances. As discussed in Section 2, alternative invariant distances have been proposed for certain group invariant classification tasks. This includes Hausdorff and Chamfer
distance [30, 31], as well as optimal transport for graphs [27-29]. Any group invariant function  $d_{\mathrm{in}}$  is preserved by the action-induced distance:  $\min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, g \bullet x') = \min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, x') = d_{\mathrm{in}}(x, x')$ . Thus, all previous results for invariant classification are compatible with our notion of robustness.
Local budgets and local robustness. For data that is composed of  $N$  distinct elements, such as point clouds, one may want local budgets  $\epsilon_1, \ldots, \epsilon_N$ . For tasks that involve  $M$  distinct predictions, such as segmentation, one may only be interested in the robustness of some subset of predictions. Definition 1 can be extended to accommodate local budgets and robustness, see Appendix G.
5 Achieving provable robustness for group equivariant tasks
Now we have a sound notion of robustness that overcomes the limitations discussed in Section 1. But it is not clear how to achieve provable robustness. Given a task  $y: \mathbb{X} \to \mathbb{Y}$  that is equivariant with respect to the action of a group  $\mathbb{G}$ , we want a model  $f: \mathbb{X} \to \mathbb{Y}$  and a corresponding algorithm that can verify the  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness of a prediction  $f(x)$ .
A challenge is that the action-induced distance  $\hat{d}_{\mathrm{in}}$  , which defines our perturbation model and thus underlies the optimization domain in Eq. (1), is generally not tractable. For example, deciding whether the graph edit distance is smaller than some  $\epsilon \in \mathbb{R}$  is NP-hard [91]. A solution would be to relax the optimization domain in order to pessimistically bound the model's actual robustness. This is, in essence, the approach taken by works on robustness to graph optimal transport perturbation models (e.g. [27, 28]). Instead of optimizing over discrete correspondences between objects, they optimize over couplings that define soft correspondences. There is however a more straight-forward solution that lets us take advantage of years of research in geometric and robust machine learning: Applying the principles of geometric machine learning and using a model with matching equivariances.
Proposition 2. Consider a model  $f: \mathbb{X} \to \mathbb{Y}$  that is equivariant with respect to the action of a group  $\mathbb{G}$ . Any prediction  $f(x)$  is  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robust if and only if it is  $(\{e\}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robust, i.e. fulfills traditional adversarial robustness.
$$
\begin{array}{l} \text {P r o o f . B e c a u s e f i s e q u i v a r i a n t , w e h a v e \forall g : g ^ {- 1} \bullet f (g \bullet x ^ {\prime}) = g ^ {- 1} \bullet g \bullet f (x ^ {\prime}) = f (x ^ {\prime}) =} \\ e ^ {- 1} \bullet f (e \bullet x ^ {\prime}) \text {a n d t h u s d _ {o u t} (f (x) , g ^ {- 1} \bullet f (g \bullet x ^ {\prime})) = d _ {o u t} (f (x) , e ^ {- 1} \bullet f (e \bullet x ^ {\prime}))}. \end{array}
$$
By using model  $f$  with the same equivariances as ground truth  $y$ , we reduce the problem of proving robustness to that of proving traditional adversarial robustness i.e.  $\mathbb{G} = \{e\}$ . Again, note that robustness does not mean that a prediction remains constant, but that it transforms in compliance with semantics-preserving transformations of the input, even under small perturbations (see Eq. (1)).
Discussion. Proposition 2 provides another strong argument for the use of geometric machine learning models. These models facilitate the problem of achieving provable robustness for group equivariant tasks — without relaxing the action-induced distance and thus weakening our guarantees. We can instead build upon existing robustness certification procedures for equivariant models, such as transformers, PointNet and graph convolutional networks, under traditional perturbation models, like  $\ell_2$  or  $\ell_0$  perturbations [8-17, 34-46]. We just need to use our knowledge about the equivariances of tasks and models to reinterpret what is actually certified by these procedures.
Relation to orbit-based certificates. A related result was discussed for group invariant point cloud classifiers in [26]: If  $\mathbb{X} = \mathbb{R}^{N\times D}$  and classifier  $f$  is constant within Frobenius norm ball  $\mathbb{B}$  and invariant w.r.t. the action of group  $\mathbb{G}$ , then it is also constant within  $\{g\bullet X^{\prime}\mid X^{\prime}\in \mathbb{B},g\in \mathbb{G}\}$ . However, this work did not discuss whether being constant within this set is desirable, how it relates to task invariance and what this result tells us about the adversarial robustness of the classifier.
Adversarial attacks. While we focus on provable robustness, Proposition 2 also applies to adversarial attacks, i.e. proving non-robustness via counterexample. Even when the equivariances of  $y$  and  $f$  do not match, traditional attacks are feasible solutions to Eq. (1) since they amount to constraining  $g$  to set  $\{e\}$ . For completeness, we perform experiments with adversarial attacks in Appendix A.4.
5.1 Equivariance-preserving randomized smoothing
Equivariant models reduce the problem of proving robustness for group equivariant tasks to that of proving traditional robustness. However, specialized procedures to make these proofs are only available for a limited range of architectures. For example, specialized certification procedures for
point cloud models are limited to the PointNet architecture [42, 92], and there are no procedures to prove the robustness of models with continuous equivariances, such as rotation equivariance.
For such models, one could try to apply randomized smoothing (recall Section 3). By choosing a suitable smoothing scheme  $\xi$  and measures  $(\mu_x)_{x\in \mathbb{X}}$  for distances  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$ , one can transform any base model  $h$  into smoothed model  $f$  and prove its  $\{(e\}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness. However, base model  $h$  having the same equivariances as task  $y$  does not guarantee that smoothed model  $f$  has the same equivariances. Thus, one can generally not use Proposition 2 to prove  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness. We propose the following sufficient condition for verifying that a specific smoothing scheme and family of measures are equivariance-preserving (proof in Appendix E.1).
Proposition 3. Assume two measurable spaces  $(\mathbb{X},\mathbb{D})$ $(\mathbb{V},\mathbb{F})$  , an output space  $\mathbb{Y}$  and a measurable base model  $h:\mathbb{X}\to \mathbb{V}$  that is equivariant with respect to the action of group  $\mathbb{G}$  . Further assume that  $\mathbb{G}$  acts on  $\mathbb{X}$  and  $\mathbb{V}$  via measurable functions. Let  $\xi :\Delta (\mathbb{V},\mathbb{F})\to \mathbb{Y}$  be a smoothing scheme that maps from the set of probability measures  $\Delta (\mathbb{V},\mathbb{F})$  on intermediate space  $(\mathbb{V},\mathbb{F})$  to the output space. Define  $T_{\mathbb{X},g}(\cdot)$  to be the group action on set  $\mathbb{X}$  for a fixed  $g$  , i.e.  $T_{\mathbb{X},g}(x) = g\bullet_{\mathbb{X}}x$  . Then, the smoothed model  $f(x) = \xi (\mu_x\circ h^{-1})$  is equivariant with respect to the action of group  $\mathbb{G}$  if both
- the family of measures  $(\mu_x)_{x \in \mathbb{X}}$  is equivariant, i.e.  $\forall x \in \mathbb{X}, g \in \mathbb{G}: \mu_{g \bullet x} = \mu_x \circ T_{\mathbb{X}, g}^{-1}$ .
- and smoothing scheme  $\xi$  is equivariant, i.e.  $\forall \nu \in \Delta (\mathbb{V},\mathbb{F}),g\in \mathbb{G}:\xi (\nu \circ T_{\mathbb{V},g}^{-1}) = g\bullet \xi (\nu)$
Note that  $\circ$  is a composition of functions, whereas  $\bullet$  is a group action. The intuition behind Proposition 3 is that, if family of measures  $(\mu_x)_{x\in \mathbb{X}}$ , and base model  $h$ , and smoothing scheme  $\xi$  are equivariant, then we have a chain of equivariant functions which is overall equivariant. We provide a visual example of such an equivariant chain of functions in Fig. 31.
In the following, we show that various schemes and measures preserve practically relevant equivariances and can thus be used in conjunction with Proposition 2 to prove  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness. For the sake of readability, we provide a high-level discussion, leaving the formal propositions in Appendix E.2. We summarize our results in Tables 1 and 2. In Appendix D, we discuss how to derive robustness guarantees for arbitrary combinations of these schemes and measures.
Componentwise smoothing schemes. The most common type of smoothing scheme can be applied whenever the intermediate and output space have  $M$  distinct components, i.e.  $\mathbb{V} = \mathbb{A}^M$  and  $\mathbb{Y} = \mathbb{B}^M$  for some  $\mathbb{A}, \mathbb{B}$ . It smooths each of the  $M$  base model outputs independently. This includes majority voting [20], expected value smoothing [75, 76] and median smoothing [77], which can be used for tasks like classification, segmentation, node classification, regression, uncertainty estimation and object detection [14, 20, 51, 75-81]. Such schemes preserve equivariance to groups acting on  $\mathbb{V}$  and  $\mathbb{Y}$  via permutation. Note that  $\mathbb{G}$  need not be the symmetric group  $S_N$  to act via permutation. For instance, the identity function is a permutation, meaning componentwise smoothing schemes can be used to prove robustness for arbitrary group invariant tasks. See Proposition 4.
Expected value smoothing scheme. When the intermediate and output space are real-valued, i.e.  $\mathbb{V} = \mathbb{Y} = \mathbb{R}^M$ , one can make predictions via the expected value [75, 76]. Due to linearity of expectation, this scheme does not only preserve permutation equivariance but also equivariance to affine transformations. However, certifying the smoothed predictions requires that the support of the output distribution is bounded by a hyperrectangle, i.e.  $\mathrm{supp}(\mu_x\circ h^{-1})\subseteq \{\pmb {y}\in \mathbb{R}^M\mid \forall m:a_m\leq y_m\leq b_m\}$  for some  $a,b\in \mathbb{R}^{M}$  [75, 76]. See Proposition 5.
Median smoothing scheme. When the support of the model's output distribution is real-valued and potentially unbounded, one can make smoothed predictions via the elementwise median [77]. This scheme does not only preserve permutation equivariance but also equivariance to elementwise linear transformations, such as scaling. See Proposition 6.
Center smoothing scheme. Center smoothing [93] is a flexible scheme that can be used whenever  $d_{\mathrm{out}}$  fulfills a relaxed triangle inequality. It predicts the center of the smallest  $d_{\mathrm{out}}$  ball with measure of at least  $\frac{1}{2}$ . Center smoothing has been applied to challenging tasks like image reconstruction, dimensionality reduction, object detection and image segmentation [93, 51]. We prove that center smoothing is equivariant to any group acting isometrically w.r.t.  $d_{\mathrm{out}}$ . For example, when  $\mathbb{V} = \mathbb{Y} = \mathbb{R}^{M\times D}$  and  $d_{\mathrm{out}}$  is the Frobenius norm, center smoothing guarantees  $(\mathbb{G},d_{\mathrm{in}},d_{\mathrm{out}},\epsilon ,\delta)$ -equivariant-robustness for any group acting on  $\mathbb{R}^{N\times D}$  via permutation, rotation, translation or reflection. See Proposition 7.
Product measures. Many randomized smoothing methods use product measures. That is, they use independent noise to randomize elements of an  $N$ -dimensional input space  $\mathbb{X} = \mathbb{A}^N$ . The most
popular example are exponential family distributions, such as Gaussian, uniform and Laplacian noise, which can be used to prove robustness for various  $\ell_p$  distances [19, 20, 94, 95], Mahalanobis distance [60, 96] and Wasserstein distance [97]. Product measures preserve equivariance to groups acting via permutation. Again, group  $\mathbb{G}$  need not be symmetric group  $S_{n}$  to act via permutation. For instance, rotating a square image by  $90^{\circ}$  (see Fig. 2) is a permutation of its pixels. See Proposition 8.
Isotropic Gaussian measures. Isotropic Gaussian measures are particularly useful for  $\mathbb{X} = \mathbb{R}^{N\times D}$ . They guarantee robustness when  $d_{\mathrm{in}}$  is the Frobenius distance [43, 46] and preserve equivariance to isometries, i.e. permutation, rotation, translation and reflection. Combined with Proposition 2, this guarantees robustness w.r.t. the aforementioned point cloud registration distance. See Proposition 9.
Transformation-specific measures. A standard tool for proving transformation-specific robustness (see Section 2) is transformation-specific smoothing [46, 60, 63-65], i.e. applying randomly sampled transformations from a parametric family  $(\psi_{\theta})_{\theta \in \Theta}$  with  $\psi_{\theta}:\mathbb{X}\to \mathbb{X}$  to the inputs of a model. If all  $\psi_{\theta}$  with  $\theta \in \Theta$  are equivariant to the actions of group  $\mathbb{G}$ , then transformation-specific smoothing preserves this equivariance. For example, random scaling [65] preserves rotation equivariance and additive noise (e.g. Gaussian) preserves translation equivariance for  $\mathbb{X} = \mathbb{R}^{N\times D}$ . See Proposition 10.
Sparsity-aware measures. Sparsity-aware noise [14] can be applied to discrete graph-structured data to guarantee robustness to edge and attribute insertions and deletions. We prove that sparsity-aware measures preserve equivariance to groups acting via graph isomorphisms. As we discuss in the next section, this guarantees robustness w.r.t. the graph edit distance. See Proposition 11.
5.2 Deterministic edit distance certificates for graph and node classification
Besides sparsity-aware smoothing, there are also deterministic procedures for proving robustness for specific graph neural networks, namely graph convolutional networks [98] and APPNP [99]. They are however limited to uniform costs. To be more specific, let  $\mathbb{X}$  be the set of all graphs  $\{0,1\}^{N\times D}\times \{0,1\}^{N\times N}$  with  $N$  nodes and  $D$  binary attributes and let  $(A)_{+} = \max (A,0)$  and  $(A)_{-} = \min (A,0)$  with elementwise maximum and minimum. Define  $d_{\mathrm{in}}((X,A),(X',A'))$  as
$$
c _ {\boldsymbol {X}} ^ {+} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {+} | | _ {0} + c _ {\boldsymbol {X}} ^ {-} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {-} | | _ {0} + c _ {\boldsymbol {A}} ^ {+} \cdot | | (\boldsymbol {A} ^ {\prime} - \boldsymbol {A}) _ {+} | | _ {0} + c _ {\boldsymbol {A}} ^ {-} \cdot | | (\boldsymbol {A} ^ {\prime} - \boldsymbol {A}) _ {-} | | _ {0}, \tag {2}
$$
with costs  $c_{\mathbf{X}}^{+}, c_{\mathbf{X}}^{-}, c_{\mathbf{A}}^{+}, c_{\mathbf{A}}^{-}$  for insertion and deletion of attributes and edges. Prior work can only prove robustness for costs in  $\{\infty, 1\}$ , i.e. disallow certain types of perturbations and use uniform cost for the remaining ones. In Appendix F we generalize each existing deterministic graph and node classification guarantee to non-uniform costs. This includes procedures based on convex outer adversarial polytopes [10], policy iteration [11], interval bound propagation [12], bilinear programming [13], and simultaneous linearization and dualization [15]. Our proofs mostly require solving different knapsack problems with local constraints and only two distinct costs - which can be done efficiently via dynamic programming or linear relaxations with analytic solutions (see Appendix F.1). As per Proposition 2, proving  $(\{e\}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness of isomorphism equivariant models in this way proves  $(S_N, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robustness. Thus, these procedures guarantee robustness w.r.t graph edit distance  $d_{\mathrm{in}}((\mathbf{X}, \mathbf{A}), (\mathbf{X}', \mathbf{A}') = \min_{\mathbf{P} \in S_N} d_{\mathrm{in}}((\mathbf{X}, \mathbf{A}), (\mathbf{P}\mathbf{X}', \mathbf{P}\mathbf{A}'\mathbf{P}^T))$ .
5.3 Limitations
Group equivariance covers various common task symmetries. However, there are symmetries that do not fit into this framework, such as local gauge equivariance [100-102] or wave function symmetries [103-112]. A limitation of relying on group equivariant models for provable robustness is that there are tasks where non-equivariant models have better empirical performance, for example vision transformers [113]. Models that are in principle equivariant may also lose their equivariance due to domain-specific artifacts like image interpolation. Finally, it should be noted that providing guarantees of the form  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$  requires a-priori knowledge that the task is equivariant to group  $\mathbb{G}$ . These limitations are however not relevant for many important domains in which equivariant models are the de-facto standard (e.g. graphs, point clouds and molecules).
In the following, we demonstrate how sound robustness guarantees for tasks with discrete and continuous domains and equivariances can be obtained via equivariance-preserving randomized smoothing.
Figure 4: Provable robustness of smoothed  $(\sigma = 1\mathrm{fin})$  DimeNet++ force predictions on MD17. The average provable bounds  $\delta$  on prediction changes are 2 to 13 times smaller than the average test errors (0.19 to  $0.74\mathrm{kcal / mol / A}$
Figure 3: Provable robustness of smoothed  $(\sigma = 0.2)$  PointNet and DGCNN point cloud classifiers on ModelNet40. Correspondence distance  $\epsilon$  is the Frobenius distance between point clouds after finding an optimal matching via permutation.
We additionally evaluate our graph edit distance certificates and how the cost of edit operations affects provable robustness. All experimental details are specified in Appendix B. Certificates are evaluated on a separate test set. Randomized smoothing methods are evaluated using sampling and hold with high probability. Each experiment is repeated 5 times. We visualize the standard deviation using shaded areas. When the shaded area is too small, we report its maximum value. An implementation will be made available at https://cs.cit.tum.de/daml/equivariance-robustness.
Point cloud classification. The first task we consider is point cloud classification, i.e.  $\mathbb{X} = \mathbb{R}^{N\times 3}$  and  $\mathbb{Y} = \{1,\dots ,K\}$ . Natural distances  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$  are the Frobenius distance and 0-1-loss. The task is permutation invariant, i.e. symmetric group  $\mathbb{G} = S_N$  acts on  $\mathbb{X}$  via permutation and on  $\mathbb{Y}$  via identity. Thus, we need to prove robustness w.r.t. to the correspondence distance  $\hat{d}_{\mathrm{in}}$ , i.e. the Frobenius distance after finding an optimal matching between rows via permutation. To preserve invariance and use Proposition 2 while randomly smoothing, we use Gaussian measures and majority voting, i.e. we predict the most likely label under Gaussian input perturbations. As invariant base models  $h$  we choose PointNet [114] and DGCNN [115], which are used in prior work on robust point cloud classification [30, 31]. We evaluate the robustness guarantees for smoothing standard deviation  $\sigma = 0.2$  on ModelNet40 [116]. This dataset consists of point cloud representations of CAD models from 40 different classes. Fig. 3 shows the certified accuracy, i.e. the percentage of predictions that are correct and provably robust. Both smoothed models have a high accuracy above  $80\%$ , but DGCNN has significantly higher provable robustness. In Appendix A.1 we repeat the experiment for different values of  $\sigma$  and compare our guarantees to those of 3DVerifier [92].
Molecular force prediction. Next, we consider a task with continuous equivariances: Predicting the forces acting on each atom in a molecule, which can for instance be used to simulate molecular dynamics. We have  $\mathbb{X} = \mathbb{R}^{N\times 3}$  (atomic numbers are treated as constant) and  $\mathbb{Y} = \mathbb{R}^{N\times 3}$ . Suitable distances  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$  are the Frobenius distance and the average  $\ell_2$  error  $\sum_{n = 1}^{N}||\mathbf{Y}_n - \mathbf{Y}_n^{\prime}||_2 / N$ . The task is permutation, rotation and translation equivariant, i.e.  $\mathbb{G} = S_N\times SE(3)$ . Thus,  $\hat{d}_{\mathrm{in}}$  is the point cloud registration distance. To preserve equivariance, we use Gaussian measures and center smoothing. We choose DimeNet++ [117] as equivariant base model  $h$ . We evaluate the provable robustness for smoothing standard deviation  $\sigma = 1$  fm on MD17 [118], a collection of 8 datasets, each consisting of a large number of configurations of a specific molecule. With just 1000 training samples, the models achieve low average test errors between 0.19 and  $0.74\mathrm{kcal / mol / A}$ . We use 1000 samples per test set to evaluate the robustness guarantees. Fig. 4 shows the average upper bounds  $\delta$  on the change of the predicted force vectors for small perturbations between 0 and  $2\mathrm{fm}$ . These average  $\delta$  are smaller than the average test errors by factors between 2 and 13. The standard deviation across seeds is below  $3\times 10^{-4}\mathrm{kcal / mol / A}$  for all  $\epsilon$ . In Appendix A.2 we show that the maximum  $\epsilon$  and  $\delta$  grow approximately linearly with  $\sigma$  and repeat our experiments with SchNet [119] and SphereNet [120] base models. Note that we are not (primarily) proving robustness to malicious perturbations, but robustness to measurement errors or errors in previous simulation steps.
Node classification. Finally, we consider a task with discrete domain and co-domain: Node classification, i.e.  $\mathbb{X} = \{0,1\}^{N\times D}\times \{0,1\}^{N\times N}$  and  $\mathbb{Y} = \{1,\dots ,K\} ^N$  . Distance  $d_{\mathrm{in}}$  is naturally defined via a weighted sum of inserted and deleted bits (see Eq. (2)). Output distance  $d_{\mathrm{out}}$  is the
Figure 5: Randomized smoothing guarantees for Figure 6: Convex adversarial polytope guaran-GCNs on Cora-ML. Increasing the cost of ad- tees for GCNs on Cora-ML. Increasing the cost versarial edge insertions increases the provable of attribute insertions increases the provable ro-ro Bustness for the same perturbation budgets  $\epsilon$ . Bustness for the same perturbation budgets  $\epsilon$ .