SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Provable Adversarial Robustness for Group Equivariant Tasks: Graphs, Point Clouds, Molecules, and More
Jan Schuchardt, Yan Scholten, Stephan Gunnemann
{j.schuchardt, y.scholten, s.guennemann}@tum.de Department of Computer Science & Munich Data Science Institute Technical University of Munich
A machine learning model is traditionally considered robust if its prediction remains (almost) constant under input perturbations with small norm. However, real-world tasks like molecular property prediction or point cloud segmentation have inherent equivariances, such as rotation or permutation equivariance. In such tasks, even perturbations with large norm do not necessarily change an input's semantic content. Furthermore, there are perturbations for which a model's prediction explicitly needs to change. For the first time, we propose a sound notion of adversarial robustness that accounts for task equivariance. We then demonstrate that provable robustness can be achieved by (1) choosing a model that matches the task's equivariances (2) certifying traditional adversarial robustness. Certification methods are, however, unavailable for many models, such as those with continuous equivariances. We close this gap by developing the framework of equivariance-preserving randomized smoothing, which enables architecture-agnostic certification. We additionally derive the first architecture-specific graph edit distance certificates, i.e. sound robustness guarantees for isomorphism equivariant tasks like node classification. Overall, a sound notion of robustness is an important prerequisite for future work at the intersection of robust and geometric machine learning.
Group equivariance and adversarial robustness are two important model properties when applying machine learning to real-world tasks involving images, graphs, point clouds and other data types:
Group equivariance is an ubiquitous form of task symmetry [1]. For instance, we do not know how to optimally classify point clouds, but know that the label is not affected by permutation. We cannot calculate molecular forces in closed form, but know that the force vectors rotate as the molecule rotates. Directly enforcing such equivariances in models is an effective inductive bias, as demonstrated by the success of convolutional layers [2], transformers [3] and graph neural networks [4].
Adversarial robustness [5-7] is a generalized notion of model Lipschitzness: A small change to a model's input  $x$  should only cause a small change to its prediction  $f(x)$ . If a model is adversarially robust, then its accuracy will not be negatively affected by sensor noise, measurement errors or other small perturbations that are unavoidable when working with real-world data.
For the first time, we address the following question: What is adversarial robustness in tasks that are group equivariant? This is necessary because the notions of input and output similarity used in prior work on adversarial robustness (e.g.  $\ell_p$  distances) are not suitable for group equivariant tasks.
Figure 1: The right graph is constructed by inserting and deleting four edges. While their  $\ell_0$  distance is large, the graphs are isomorphic and should thus have the same set of node labels.
Figure 2: The predicted trajectory (blue) should rotate as the image rotates (green), even in the presence of camera noise and other perturbations. It should not remain constant (red).
Fig. 1 illustrates why group equivariant tasks require rethinking input similarity. The right graph is constructed by modifying a large fraction of edges, meaning the perturbation of the adjacency matrix has a large  $\ell_0$  norm. Prior work [8-17] may deem it too large for a node classifiers' prediction to be robust. However, we know the graphs are isomorphic, meaning they are the same geometric object and should have the same set of labels. Fig. 2 illustrates why group equivariant tasks also require rethinking output similarity. Prior work considers a prediction robust if it remains (almost) constant. But when predicting trajectories from drone footage, we know that they should rotate as the drone rotates - even in the presence of camera noise. The predictions should explicitly not remain constant.
To address these two issues we propose a sound notion of adversarial robustness for group equivariant tasks that a) measures input similarity by lifting distance functions to group invariant distance functions and b) jointly considers transformations of the input and output space. This notion of adversarial robustness applies to arbitrary tasks, groups and distance functions.
A natural goal after introducing a novel notion of robustness is developing (provably) robust models. We show that robustness can be guaranteed by being robust under the traditional notion of adversarial robustness – if the model's equivariances match the equivariances of the task it is used for. Importantly, this implies that existing robustness guarantees may actually hold for significantly larger sets of perturbed inputs. For instance, proving the robustness of a graph neural network w.r.t.  $\ell_0$  distance is in fact proving robustness w.r.t graph edit distance with uniform cost for insertion and deletion.
Although equivariant models make provable robustness more attainable, there are no certification procedures for many architectures. For instance, there is no prior work on proving robustness for rotation equivariant models. To close this gap, we develop the framework of equivariance-preserving randomized smoothing. It specifies sufficient conditions under which models retain their equivariances when undergoing randomized smoothing – a state-of-the-art approach for simultaneously increasing and proving the robustness of arbitrary models [18-20]. In addition to that, we generalize the aforementioned graph edit guarantees to arbitrary, user specified costs. Varying these costs allows for a fine-grained analysis of a model's robustness to graph perturbations.
To summarize, our core contributions are that we
- propose a sound notion of adversarial robustness for group equivariant tasks,
show that using equivariant models facilitates achieving (provable) robustness,
- develop the framework of equivariance-preserving randomized smoothing,
- and generalize each existing graph and node classification robustness certificate for  $\ell_0$  perturbations to graph edit distance perturbations with user-specified costs.
Overall, reconsidering what adversarial robustness means in equivariant tasks is an important prerequisite for future work at the intersection of robust and geometric machine learning.
There is little prior work that studies equivariance and adversarial robustness jointly. The work that exists [21-31] only considers group invariant classification (a special case of equivariance) and does not consider how a task's equivariances should influence what we consider robust.
Model invariance and robustness. Prior work mostly focuses on a trade-off between invariance and robustness in image classification, i.e., whether increasing robustness to rotation or translation
decreases robustness to  $\ell_p$  perturbations and vice-versa [21-25]. Schuchardt and Gunnemann [26] used knowledge about the invariances of point cloud classifiers to prove that they are constant within larger regions than could be shown using previous approaches.
Group invariant distances. Recently, stability results for graph classifiers under isomorphism invariant optimal transport distances have been derived [27-29]. For point cloud classifiers, using the permutation invariant Chamfer or Hausdorff distance to craft attacks has been proposed [30, 31]. These works only focus on invariance and specific domains and do not consider that distances should be task-dependent: A rotation invariant distance for images may be desirable when segmenting cell nuclei, but not when classifying hand-written digits, since it would fail to distinguish 6 and 9.
String edit distance. In concurrent work, Huang et al. [32] use randomized smoothing to prove robustness of classifiers w.r.t. string edit distance, i.e., the number of substitutions that are needed to convert one string from alphabet  $\Sigma \cup \{\bot\}$  into another, up to insertion of alignment tokens  $\bot$ . Their work further emphasizes the need for invariant distance functions in domains with symmetries, and the usefulness of randomized smoothing for proving robustness w.r.t. such distances.
Robustness of models with equivariances. Aside from work that studies invariance and adversarial robustness jointly, there is a rich literature investigating the robustness of models that happen to have equivariances. This includes convolutions [5-7, 33], transformers[34-37], point cloud models [30, 31, 38-46] and graph neural networks [8-17, 47-55]. The models are however treated as a series of matrix multiplications and nonlinearities, without accounting for their equivariances or the equivariances of the tasks they are used for. Nevertheless, many methods can actually be reused for proving (non-)robustness under our proposed notion of adversarial robustness (see Section 5).
Transformation-specific robustness. A subfield of robust machine learning focuses on robustness to unnoticeable parametric transformations (e.g. small rotations) [42, 46, 56-65]. These works implicitly assume that large transformations lead to easily identifiable out-of-distribution samples. This is not the case with equivariant tasks: For instance, a molecule rotated by  $180^{\circ}$  is still the same geometric object. Furthermore, they do not consider unstructured perturbations. Nevertheless, transformation-specific robustness can be framed as a special case of our proposed notion (see Appendix J).
Semantics-aware robustness. Our work is closely related to different proposals to include ground truth labels in the definition of adversarial robustness [22, 52, 66-71]. A problem is that the ground truth is usually unknown, which limits experimental evaluation to simple data generating distributions [52, 70] or using human study participants [22, 67, 71]. Geisler et al. [72] overcome this problem in the context of neural combinatorial optimization by using adversarial perturbations that are known to change the ground truth of a decision problem. Group equivariant tasks admit a similar approach, since we know how the ground truth changes for specific input transformations.
Group theory and equivariance. Discussing group equivariance requires a few algebraic concepts. A group is a set  $\mathbb{G}$  with identity element  $e$  and associative operator  $\cdot : \mathbb{G} \times \mathbb{G} \to \mathbb{G}$  such that  $\forall g \in \mathbb{G}: e \cdot g = g \cdot e = g$  and each  $g \in \mathbb{G}$  has an inverse element  $g^{-1}$  with  $g^{-1} \cdot g = g \cdot g^{-1} = e$ . We are interested in transformations with a group structure, such as rotations. Given set  $\mathbb{X}$ , a (left) group action is a function  $\bullet_{\mathbb{X}}: \mathbb{G} \times \mathbb{X} \to \mathbb{X}$  that transforms elements of  $\mathbb{X}$  and preserves the structure of  $\mathbb{G}$ , i.e.  $g \bullet_{\mathbb{X}} h \bullet_{\mathbb{X}} x = (g \cdot h) \bullet_{\mathbb{X}} x$ . For instance, rotation by  $\phi^{\circ}$  and then by  $\psi^{\circ}$  is the same as rotation by  $(\psi + \phi)^{\circ}$ . A group may act differently on different sets. For example, rotation group  $SO(3)$  may act on point clouds via matrix multiplication while acting on class labels via the identity function. When clear from context, we drop the subscripts. A function  $f: \mathbb{X} \to \mathbb{Y}$  is equivariant if each action on its input is equivalent to an action on its output, i.e.  $\forall x \in \mathbb{X}, g \in \mathbb{G}: f(g \bullet_{\mathbb{X}} x) = g \bullet_{\mathbb{Y}} f(x)$ .
Adversarial robustness means that any small change to clean input  $x$  only causes a small change to prediction  $f(x)$ , i.e.  $\max_{x' \in \mathbb{B}_x} d_{\mathrm{out}}(f(x), f(x')) \leq \delta$  with  $\mathbb{B}_x = \{x' \mid d_{\mathrm{in}}(x, x') \leq \epsilon\}$ , and  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$  quantifying input and output distance. We refer to the set of admissible perturbed inputs  $\mathbb{B}_x$  as the perturbation model. A common special case is  $d_{\mathrm{out}}(y, y') = \mathbb{1}[y \neq y']$  and  $\delta = 0$  [5-7]. Other forms of robustness involve training data [73, 74], but we focus on test-time perturbations.
Randomized smoothing [18-20] is a paradigm for increasing and proving the robustness of models in an architecture-agnostic manner. It works by randomizing the inputs of a base model  $h$  to construct a more robust smoothed model  $f$ . While originally proposed for provably robust image classification,
it has evolved into a much more general framework that can be applied to various domains and tasks [14, 20, 51, 75-81]. Consider a measurable input space  $(\mathbb{X},\mathbb{D})$  and a measurable base model  $h:\mathbb{X}\to \mathbb{V}$  that maps to a measurable intermediate space  $(\mathbb{V},\mathbb{F})$  (e.g. logits). Given the base model  $h$  and a family of probability measures  $(\mu_{x})_{x\in \mathbb{X}}$  on  $(\mathbb{X},\mathbb{D})$  indexed by  $\mathbb{X}$  (e.g. Gaussians with mean  $x$ ), we can define the input-dependent pushforward measure  $\mu_x\circ h^{-1}$  for any input  $x\in \mathbb{X}$ .<sup>1</sup> We can further define a smoothing scheme  $\xi :\Delta (\mathbb{V},\mathbb{F})\to \mathbb{Y}$  that maps from probability measures  $\Delta (\mathbb{V},\mathbb{F})$  on the intermediate space to an output space  $\mathbb{Y}$  (e.g. logit distributions to labels).<sup>2</sup> This lets us construct the smoothed model  $f(x) = \xi (\mu_x\circ h^{-1})$ , which makes a prediction for input  $x\in \mathbb{X}$  based on some quantity of the input-dependent pushforward measure  $\mu_x\circ h^{-1}$  (e.g. the expected value). Intuitively, if two inputs  $x$  and  $x^{\prime}$  are sufficiently similar, then the smoothing measures  $\mu_{x}$  and  $\mu_x^\prime$  will have a large overlap, thus leading to similar smoothed predictions. The combination of measures  $(\mu_x)_{x\in \mathbb{X}}$  and smoothing scheme  $\xi$  determines for which  $d_{\mathrm{in}}$  and  $d_{\mathrm{out}}$  robustness is guaranteed. We discuss these combinations in more detail in Appendix D.
4 Redefining robustness for group equivariant tasks
For the first time, we seek to define adversarial robustness for group equivariant tasks. By task we mean an unknown function  $y:\mathbb{X}\to \mathbb{Y}$  that is equivariant with respect to the action of a group  $\mathbb{G}$ . We assume that it is approximated by a (learned) model  $f:\mathbb{X}\rightarrow \mathbb{Y}$  whose robustness we want to determine. The model does not have to be equivariant. Like traditional adversarial robustness, we assume there are functions  $d_{\mathrm{in}}:\mathbb{X}\times \mathbb{X}\to \mathbb{R}_{+}$  and  $d_{\mathrm{out}}:\mathbb{Y}\times \mathbb{Y}\to \mathbb{R}_{+}$ , which define what constitutes a small change in domain  $\mathbb{X}$  and co-domain  $\mathbb{Y}$ , if we are not concerned with group symmetries. For instance,  $\ell_2$  distance is a natural notion of similarity between Euclidean coordinates.
For ease of exposition, we further assume that all considered optimization domains are compact (so that minima and maxima exist) and that group  $\mathbb{G}$  acts isometrically on  $\mathbb{X}$ , i.e.  $\forall x, x' \in \mathbb{X}, \forall g \in \mathbb{G}: d_{\mathrm{in}}(g \bullet x, g \bullet x') = d_{\mathrm{in}}(x, x')$ . This covers most practically relevant use cases. For completeness, we discuss non-compact sets and non-isometric actions in Appendices H and I.
4.1 Perturbation model for group equivariant tasks
Our first goal is to resolve the shortcomings of using typical input distance functions  $d_{\mathrm{in}}$  to define what constitutes small input perturbations in group equivariant tasks. We seek some improved function  $\hat{d}_{\mathrm{in}}$  that accounts for the equivariance of  $y$  and simultaneously captures the similarity of objects in domain  $\mathbb{X}$ , as defined by original distance function  $d_{\mathrm{in}}$ . To this end, we define three desiderata.
Desideratum 1. We know that any perturbed  $x' \in \mathbb{X}$  and  $g \bullet x'$  with  $g \in \mathbb{G}$  are up to symmetry the same geometric object with the same semantic content, i.e.  $y(g \bullet x') = g \bullet y(x')$ . But as illustrated in Fig. 1, group actions may cause a drastic change w.r.t. distance  $d_{\mathrm{in}}$ . Thus, even if  $f(x)$  is "robust" within  $\mathbb{B}_x = \{x' \mid d_{\mathrm{in}}(x, x') \leq \epsilon\}$  for some  $\epsilon$ , there may still be a  $x' \in \mathbb{B}_x$  and  $g \bullet x' \notin \mathbb{B}_x$  that lead to two completely different predictions. If a prediction can be altered without changing the semantic content of an input, it can hardly be considered robust. This problem cannot be resolved by requiring robustness for very large  $\epsilon$  so that  $\mathbb{B}_x$  covers all symmetric objects. Doing so would also include objects with substantially different semantic content from clean input  $x$ , which should actually lead to different predictions. Instead, we need a  $\hat{d}_{\mathrm{in}}$  that is constant for all symmetric forms of  $x'$ , so that robustness to one implies robustness to all:  $\forall x, x' \in \mathbb{X}, g \in \mathbb{G}: \hat{d}_{\mathrm{in}}(x, g \bullet x') = \hat{d}_{\mathrm{in}}(x, x')$ .
Desideratum 2. While the first desideratum accounts for equivariance, a model should also be robust to sensor noise, measurement errors and other small perturbations that are not necessarily group actions. Therefore, elements that are close with respect to the original distance function  $d_{\mathrm{in}}$  should remain close under our new distance function  $\hat{d}_{\mathrm{in}}$ , i.e.  $\forall x, x': \hat{d}_{\mathrm{in}}(x, x') \leq d_{\mathrm{in}}(x, x')$ .
Desideratum 3. The first two desiderata could be fulfilled by  $\hat{d}_{\mathrm{in}}(x,x^{\prime}) = 0$  or a function that arbitrarily changes the ordering of inputs w.r.t. distance. To appropriately capture similarity,  $\hat{d}_{\mathrm{in}}$  should not only underapproximate  $d_{\mathrm{in}}$ , but preserve it as best as possible. Let  $\mathbb{D}$  be the set of functions from  $\mathbb{X}\times \mathbb{X}$  to  $\mathbb{R}_+$  that fulfill the first two desiderata. We require that  $\hat{d}_{\mathrm{in}}(x,x^{\prime}) = \max_{\gamma \in \mathbb{D}}\gamma (x,x^{\prime})$
<sup>1</sup>Note that  $h^{-1}$  is not the inverse but the pre-image. The base model need not be invertible.
$^{2}\mathbb{V}$  and  $\mathbb{Y}$  can also be the same set, as is the case with majority voting for classifiers.
Proposition 1. A function  $\hat{d}_{\mathrm{in}}: \mathbb{X} \times \mathbb{X} \to \mathbb{R}_+$  that fulfills all three desiderata for any original distance function  $d_{\mathrm{in}}: \mathbb{X} \times \mathbb{X} \to \mathbb{R}_+$  exists and is uniquely defined:  $\hat{d}_{\mathrm{in}}(x, x') = \min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, g \bullet x')$ .
We prove this result in Appendix C. We refer to  $\hat{d}_{\mathrm{in}}$  as the action-induced distance. It is the distance after optimally aligning the perturbed input  $x^{\prime}$  with the clean input  $x$  via a group action.
Example: Point cloud registration distance. Let  $\mathbb{X} = \mathbb{R}^{N\times D}$ . Consider  $\mathbb{G} = S_N\times SE(D)$ , the set of permutation matrices, rotation matrices and translation vectors. Let  $\mathbb{G}$  act via  $(\pmb {P},\pmb {R},t)\bullet \pmb {X} = \pmb {P}\left(\pmb {X}\pmb {R}^T +\mathbf{1}_N\pmb {t}^T\right)$  and let  $d_{\mathrm{in}}$  be the Frobenius distance. Then  $\hat{d}_{\mathrm{in}}$  is the registration distance [82], i.e. the distance after finding an optimal correspondence and applying an optimal rigid transformation.
Example: Graph edit distance. Let  $\mathbb{X}$  be the set of all adjacency matrices  $\{0,1\}^{N\times N}$  with  $d_{\mathrm{in}}(A,A^{\prime}) = ||A - A^{\prime}||_{0}$ . Consider  $\mathbb{G} = S_N$ , the set of permutation matrices. Let  $\mathbb{G}$  act via  $\pmb{P}\bullet \pmb{A} = \pmb{P}\pmb{A}\pmb{P}^{T}$ . Then  $d_{\mathrm{in}}(A,A^{\prime})$  is the graph edit distance with uniform cost [83-85]. That is, the number edges that have to be inserted or deleted to transform  $\pmb{A}$  into a graph isomorphic to  $\pmb{A}^{\prime}$ .
As demonstrated by these examples, we do not claim to have invented this notion of distance for equivariant domains. Our contribution is justifying, in a principled manner, why this notion of distance should be used to define adversarial robustness for group equivariant tasks.
Perturbation model. Now that we have an appropriate input distance for group equivariant tasks, we can use it to define the set of admissible perturbed inputs  $\mathbb{B}_x$  as  $\{x' \mid \min_{g \in \mathbb{G}} d_{\mathrm{in}}(x, g \bullet x') \leq \epsilon\}$  with some small  $\epsilon \in \mathbb{R}_+$ . Because every  $g \in \mathbb{G}$  has an inverse element  $g^{-1}$ , this set is identical to  $\{g \bullet x' \mid g \in \mathbb{G}, d_{\mathrm{in}}(x, x') \leq \epsilon\}$ . We use this equivalent representation because it lets us disentangle group actions from other perturbations. That is,  $f(x)$  should be robust to inputs that can be generated via a small perturbation w.r.t. the original distance  $d_{\mathrm{in}}$  followed by a group action.
4.2 Output distance for group equivariant tasks
Fig. 2 illustrates that we also need to reconsider what a small change to a model's prediction is. Letting a group element act on a prediction may cause a large change w.r.t.  $d_{\mathrm{out}}$ . In particular, we may have  $d_{\mathrm{out}}(y(x), y(g \bullet x)) = d_{\mathrm{out}}(y(x), g \bullet y(x)) \gg 0$ , even though  $x$  and  $g \bullet x$  have a distance of zero w.r.t. the action-induced distance. Thus, we would consider even the ground truth  $y$  itself to be non-robust. Using action-induced distances, i.e.  $\min_{g \in \mathbb{G}} d_{\mathrm{out}}(y, g \bullet y')$ , is not appropriate either. Action-induced distances cannot distinguish between a model that transforms its predictions in compliance with the ground truth and one that applies arbitrary group actions.
To appropriately measure output distance, we need to account for differences between clean prediction  $f(x)$  and perturbed prediction  $f(g \bullet x')$  that are caused by the specific group element  $g \in \mathbb{G}$  acting on the model's input. To do so, we need to first revert the effect of the group action before comparing the predictions. That is, we need to measure output distance using  $d_{\mathrm{out}}(f(x), g^{-1} \bullet f(g \bullet x'))$ .
4.3 Proposed definition of robustness for group equivariant tasks
Combining the perturbation model from Section 4.1 with the output distance from Section 4.2 leads to the following definition of adversarial robustness for group equivariant tasks:
Definition 1. Assume that ground truth function  $y: \mathbb{X} \to \mathbb{Y}$  is equivariant with respect to the action of group  $\mathbb{G}$ . Then, a prediction  $f(x)$  for clean input  $x \in \mathbb{X}$  is  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robust if
$$
\left(\max  _ {x ^ {\prime} \in \mathbb {X}} \max  _ {g \in \mathbb {G}} d _ {\text {o u t}} (f (x), g ^ {- 1} \bullet f (g \bullet x ^ {\prime})) \text {s . t .} d _ {\text {i n}} (x, x ^ {\prime}) \leq \epsilon\right) \leq \delta . \tag {1}
$$
Simply speaking, prediction  $f(x)$  should be considered robust if it is robust to unnoticeable perturbations and is approximately equivariant around  $x$ . For perturbations of size  $\epsilon$  w.r.t. the original  $d_{\mathrm{in}}$ , the prediction should not change by more than  $\delta$  and no group action should cause this error to increase beyond  $\delta$ . Note that Definition 1 depends on the equivariances of the task  $y$ , not those of the model  $f$ . Further note that the constraint involves original distance  $d_{\mathrm{in}}$ , not action-induced distance  $\hat{d}_{\mathrm{in}}$ .