SlowGuess/NeurIPS_2023_OCR huggingface dataset:
That is, setting  $A_{n,m}$  from 0 to 1 has a large value if  $q_m^{(n)}$  is a negative number. Setting  $A_{n,m}$  from 1 to 0 has a large value if  $q^{(n)}$  if a large positive number. Algorithm 1 yields a dictionary  $\alpha_n$ , where  $\alpha_n(i,j)$  indicates how much the value of  $f_{n,k}(\mathbf{X},\mathbf{A})$  changes when optimally inserting  $i$  and deleting  $j$  edges. It further yields a dictionary  $\beta_n$ , where  $\beta_n(i,j)$  contains the optimal set of edges to perturb in row  $n$  of the adjacency matrix. The last thing we need to do is to add them to the unperturbed values  $f_{n,k}(\mathbf{X},\mathbf{A})$  while accounting for the fact that perturbing edges also influences the degree and thus the degree normalization. We propose to do so via Algorithm 4
Algorithm 4 Computation of  $-1 \cdot \min_{\mathbf{X}', \mathbf{A}' \in \mathbb{B}} f_{n,k}$  from the precomputed worst-case changes without degree normalization stored in dictionary  $\alpha_n$ .
max Adds  $\leftarrow \lfloor \rho_{n} / c^{+}\right]$    
for  $i = 0,\dots ,$  max Adds do   
max_DELs  $\leftarrow \lfloor (\rho_n - c^+\cdot i) / c^- \rfloor$    
for  $j = 0,\ldots ,$  max_DELs do olddegree  $\leftarrow \mathbf{1}^T\tilde{A}_n$  new-degree  $\leftarrow$  old-degree  $+i - j$    
new-degree  $\leftarrow$  max(1,min(new-degree,N))  $\triangleright$  Between 1 and  $N$  , due to self-loop.  $\alpha_{n}(i,j)\gets \alpha_{n}(i,j) -$  oldDegree  $\cdot f_{n,k}(\pmb {X},\pmb {A})$ $\alpha_{n}(i,j)\gets \alpha_{n}(i,j)$  / new-degree   
return  $\alpha_{n}$
Note that we compute the negative value of the lower bound, because all discussed algorithms are designed for maximization problems.
Combining the nodewise guarantees. Now we have, for each node  $n$  and every possible number of insertions  $i$  and deletions  $j$ , the negative of a lower bound on  $f_{n,k}$ , which is stored in  $\alpha_{n}(i,j)$ . The last step is to combine these nodewise guarantees while complying with global budget  $\epsilon$ . To this end, we can just reuse Algorithm 2 with  $c^{+} = c_{\mathbf{A}}^{+}$  and  $c^{-} = c_{\mathbf{A}}^{-}$ . It yields a sequence  $t$  of length  $N$ , whose  $n$ th element is the set of the worst indices to perturb in row  $n$  of the adjacency matrix. It further yields a lower bound on the negative classification margin  $-1 \cdot (F(\mathbf{X}',\mathbf{A}')_y - F(\mathbf{X}',\mathbf{A}')_k)$  for a specific  $k$ . If after performing the above procedure for all  $k \neq y$  all classification margins are positive, then the prediction is provably robust.
Enforcing symmetry via dualization. An important aspect of the discussion in [15] is that one may want to introduce the additional constraint  $A' = A'^T$  when proving robustness for symmetric graphs.
The authors show that
$$
\begin{array}{l} \min  _ {(m X, \boldsymbol {A} ^ {\prime}) \in \mathbb {B}} \sum_ {n = 1} ^ {N} \underline {{f _ {n , k}}} (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) \quad \text {s . t .} \boldsymbol {A} ^ {\prime} = \boldsymbol {A} ^ {\prime T} \\ \geq \max  _ {\boldsymbol {\Lambda} \in \mathbb {R} ^ {N \times N}} \min  _ {(m X, \boldsymbol {A} ^ {\prime}) \in \mathbb {B}} \sum_ {n = 1} ^ {N} \underline {{f _ {n , k}}} (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) + \operatorname {T r} ((\boldsymbol {\Lambda} ^ {T} - \boldsymbol {\Lambda}) \boldsymbol {A} ^ {\prime}), \\ \end{array}
$$
with dual variable  $\pmb{\Lambda} \in \mathbb{R}^{N \times N}$ . Note that  $\mathrm{Tr}((\pmb{\Lambda}^T - \pmb{\Lambda})\pmb{A}) = \sum_{n=1}^{N} \sum_{m=1}^{M} (\pmb{\Lambda}^T - \pmb{\Lambda})_{n,m}^T \cdot A_{n,m}'$ .
The inner optimization problem can be solved exactly via the "nodewise guarantees" and "combining the nodewise guarantees" steps above, after replacing each parameter vector  $\pmb{q}^{(n)}$  of the linearized models  $\underline{f}_{n,k}$  defined in Eq. (14) with  $\pmb{q}^{(n)} + \left(\mathbf{1}^T\tilde{\pmb{A}}_n'\right)\cdot (\pmb {\Lambda} - \pmb {\Lambda}^T)_n$ .
Because the inner optimization problem is solved exactly, dual variable  $\Lambda$  can be optimized via gradient ascent on  $\mathrm{Tr}((\mathbf{\Lambda}^T - \mathbf{\Lambda})\mathbf{A}')$  (Danskin's theorem).
F.6 Policy iteration for adjacency perturbations
Bojchevski and Gunnemann [11] derive robustness certificates for models where the predictions are a linear function of the (personalized) PageRank. Specifically, they consider the following architecture called  $\pi$ -PPNP:
$$
\boldsymbol {Y} = \operatorname {s o f t m a x} (\boldsymbol {\Pi} \boldsymbol {H}), \quad \boldsymbol {H} _ {v,:} = f _ {\theta} (\boldsymbol {X} _ {v,:}), \quad \boldsymbol {\Pi} = (1 - \alpha) (\boldsymbol {I} _ {N} - \alpha \boldsymbol {D} ^ {- 1} \boldsymbol {A}) ^ {- 1}
$$
where  $\mathbf{X}$  the feature matrix of the graph,  $f$  is a neural network with parameter  $\theta$ ,  $H_{v,:}$  the prediction for node  $v$  and  $\Pi$  the personalized PageRank matrix with teleport probability  $\alpha$  (i.e.  $\Pi_{v,:}$  is the personalized PageRank of node  $v$ ). Further note that  $I_N$  is the identity matrix and  $D$  is the degree matrix  $D_{ii} = \sum_{j} = A_{ij}$  of a graph  $G = (\mathcal{V},\mathcal{E})$  with nodes  $\mathcal{V}$  and edges  $\mathcal{E}$ .
Their certificates are against adjacency perturbations with uniform costs  $(c_{A}^{+} = c_{A}^{-} = 1)$  and we will generalize their certificate to arbitrary costs for edge insertion and deletion. Let the following set denote all admissible perturbed graphs under costs  $c_{A}^{+}$  for edge insertion and  $c_{A}^{-}$  for edge deletion:
$$
\begin{array}{l} \mathcal {Q} _ {\mathcal {F}} = \left\{\left(\mathcal {V}, \tilde {\mathcal {E}} := \mathcal {E} _ {f} \cup \mathcal {F} _ {+}\right) \mid \mathcal {F} _ {+} \in \mathcal {P} (\mathcal {F}), \right. \\ c _ {A} ^ {+} \cdot | \tilde {\mathcal {E}} \backslash \mathcal {E} | + c _ {A} ^ {-} \cdot | \mathcal {E} \backslash \tilde {\mathcal {E}} | \leq \epsilon , \tag {15} \\ c _ {A} ^ {+} \cdot | \tilde {\mathcal {E}} ^ {v} \setminus \mathcal {E} ^ {v} | + c _ {A} ^ {-} \cdot | \mathcal {E} ^ {v} \setminus \tilde {\mathcal {E}} ^ {v} | \leq \rho_ {v}, \forall v \} \\ \end{array}
$$
where  $\mathcal{E}_f$  is a set of fixed edges that cannot be modified,  $\mathcal{F} \subseteq \mathcal{V} \times \mathcal{V}$  the set of fragile edges that can be modified. Here,  $\mathcal{F}_+ \subseteq \mathcal{F}$  denotes the set of edges that are included in the perturbed graph (analogously  $\mathcal{F}_- \subseteq \mathcal{F}$  the set of edges that are not included anymore in the perturbed graph). Here,  $\epsilon$  denotes the global budget and  $\rho_v$  the local budget for node  $v$ .
We assume a fixed graph  $G$ , set of fragile edges  $\mathcal{F}$ , global budget  $B$  and local budgets  $b_{v}$ . Note that since  $\pi$ -PPNP separates prediction from propagation we can further assume fixed model logits  $\pmb{H}$ . Following Bojchevski and Gunnemann [11] we further define the worst-case margin between classes  $y_{t}$  and  $c$  under any perturbed graph  $\tilde{G} \in \mathcal{Q}_{\mathcal{F}}$  (Problem 1 of Bojchevski and Gunnemann [11]):
$$
m _ {y _ {t}, c} ^ {*} (t) = \min  _ {\tilde {G} \in \mathcal {Q} _ {\mathcal {F}}} m _ {y _ {t}, c} (t) = \min  _ {\tilde {G} \in \mathcal {Q} _ {\mathcal {F}}} \boldsymbol {\pi} _ {\tilde {G}} \left(\boldsymbol {e} _ {t}\right) ^ {T} \left(\boldsymbol {H} _ {:, y _ {t}} - \boldsymbol {H} _ {:, c}\right) \tag {16}
$$
where  $e$  is the unit vector and  $\pi_{\tilde{G}}(e_t)$  the personalized PageRank vector of target node  $t$  under perturbed graph  $\tilde{G}$ . We want to show that a specific target node  $t$  is certifiably robust w.r.t. the logits  $H$  and the set  $Q_{\mathcal{F}}$ . This is the case if  $m_{y_t,*}^*(t) = \min_{c \neq y_t} m_{y_t,c}^*(t) > 0$ . In the following we will derive guarantees for this under local and global adversarial budgets, respectively.
Local constraints only. Bojchevski and Gunnemann [11] phrase the problem of Equation 16 as a more general average cost infinite horizon Markov decision process and present a policy iteration algorithm that solves it in polynomial time. Following their derivations we also define  $\boldsymbol{r} = -(\boldsymbol{H}_{:,y_t} - \boldsymbol{H}_{:,c})$ , where  $\boldsymbol{r}_v$  denotes the rewards a random walker gets for visiting node  $v$  [11]. The following adapted policy iteration computes the worst-case graph  $\tilde{G} \in \mathcal{Q}_{\mathcal{F}}$  under arbitrary costs:
Algorithm 5 Policy Iteration with local budgets under arbitrary costs
Require: Graph  $G = (\mathcal{V},\mathcal{E})$  ,reward  $\pmb{r}$  ,set of fixed edges  $\mathcal{E}_f$  fragile edges  $\mathcal{F}$  ,local budgets  $b_{v}$  and costs  $c_{A}^{+}$  , for edge insertion and  $c_{A}^{-}$  for edge deletion   
1: Initialization: arbitrary  $\mathcal{W}_0\subseteq \mathcal{F},\mathcal{A}^G$  corresponding to  $G$    
2: while  $\mathcal{W}_k\neq \mathcal{W}_{k - 1}$  do   
3: Solve  $(I_N - \alpha D^{-1}A)x = r$  for  $\pmb{x}$  where  $A_{ij} = 1 - A_{ij}^{G}$  if  $(i,j)\in \mathcal{W}_k$    
4:  $l_{ij}\gets (1 - 2A_{ij}^{G})(x_{j} - \frac{x_{i} - r_{i}}{\alpha})$  for all  $(i,j)\in \mathcal{F}$    
5: for  $v\in \mathcal{V}$  do   
6:  $m\gets \arg \max_{m}\sum_{(v,j)\in F^{+}\cup F^{-}}m_{j}l_{v,j}$  s.t.  $\begin{array}{r}\sum_{(v,j)\in F^{-}}c_{A}^{-}m_{j} + \sum_{(v,j)\in F^{+}}c_{A}^{+}m_{j}\leq \end{array}$ $\rho_v,m_j\in \{0,1\}$    
7:  $\mathcal{L}_v\gets \{(v,j)\in \mathcal{F}\mid m_j = 1\}$    
8:  $\mathcal{W}_k\gets \bigcup_v\mathcal{L}_v$    
9:  $k\gets k + 1$  return Wk
Line 6 of the policy iteration requires us to find those fragile edges with the largest score  $l$  under the local budget  $\rho_v$ . This is an instance of the Knapsack problem, which we discussed earlier. In practice we solve this using dynamic programming, specifically we call Algorithm 2 for row-vector  $V_{1,j} \gets l_{v,j}$ ,  $\epsilon \gets \infty$  and  $\rho_1 \gets \rho_v$ . Finally, note that Bojchevski and Gunnemann [11] show the correctness of the policy iteration for arbitrary sets of admissible perturbed graphs  $\mathcal{Q}_{\mathcal{F}}$ , thus the correctness of Algorithm 5 follows from their proof. In particular, the additional costs for insertion and deletion does not change the fact that we can model the problem as a Markov decision process.
Local and global constraints. Lastly, for local and global constraints we can use the same auxiliary graph as introduced by Bojchevski and Gunnemann [11]. In particular, the additional constraints are only additional enrichments of the Linear Program resulting from the auxiliary graph, yielding a quadratically constrained linear program. To account for additional costs, we have to replace constraint (4f) of their QCLP with the following constraints that also considers additional costs:
$$
\sum_ {(i, j) \in \mathcal {F}} c _ {\boldsymbol {A}} ^ {-} \cdot [ (i, j) \in \mathcal {E} ] \beta_ {i, j} ^ {0} + c _ {\boldsymbol {A}} ^ {+} \cdot [ (i, j) \notin \mathcal {E} ] \beta_ {i, j} ^ {1} \leq \epsilon
$$
F.7 Sparsity-aware randomized smoothing for attribute and adjacency perturbations
Bojchevski et al. [14] present robustness certificates for sparse data based on randomized smoothing. Their main idea is to introduce a sparsity-aware smoothing distribution that preserves the sparsity of the underlying data distribution. Such sparsity-aware certificates for discrete data are currently state-of-the-art for certifying robustness of GNNs against structure and attribute perturbations.
Bojchevski et al. [14] derive robustness guarantees only under uniform costs ( $c_{\mathbf{A}}^{+} = c_{\mathbf{A}}^{-} = c_{\mathbf{X}}^{+} = c_{\mathbf{X}}^{-} = 1$ ). Here we make use of our findings (Appendix E) and generalize their certificate to graph edit distances under arbitrary costs for insertion and deletion. Specifically for sparse graphs we model adversaries that perturb nodes by adding ones (flip  $0 \to 1$ ) to or deleting ones (flip  $1 \to 0$ ) of the adjacency or feature matrix. Since adversaries can perturb both the features  $\mathbf{X}$  and edges  $\mathbf{A}$  of a graph  $G = (\mathbf{A}, \mathbf{X})$  we consider the threat model  $\mathcal{B}_{\epsilon}(G) = \{\tilde{G} \mid \delta(G, \tilde{G}) \leq \epsilon\}$  with
$$
\begin{array}{l} \delta (G, \tilde {G}) = c _ {\mathbf {A}} ^ {+} \sum_ {i} \sum_ {j} \mathbb {I} (\tilde {\mathbf {A}} _ {i j} = \mathbf {A} _ {i j} + 1) + c _ {\mathbf {A}} ^ {-} \sum_ {i} \sum_ {j} \mathbb {I} (\tilde {\mathbf {A}} _ {i j} = \mathbf {A} _ {i j} - 1) \\ + c _ {\boldsymbol {X}} ^ {+} \sum_ {i} \sum_ {j} \mathbb {I} (\tilde {\boldsymbol {X}} _ {i j} = \boldsymbol {X} _ {i j} + 1) + c _ {\boldsymbol {X}} ^ {-} \sum_ {i} \sum_ {j} \mathbb {I} (\tilde {\boldsymbol {X}} _ {i j} = \boldsymbol {X} _ {i j} - 1) \\ \end{array}
$$
where  $\mathbb{I}$  is an indicator function and  $c_{\pmb{A}}^{+}, c_{\pmb{A}}^{-}, c_{\pmb{X}}^{+}, c_{\pmb{X}}^{-}$  the corresponding costs for addition and deletion.
Sparsity-aware smoothing distribution. Bojchevski et al. [14] propose a family of smoothing distributions that preserves sparsity of the underlying data. Applied to graphs, the distribution of randomly perturbed graphs  $(\tilde{X},\tilde{A})$  given clean graph  $(X,A)$  is defined by the probability mass
function  $Q:\{0,1\}^{N\times D}\times \{0,1\}^{N\times N}\to [0,1]$  with
$$
Q (\tilde {\boldsymbol {X}}, \tilde {\boldsymbol {A}}) = \prod_ {i, j} q _ {\boldsymbol {X}} \left(\tilde {\boldsymbol {X}} _ {i, j}\right) \prod_ {i, j} q _ {\boldsymbol {A}} \left(\tilde {\boldsymbol {A}} _ {i, j}\right) \tag {18}
$$
and elementwise functions  $q_{\mathbf{X}}, q_{\mathbf{A}}: \{0,1\} \to [0,1]$  with
$$
q _ {\boldsymbol {X}} (\tilde {\boldsymbol {X}} _ {i, j}) = \left\{ \begin{array}{l l} (p _ {\boldsymbol {X}} ^ {+}) ^ {1 - \boldsymbol {X} _ {i j}} (p _ {\boldsymbol {X}} ^ {-}) ^ {\boldsymbol {X} _ {i j}} & \text {i f} \tilde {\boldsymbol {X}} _ {i, j} \neq \boldsymbol {X} _ {i, j} \\ 1 - (p _ {\boldsymbol {X}} ^ {+}) ^ {1 - \boldsymbol {X} _ {i j}} (p _ {\boldsymbol {X}} ^ {-}) ^ {\boldsymbol {X} _ {i j}} & \text {o t h e r w i s e} \end{array} \right.
$$
$$
q _ {\boldsymbol {A}} (\tilde {\boldsymbol {A}} _ {i, j}) = \left\{ \begin{array}{l l} (p _ {\boldsymbol {A}} ^ {+}) ^ {1 - \boldsymbol {A} _ {i j}} (p _ {\boldsymbol {A}} ^ {-}) ^ {\boldsymbol {A} _ {i j}} & \text {i f} \tilde {\boldsymbol {A}} _ {i, j} \neq \boldsymbol {A} _ {i, j} \\ 1 - (p _ {\boldsymbol {A}} ^ {+}) ^ {1 - \boldsymbol {A} _ {i j}} (p _ {\boldsymbol {A}} ^ {-}) ^ {\boldsymbol {A} _ {i j}} & \text {o t h e r w i s e} \end{array} \right.
$$
where parameters  $p_{\mathbf{X}}^{+}, p_{\mathbf{X}}^{-}, p_{\mathbf{A}}^{+}, p_{\mathbf{A}}^{-} \in [0,1]$  specify the probability of randomly inserting or deleting attributes and edges, respectively. Note that by using different probabilities to flip  $0 \rightarrow 1$  with probability  $p^{+}$  and  $1 \rightarrow 0$  with probability  $p^{-}$ , the smoothing distribution allows to preserve sparsity of the data especially for  $p^{+} \ll p^{-}$ .
Graph edit distance certificates. The certificate of Bojchevski et al. [14] guarantees robustness against the threat model  $\mathcal{B}_{r_A^+,r_A^-,r_X^+,r_X^-}(G)$ , which bounds the perturbations individually by having separate radii for addition  $r^+$  and deletion  $r^-$ . To certify robustness under graph edit distance with arbitrary costs and given budget  $\epsilon$ , we have to certify robustness with respect to all balls  $\mathcal{B}_{r_A^+,r_A^-,r_X^+,r_X^-}(G)$  with
$$
c _ {\pmb {A}} ^ {+} \cdot r _ {\pmb {A}} ^ {+} + c _ {\pmb {A}} ^ {-} \cdot r _ {\pmb {A}} ^ {-} + c _ {\pmb {X}} ^ {+} \cdot r _ {\pmb {X}} ^ {+} + c _ {\pmb {X}} ^ {-} \cdot r _ {\pmb {X}} ^ {-} \leq \epsilon .
$$
Finally note that in practice we do not have to consider all balls since if we can certify one radius the classifier is also robust for smaller radii [14]. Therefore the number of combinations one has to consider reduces significantly in practice.
G Local budgets and local robustness
As discussed in Section 4, the domain  $\mathbb{X}$  may be composed of  $N$  distinct elements, i.e.  $\mathbb{X} = \mathbb{A}^N$  for some set  $\mathbb{A}$ . Similarly, the task may require making  $M$  distinct predictions, i.e. the co-domain is  $\mathbb{Y} = \mathbb{B}^M$  for some set  $\mathbb{B}$ . In certain tasks, like node classification, it is common to enforce local distance constraints on each of the  $N$  elements of a perturbed input  $x'$  and investigate the robustness of some subset of prediction elements  $\mathbb{T} \subseteq \{1, \dots, M\}$ . In the following, we discuss how to generalize our definition of robustness (see Definition 1) to enforce such local budget constraints and quantify such local robustness.
For this discussion, recall that a model is  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robust if
$$
\left(\max  _ {x ^ {\prime} \in \mathbb {X}} \max  _ {g \in \mathbb {G}} d _ {\mathrm {o u t}} (f (x), g ^ {- 1} \bullet f (g \bullet x ^ {\prime})) \text {s . t .} d _ {\mathrm {i n}} (x, x ^ {\prime}) \leq \epsilon\right) \leq \delta .
$$
Local budgets. Local budget constraints can be easily accounted for by replacing the original optimization domain  $\{g\bullet x^{\prime}\mid g\in \mathbb{G},x^{\prime}\in \mathbb{X},d_{\mathrm{in}}(x,x^{\prime})\leq \epsilon \}$  in the above definition with
$$
\left\{g \bullet x ^ {\prime} \mid g \in \mathbb {G}, x ^ {\prime} \in \mathbb {A} ^ {N}, d _ {\text {i n}} (x, x ^ {\prime}) \leq \epsilon , \forall n: d _ {\text {l o c}} \left(x _ {n}, x _ {n} ^ {\prime}\right) \leq \rho_ {n} \right\},
$$
with global distance  $d_{\mathrm{in}}: \mathbb{A}^N \to \mathbb{R}_+$ , global budget  $\epsilon$ , local distance  $d_{\mathrm{loc}}: \mathbb{A} \to \mathbb{R}_+$ , and local budgets  $\rho_1, \ldots, \rho_N \in \mathbb{R}_+$ .
Local robustness. Quantifying the robustness of some subset of prediction indices  $\mathbb{T} \subseteq \{1, \dots, M\}$  requires a function  $d_{\mathrm{out}}: \mathbb{B}^{|T|} \times \mathbb{B}^{|T|} \to \mathbb{R}_+$ . We need to be careful about where we introduce the indexing to pass the original  $M$ -dimensional predictions into  $d_{\mathrm{out}}$ . It is not correct to measure output distance using  $d_{\mathrm{out}}(f(x)_{\mathbb{T}}, g^{-1} \bullet f(g \bullet x')_{\mathbb{T}})$ , i.e. it is not correct to index before reverting the effect of group action  $g \in \mathbb{G}$ . Afterall, group  $\mathbb{G}$  may act differently on  $\mathbb{B}^M$  and  $\mathbb{B}^{|T|}$ . For example, rotating a point cloud around its center of mass and then subsampling it is not the same as rotating around the center of mass of the subsampled point cloud. Therefore,  $d_{\mathrm{out}}(f(x)_{\mathbb{T}}, g^{-1} \bullet f(g \bullet x')_{\mathbb{T}})$  may be very large, even if  $f$  is perfectly equivariant and not affected by the small perturbation  $(x' - x)$ . Instead, we need to measure output distance using
$$
d _ {\text {o u t}} (f (x) _ {\mathbb {T}}, (g ^ {- 1} \bullet f (g \bullet x ^ {\prime})) _ {\mathbb {T}}).
$$
Combining local budgets and local robustness leads to the following definition:
Definition 3. Consider a ground truth function  $y: \mathbb{X} \to \mathbb{Y}$  with  $\mathbb{X} = \mathbb{A}^N$  and  $\mathbb{Y} = \mathbb{B}^N$  for some sets  $\mathbb{A}, \mathbb{B}$ . Further consider input distance function  $d_{\mathrm{in}}: \mathbb{A}^N \times \mathbb{A}^N \to \mathbb{R}_+$ , local distance function  $d_{\mathrm{loc}}: \mathbb{A} \times \mathbb{A} \to \mathbb{R}_+$ , a set of output indices  $\mathbb{T} \subseteq \{1, \dots, M\}$  and an output distance function  $d_{\mathrm{out}}: \mathbb{B}^{|T|} \times \mathbb{B}^{|T|} \to \mathbb{R}_+$ . Assume that  $y$  is equivariant with respect to the action of group  $\mathbb{G}$ . Then, a prediction  $f(x)$  for clean input  $x \in \mathbb{X}$  is  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \rho, \delta, \mathbb{T})$ -equivariant-robust if
$$
\max_{x^{\prime}\in \mathbb{X}}\max_{g\in \mathbb{G}}d_{\text{out}}\left(f(x)_{\mathbb{T}},(g^{-1}\bullet f(g\bullet x^{\prime}))_{\mathbb{T}}\right)
$$
$$
s. t. d _ {\text {i n}} (x, x ^ {\prime}) \leq \epsilon , \quad \forall n: d _ {\text {l o c}} \left(x _ {n}, x _ {n} ^ {\prime}\right) \leq \rho_ {n},
$$
is less than or equal to  $\delta$ .
Similar to what we discussed in Section 5, using an equivariant model  $f$  means that  $d_{\mathrm{out}}\left(f(x)_{\mathbb{T}},(g^{-1}\bullet f(g\bullet x^{\prime}))_{\mathbb{T}}\right) = d_{\mathrm{out}}\left(f(x)_{\mathbb{T}},f(x^{\prime})_{\mathbb{T}}\right)$  and we recover the traditional notion of adversarial robustness with local constraints for the subset of predictions  $\mathbb{T}$ .
H Definition of robustness for non-compact sets
For our discussion in Section 4, we assumed compactness of all optimization domains, so that minima and maxima always exist. In particular, we assumed that the set  $\{x' \in \mathbb{X} \mid d_{\mathrm{in}}(x,x') \leq \epsilon\}$  is compact for all  $\epsilon \in \mathbb{R}_+$  and thus contains all its limit points. If this is not the case, the minimum  $\min_{g \in \mathbb{G}} d_{\mathrm{in}}(x,g \bullet x')$  may not exist for certain groups  $\mathbb{G}$  and perturbed inputs  $x' \in \mathbb{X}$ . In this case, the action-induced distance needs to be defined as  $\hat{d}_{\mathrm{in}}(x,x') = \inf_{g \in \mathbb{G}} d_{\mathrm{in}}(x,g \bullet x')$ .
Furthermore, our discussion on perturbation models in Section 4.1 may no longer hold. We may have  $\{x' \mid \hat{d}_{\mathrm{in}}(x,x') \leq \epsilon\} \supseteq \{g \bullet x' \mid g \in \mathbb{G}, d_{\mathrm{in}}(x,x') \leq \epsilon\}$ , since the l.h.s. set also contains perturbed inputs  $x' \in \mathbb{X}$  that can be mapped arbitrarily close to a  $d_{\mathrm{in}}$ -ball of radius  $\epsilon$ , whereas the r.h.s. set only contains perturbed inputs that can be mapped into the interior of the ball (via the inverse group element  $g^{-1}$ ). To ensure equality, we also need to include these limit points. This leads us to the following definition:
Definition 4. Assume that ground truth function  $y: \mathbb{X} \to \mathbb{Y}$  is equivariant with respect to the action of group  $\mathbb{G}$ . Further define the set of limit points
$$
\mathbb {L} = \left\{g \bullet x ^ {\prime} \mid \inf  _ {h \in \mathbb {G}} d _ {\text {i n}} (x, h \bullet x ^ {\prime}) = \epsilon \wedge g \in \arg \inf  _ {g \in \mathbb {G}} d _ {\text {i n}} (x, g \bullet x ^ {\prime}) \right\}.
$$
Then, a prediction  $f(x)$  for clean input  $x \in \mathbb{X}$  is  $(\mathbb{G}, d_{\mathrm{in}}, d_{\mathrm{out}}, \epsilon, \delta)$ -equivariant-robust if
$$
\left(\max  _ {x ^ {\prime} \in \mathbb {X}} \max  _ {g \in \mathbb {G}} d _ {\mathrm {o u t}} (f (x), g ^ {- 1} \bullet f (g \bullet x ^ {\prime})\right) \text {s . t .} d _ {\mathrm {i n}} (x, x ^ {\prime}) \leq \epsilon \lor x ^ {\prime} \in \mathbb {L}) \leq \delta .
$$
One could also constrain  $x'$  to be in the closure of  $\{x' \in \mathbb{X} \mid d_{\mathrm{in}}(x, x') \leq \epsilon\}$ . This would however correspond to a potentially stronger notion of robustness, since the closure may also contain limit points that are not in  $\mathbb{L}$ . The model may thus need to be robust to a larger set of perturbed inputs.
Note that all our discussions in Section 5 also apply to this notion of robustness.
I Definition of robustness for non-isometric group actions
For our discussion in Section 4, we assumed that group  $\mathbb{G}$  acts isometrically on input space  $\mathbb{X}$ , i.e.  $\forall x, x' \in \mathbb{X}, \forall g \in \mathbb{G}: d_{\mathrm{in}}(g \bullet x, g \bullet x') = d_{\mathrm{in}}(x, x')$ . If this is not the case, one could also try to define an action-induced distance as follows:
$$
\hat {d} _ {\mathrm {i n}} (x, x ^ {\prime}) = \min  _ {g, h \in \mathbb {G}} d _ {\mathrm {i n}} (h \bullet x, g \bullet x ^ {\prime}),
$$