SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Corollary 4. It follows immediately from applying Theorem 13 in [8] to Corollary 3 with  $\mathcal{G} = \mathcal{G}_{G\to G^{\prime}}$
C.1 Example: Gaussian data model with approximate symmetries
Example 3.1 considers  $\mathcal{G} = \mathcal{S}_3, \mathcal{G} = \mathcal{S}_2, \mathcal{X} = \mathbb{R}^3, \mathcal{Y} = \mathbb{R}^3$ , and  $x \sim \mathcal{N}(0, \sigma_X^2 I_d)$ . The target function is linear, i.e.,  $f^*(x) = \Theta^\top x$  for some  $\Theta \in \mathbb{R}^{3 \times 3}$ . In other words, we are learning linear functions on a fixed graph domain with 3 nodes. Suppose the target function is  $\mathcal{S}_2$ -equivariant such that it has the form
$$
\Theta = \left[ \begin{array}{l l l} a & b & c \\ b & a & c \\ d & d & e \end{array} \right], \quad a, b, c, d, e \in \mathbb {R}. \tag {44}
$$
Now, we project  $\Theta$  in (44) to  $S_{3}$ -equivariant space using the intertwined average 6 with the canonical permutation representation of  $S_{3}$ . A direct calculation yields
$$
\Psi_ {\mathcal {S} _ {3}} (\Theta) = \left[ \begin{array}{l l l} \frac {1}{3} (2 a + e) & \frac {1}{3} (b + c + d) & \frac {1}{3} (b + c + d) \\ \frac {1}{3} (b + c + d) & \frac {1}{3} (2 a + e) & \frac {1}{3} (b + c + d) \\ \frac {1}{3} (b + c + d) & \frac {1}{3} (b + c + d) & \frac {1}{3} (2 a + e) \end{array} \right] \tag {45}
$$
$$
\Psi_ {\mathcal {S} _ {3}} ^ {\perp} (\Theta) = \Theta - \Psi_ {\mathcal {S} _ {3}} (\Theta) = \left[ \begin{array}{c c c} \frac {1}{3} (a - e) & \frac {1}{3} (2 b - c - d) & \frac {1}{3} (- b + 2 c - d) \\ \frac {1}{3} (2 b - c - d) & \frac {1}{3} (a - e) & \frac {1}{3} (- b + 2 c - d) \\ \frac {1}{3} (- b - c + 2 d) & \frac {1}{3} (- b - c + 2 d) & \frac {1}{3} (- 2 a + 2 e). \end{array} \right] \tag {46}
$$
Therefore, the bias term evaluates to
$$
- \sigma_ {X} ^ {2} \| \Psi_ {S _ {3}} ^ {\perp} (\Theta) \| ^ {2} = - \sigma_ {X} ^ {2} \left(\frac {2 (a - e) ^ {2}}{3} + \frac {2 (- 2 b + c + d) ^ {2}}{9} + \frac {2 (b - 2 c + d) ^ {2}}{9} + \frac {2 (b + c - 2 d) ^ {2}}{9}\right). \tag {47}
$$
For the variance term, recall  $\chi_{\psi_{S_3}}, \chi_{\phi_{S_3}}$  are both the canonical permutation representations of  $S_3$ , we have
$$
\left(\chi_ {\psi_ {S _ {3}}} \mid \chi_ {\phi_ {S _ {3}}}\right) = \frac {1}{6} \left(3 ^ {2} + 1 ^ {2} + 1 ^ {2} + 1 ^ {2} + 0 ^ {2} + 0 ^ {2}\right) = 2. \tag {48}
$$
Figure 6: Choosing the symmetry group corresponding to the target function usually yields the best generalization ((a), (b), (d)), but not always: when the number of training data  $n$  is small and the target function  $f$  is approximately equivariant with respect to a larger group, choosing the larger symmetry group could yield further generalization gain, as shown in (c) empirically. The dashed gray vertical line highlights the theoretical threshold  $n^* \approx 35$ , before which using  $S_3$  yields better generalization than  $S_2$ , validating our theoretical analysis. We set  $\sigma_X^2 = 1$ ,  $\sigma_\xi^2 = \frac{1}{64}$ , conduct 10 random runs and compute the generalization error based on 300 test points. We obtain the estimators via stochastic gradient descent, and enforce the symmetry via tying weights. The titles of each subplot indicate the symmetry of the target function, and display the target function values.
Therefore, the variance term evaluates to
$$
\sigma_ {\xi} ^ {2} \frac {N ^ {2} - \left(\chi_ {\psi | _ {\mathcal {G}}} \mid \chi_ {\psi | _ {\mathcal {G}}}\right)}{n - N - 1} = \sigma_ {\xi} ^ {2} \frac {7}{n - 4}. \tag {49}
$$
Putting (47) and (49) together yields the generalization gap of for the least square estimator  $f_{\hat{\Theta}}$  compared to its  $S_3$ -equivariant version  $f_{\Psi_{S_3}(\hat{\Theta})}$ .
As a comparison, when choosing the symmetry group of the target function  $\mathcal{G} = \mathcal{S}_2$ , the bias vanishes and note that  $\left(\chi_{\psi_{\mathcal{S}_2}} \mid \chi_{\phi_{\mathcal{S}_2}}\right) = \frac{1}{2}(3^2 + 1^2) = 5$ , so generalization gap is
$$
\mathbb {E} \left[ \Delta \left(f _ {\hat {\Theta}}, f _ {\Psi_ {S _ {2}} (\hat {\Theta})}\right) \right] = \sigma_ {\xi} ^ {2} \frac {4}{n - 4}. \tag {50}
$$
We see that choosing  $\mathcal{G} = S_3$  is better if  $a \approx e, b \approx c \approx d$  (i.e.,  $f^*$  is approximately  $S_3$ -invariant) and the training sample size  $n$  small, whereas  $S_2$  is better vice versa. This analysis illustrates the advantage of choosing a (suitably) larger symmetry group to induce a smaller hypothesis class when learning with limited data, and introduce useful inductive bias when the target function is approximately symmetric with respect to a larger group. We further illustrate our theoretical analysis via simulations, with details and results shown in Figure 6.
C.2 Example: Approximately Equivariant Mapping on a Geometric Graph
In this section, we illustrate a construction of an approximately equivariant mapping. We focus on a version of Definition 3 that does not take into account the symmetries of  $G'$ . Namely, we consider a definition of the approximate symmetries as
$$
\mathcal {G} _ {G \rightarrow G ^ {\prime}} := \mathcal {S} _ {c _ {1}} \times \mathcal {S} _ {c _ {2}} \dots \times \mathcal {S} _ {c _ {M}} \subset \mathcal {S} _ {N}.
$$
Equivalently, we restrict the analysis to coarsening graphs  $G'$  that are asymmetric.
Background from graphon-signal analysis. To support our construction, we cite some definitions and results from [79].
Definition 8. Let  $r > 0$ . The graphon-signal space with signals bounded by  $r$  is  $\mathcal{WL}_r \coloneqq \mathcal{W} \times L_r^\infty[0,1]$ , where  $L_r^\infty[0,1]$  is the ball of radius  $r$  in  $L^\infty[0,1]$ . The distance in  $\mathcal{WL}_r$  is defined for  $(W,s), (V,g) \in \mathcal{WL}_r$  by
$$
d _ {\square} \big ((W, s), (V, g) \big) := \| (W, s) - (V, g) \| _ {\square} := \| W - V \| _ {\square} + \| s - g \| _ {1}.
$$
$$
\delta_ {\square} \big ((W, s), (V, g) \big) = \inf  _ {\phi} d _ {\square} \big ((W, s), (V ^ {\phi}, g ^ {\phi}) \big),
$$
where  $g^{\phi}(x) = g(\phi(x))$  and  $\phi$  is a measure-preserving bijection.
Any graph-signal induces a graphon signal in the natural way, as in Definition 1. The cut norm and distance between two graph-signals is defined to be the cut norm and distance between the two induced graphon-signal respectively. Similarly, the  $L_{1}$  distance between a signal  $q$  on a graph and a signal  $s$  on  $[0,1]$  is defined to be the  $L_{1}$  distance between the induced signal from  $q$  and  $s$ . The supremum in the definition of cut distance between two induced graphon-signals is realized by some measure preserving bijection.
Sampling graphon-signals. The following construction is from [79, Section 3.4]. Let  $\Lambda = (\lambda_1, \ldots, \lambda_N) \in [0,1]^N$  be  $N$  independent uniform random samples from  $[0,1]$ , and  $(W,s) \in \mathcal{WL}_r$ . We define the random weighted graph  $W(\Lambda)$  as the weighted graph with  $N$  nodes and edge weight  $w_{i,j} = W(\lambda_i, \lambda_j)$  between node  $i$  and node  $j$ . We similarly define the random sampled signal  $s(\Lambda)$  with value  $s_i = s(\lambda_i)$  at each node  $i$ . Note that  $W(\Lambda)$  and  $s(\Lambda)$  share the sample points  $\Lambda$ . We then define a random simple graph as follows. We treat each  $w_{i,j} = W(\lambda_i, \lambda_j)$  as the parameter of a Bernoulli variable  $e_{i,j}$ , where  $\mathbb{P}(e_{i,j} = 1) = w_{i,j}$  and  $\mathbb{P}(e_{i,j} = 0) = 1 - w_{i,j}$ . We define the random simple graph  $\mathbb{G}(W, \Lambda)$  as the simple graph with an edge between each node  $i$  and node  $j$  if and only if  $e_{i,j} = 1$ . The following theorem is [79, Theorem 3.6]
Theorem 3.6 from [79] (Sampling lemma for graphon-signals). Let  $r > 1$ . There exists a constant  $N_0 > 0$  that depends on  $r$ , such that for every  $N \geq N_0$ , every  $(W, s) \in \mathcal{WL}_r$ , and for  $\Lambda = (\lambda_1, \ldots, \lambda_N) \in [0,1]^N$  independent uniform random samples from  $[0,1]$ , we have
$$
\mathbb {E} \left(\delta_ {\square} \left(\left(W, s\right), \left(\mathbb {G} (W, \Lambda), s (\Lambda)\right)\right)\right) <   \frac {1 5}{\sqrt {\log (N)}}. \tag {51}
$$
By Markov's inequality and (51), for any  $0 < p < 1$ , there is an event of probability  $1 - p$  (regarding the choice of  $\Lambda$ ) in which
$$
\delta_ {\square} \left(\left(W, s\right), \left(\mathbb {G} (W, \Lambda), s (\Lambda)\right)\right) <   \frac {1 5}{p \sqrt {\log (N)}}. \tag {52}
$$
Stability to deformations of mappings on geometric graphs. Let  $\mathcal{M}$  be a metric space with an atomless standard probability measure defined over the Borel sets (up to completion of the measure). Such a probability space is equivalent to the standard probability space  $[0,1]$  with Lebesgue measure. Namely, there are co-null sets  $A\subset \mathcal{M}$  and  $B\subset [0,1]$ , and a measure preserving bijection  $\phi :A\to B$ . Hence, graphon analysis applied as-is when replacing the domain  $[0,1]$  with  $\mathcal{M}$ . Suppose that we are interested in a target function  $f_{\mathcal{M}}:L^{1}(\mathcal{M})\to L^{1}(\mathcal{M})$  that is stable to deformations in the following sense.
Definition 9. Let  $\epsilon >0$ . A measurable bijection  $\nu :\mathcal{M}\to \mathcal{M}$  is called a deformation up to  $\epsilon$ , if there exists an event  $B_{\epsilon}\subset \mathcal{M}$  with probability greater than  $1 - \epsilon$  such that for every  $x\in B_{\epsilon}$
$$
d _ {\mathcal {M}} (\nu (x), x) <   \epsilon .
$$
The mapping  $f_{\mathcal{M}}:L^{1}(\mathcal{M})\to L^{1}(\mathcal{M})$  is called stable to deformations with stability constant  $C$ , if for any deformation  $\nu$  up to  $\epsilon$ , and every  $s\in L^1 (\mathcal{M})$ , we have
$$
\left\| f _ {\mathcal {M}} (s) - f _ {\mathcal {M}} \left(s \circ \nu\right) \circ \nu^ {- 1} \right\| _ {1} <   C \epsilon .
$$
Suppose that we observe a discretized version of the domain  $\mathcal{M}$ , defined as follows. There is a graphon  $W: \mathcal{M}^2 \to [0,1]$  defined as
$$
W (x, y) = r \left(d (x, y)\right), \tag {53}
$$
where  $r: \mathbb{R}_+ \to [0,1]$  is a decreasing function with support  $[0,\rho]$ . Instead of observing  $W$ , we observe a graph  $G = \mathbb{G}(W, \Lambda)$  with node set  $[N]$ , sampled from  $W$  on the random independent points  $\Lambda = \{\lambda_n\}_{n=1}^N \subset \mathcal{M}$  as above. Suppose moreover that any graph signal is sampled from a signal in  $L^1(\mathcal{M})$ , on the same random points  $\Lambda$ , as above. Suppose that the target  $f_{\mathcal{M}}$  on the continuous
domain is well approximated by some mapping  $f^{*}:L^{1}[N]\to L^{1}[N]$  on the discrete domain in the following sense. For every  $s\in L^1 (\mathcal{M})$ , let  $s_G$  be the graph signal sampled on the random samples  $\{\lambda_n\}_n$ . Then there is an event of high probability such that
$$
\left\| f ^ {*} s _ {G} - \left\{\left(f _ {\mathcal {M}} (s)\right) \left(x _ {n}\right) \right\} _ {n} \right\| _ {1} <   e
$$
for some small  $e$ . We hence consider  $f^{*}$  as the target mapping of the learning problem. One example of such a scenario is when there exists some Lipschitz continuous mapping  $\Theta : \mathcal{WL}_r \to \mathcal{WL}_r$  with Lipschitz constant  $L$ , such that  $f_{\mathcal{M}} = \Theta(W, \cdot)$  and  $f^{*} = \Theta(G, \cdot)$ . Indeed, by (52), for some  $p$  as small as we like, there is an event of probability  $1 - p$  in which, up to a measure preserving bijection,
$$
\begin{array}{l} \left\| f _ {\mathcal {M}} s - f ^ {*} s _ {G} \right\| _ {1} \leq \delta_ {\square} \Big (\big (W, f _ {\mathcal {M}} s \big), \big (G, f ^ {*} s _ {G} \big) \Big) \\ \leq L \delta_ {\square} \left(\left(W, s\right), \left(G, s _ {G}\right)\right) <   \frac {1 5 L}{p \sqrt {\log (N)}} = e. \tag {54} \\ \end{array}
$$
A concrete example is when  $\Theta$  is a message passing neural network (MPNN) with Lipschitz continuous message and update functions, and normalized sum aggregation [79, Theorem 4.1].
Let  $G'$  be a graph that coarsens  $G$  up to error  $\epsilon$ . In the same event as above, by (52), up to a measure preserving bijection,
$$
\delta_ {\square} \left(W _ {G ^ {\prime}}, W\right) \leq \delta_ {\square} \left(W _ {G ^ {\prime}}, W _ {G}\right) + \delta_ {\square} \left(W _ {G}, W\right) \leq \epsilon + e = u. \tag {55}
$$
We next show an approximation property that we state here informally: Since  $W(x,y) \approx 0$  for  $x$  away from  $y$ , we must have  $W_{G'}(x,y) \approx 0$  as well for a set of high measure. Otherwise,  $\delta_{\square}(W_{G'},W)$  cannot be small. By this, any approximate symmetry of  $G$  is a small deformation, and, hence,  $f^{*}$  is an approximately equivariant mapping.
Equivariant mappings on geometric graphs. In the following, we construct a scenario in which  $f^{*}$  can be shown to be approximately equivariant in a restricted sense. For simplicity, we assume  $f^{*}(s_{G})\in L^{2}[0,1]$ , and restrict to the case  $r = \mathbb{1}_{[0,\rho ]}$  in the geometric graphon  $W$  of (53). Denote the induced graphon  $W_{G^{\prime}} = T$ . Given  $h > 0$ , define the  $h$ -diagonal
$$
d _ {h} = \{(x, y) \in \mathcal {M} ^ {2} \mid d _ {\mathcal {M}} (x, y) \leq h \}.
$$
In the following, all distances are assumed to be up to the best measure preserving bijection.
If there is a domain  $S' \times T' \in \mathcal{M}^2$  outside the  $\rho$ -diagonal in which  $T(x,y) > c$  for some  $c > 0$ , by reverse triangle inequality, we must have
$$
\| W - T \| _ {\square} \geq \int_ {S ^ {\prime}} \int_ {T ^ {\prime}} T (x, y) d y d x = c \mu \left(S ^ {\prime}\right) \mu \left(T ^ {\prime}\right).
$$
Hence, since by (55),  $\| W - T \|_{\square} < u$ , for every  $S' \times T'$  that does not intersect  $d_{\rho}$ , we must have
$$
\int_ {S ^ {\prime}} \int_ {T ^ {\prime}} T (x, y) d y d x \leq u.
$$
In other words, for any two sets  $S, T$  with distance more than  $\rho$  ( $\inf_{s \in S, t \in T} d_{\mu}(s, t) > \rho$ ), we have
$$
\int_ {S} \int_ {T} T (x, y) d y d x \leq u.
$$
This formalizes the statement  $W_{G'}(x, y) \approx 0$  for  $x$  away from  $y$  from above.
Next, we develop the analysis for the special case  $\mathcal{M} = [0,1]$  with the standard metric and Lebesgue probability measure. We note that the analysis can be extended to  $\mathcal{M} = [0,1]^D$  for a general dimension  $D\in \mathbb{N}$ . For every  $z\in [0,1]$ , we have
$$
\int_ {[ z + \rho / \sqrt {2}, 1 ]} \int_ {[ 0, z - \rho / \sqrt {2} ]} T (x, y) d y d x \leq u,
$$
$$
\int_ {[ 0, z - \rho / \sqrt {2} ]} \int_ {[ z + \rho / \sqrt {2}, 1 ]} T (x, y) d y d x \leq u.
$$
Let  $\nu > 0$ . We take a grid  $\{x_{j}\} \in [0,1]$  of spacing  $\sqrt{2}\nu$ . The sets
$$
\bigcup_ {j} \left[ x _ {j} + \rho / \sqrt {2}, 1 \right] \times \left[ 0, x _ {j} - \rho / \sqrt {2} \right], \quad \bigcup_ {j} \left[ 0, x _ {j} - \rho / \sqrt {2} \right] \times \left[ x _ {j} + \rho / \sqrt {2}, 1 \right]
$$
cover  $d_{\nu}^{c}$  (where  $d_{\nu}^{c}$  is the complement of  $d_{\nu}$ ). Hence,
$$
\begin{array}{l} \iint_ {d _ {\nu} ^ {c}} T (x, y) d y d x \leq \sum_ {j = 1} ^ {1 / \sqrt {2} \nu} \int_ {[ x _ {j} + \rho / \sqrt {2}, 1 ]} \int_ {[ 0, x _ {j} - \rho / \sqrt {2} ]} T (x, y) d y d x \\ + \sum_ {j = 1} ^ {1 / \sqrt {2} \nu} \int_ {[ 0, x _ {j} - \rho / \sqrt {2} ]} \int_ {[ x _ {j} + \rho / \sqrt {2}, 1 ]} T (x, y) d y d x \\ \leq \frac {2}{\sqrt {2} \nu} u. \\ \end{array}
$$
We take  $\frac{2}{\sqrt{2}\nu} u = t$ , for  $u \ll t \ll 1$ , namely,  $\nu = \sqrt{2}\frac{u}{t}$ . For example, we may take  $t = \sqrt{2} u^{1/3}$ , and  $\nu = u^{2/3}$ , assuming that  $\rho < u^{1/3}$ . Hence, we have
$$
\iint_{\substack{d^{c}\\ u^{2 / 3}}}T(x,y)\leq \sqrt{2} u^{1 / 3}.
$$
To conclude, the probability of having an edge between nodes  $\lambda_{i}$  and  $\lambda_{j}$  in  $\overline{G_N'}$  which are further away than  $u^{2/3}$ , namely,  $d_{\mathcal{M}}(\lambda_i,\lambda_j) > u^{2/3}$ , is less than  $\sqrt{2} u^{1/3}$ .
Suppose that  $G'$  is asymmetric. This means that symmetries of  $\mathcal{G}_{G \to G'}$  can only permute between nodes that have an edge between them in the blown-up graph  $\overline{G'}_N$ . The probability of having an edge between nodes further away than  $u^{2/3}$  is less than  $\sqrt{2} u^{1/3}$ . Hence, a symmetry in  $\mathcal{G}_{G \to G'}$  can be seen as a small deformation, where for each node  $\lambda_i$  and a random uniform  $g \in \mathcal{G}_{G \to G'}$ , the probability that  $\lambda_i$  it is mapped by  $g$  to a node of distance less than  $u^{2/3}$  is more than  $1 - \sqrt{2} u^{1/3}$ .
Any symmetry  $g$  in  $\mathcal{G}_{G \to G'}$  induces a measure-preserving bijection  $\nu$  in  $\mathcal{M} = [0,1]$ , by permuting the intervals of the partition  $\mathcal{P}_N$  of Definition 1. As a result, the set of points that are mapped further away than  $u^{2/3}$  under  $\nu$  has probability upper bounded by  $\sqrt{2} u^{1/3}$ , and symmetries in  $\mathcal{G}_{G \to G'}$  can be seen as a small deformation  $\nu$  according to Definition 9 (in high probability). This means that, for any  $g \in \mathcal{G}_{G \to G'}$ ,
$$
\left\| f _ {\mathcal {M}} (s) - f _ {\mathcal {M}} (s \circ g) \circ g ^ {- 1} \right\| _ {1} <   C \sqrt {2} u ^ {1 / 3},
$$
so by the triangle inequality, combining with equation 54, we have
$$
\left\| f ^ {*} \left(s _ {G}\right) - g ^ {- 1} f ^ {*} \left(g s _ {G}\right) \right\| _ {1} <   2 e + C \sqrt {2} u ^ {1 / 3} = \epsilon^ {\prime}. \tag {56}
$$
Equation (56) leads to
$$
\begin{array}{l} \left\| f ^ {*} \left(s _ {G}\right) - \mathcal {Q} _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} \left(f ^ {*}\right)\left(s _ {G}\right)\right\| _ {1} = \left\| f ^ {*} \left(s _ {G}\right) - \frac {1}{\left| \mathcal {G} _ {G \rightarrow G ^ {\prime}} \right|} \sum_ {g \in \mathcal {G} _ {G \rightarrow G ^ {\prime}}} g ^ {- 1} f ^ {*} \left(g s _ {G}\right)\right\| _ {1} (57) \\ \leq \frac {1}{\left| \mathcal {G} _ {G \rightarrow G ^ {\prime}} \right|} \sum_ {g \in \mathcal {G} _ {G \rightarrow G ^ {\prime}}} \| f ^ {*} (s _ {G}) - g ^ {- 1} f ^ {*} (g s _ {G}) \| _ {1} <   \epsilon^ {\prime}. (58) \\ \end{array}
$$
Since for any  $q \in L^2[0,1] \cap L^\infty[0,1]$  we have  $\|q\|_2^2 \leq \|q\|_\infty \|q\|_1$ , we can bound
$$
\begin{array}{l} \| f ^ {*} (s _ {G}) - \mathcal {Q} _ {\mathcal {G} _ {G \to G ^ {\prime}}} (f ^ {*}) (s _ {G}) \| _ {2} <   \sqrt {\| f ^ {*} (s _ {G}) - \mathcal {Q} _ {\mathcal {G} _ {G \to G ^ {\prime}}} (f ^ {*}) (s _ {G}) \| _ {\infty}} \sqrt {\epsilon^ {\prime}} \\ <   \sqrt {\| f ^ {*} (s _ {G}) \| _ {\infty} + \| \mathcal {Q} _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} (f ^ {*}) (s _ {G}) \| _ {\infty}} \sqrt {\epsilon^ {\prime}} \\ <   \sqrt {2 \| f ^ {*} \left(s _ {G}\right) \| _ {\infty}} \sqrt {\epsilon^ {\prime}}, \tag {59} \\ \end{array}
$$
where the last inequality follows from  $\| \mathcal{Q}_{\mathcal{G}_{G\to G^{\prime}}}(\boldsymbol {f}^{*})(s_{G})\|_{\infty}\leq \| \boldsymbol{f}^{*}(\boldsymbol {s}_{G})\|_{\infty}$
Denote  $\| f^{*}\|_{\infty} \coloneqq \int \| f^{*}(s_{G})\|_{\infty}d\mu (s_{G})$ , and suppose that  $\| f^{*}\|_{\infty}$  is finite. Hence, if  $\mu$  is a probability measure, we have
$$
\left\| f ^ {*} - \mathcal {Q} _ {\mathcal {G} _ {G \to G ^ {\prime}}} (f ^ {*}) \right\| _ {\mu} <   \sqrt {2 \| f ^ {*} \| _ {\infty}} \sqrt {\epsilon^ {\prime}}.
$$
This shows an example of approximately equivariant mapping based on random geometric graph, where the approximation rate is also a function of the size of the graph  $N$ , and goes to zero as  $N \to \infty$  and  $\epsilon \to 0$ .
In future work, we will extend this example to more general metric space  $\mathcal{M}$  and to non-trivial symmetry groups  $\overline{\mathcal{A}}_{G'}$ . Intuitively, most random geometric graphs are "close to asymmetric". This means that for "most"  $G'$ , the symmetries of  $\overline{\mathcal{A}}_{G'}$  can only permute between nodes connected by an edge, and so are the symmetries of  $\mathcal{G}_{G \to G'}$ . For this, we need to extend Definition 9 by treating  $G'$  probabilistically.
D Experiment Details
In this section, we provide additional details of our experiments. We first give a brief introduction of standard graph neural networks (Section D.1), followed by in-depth explanations of our applications in image inpainting (Section D.2), traffic flow prediction (Section D.3), and human pose estimation (Section D.4). All experiments were conducted on a server with 256 GB RAM and 4 NVIDIA RTX A5000 GPU cards.
D.1 Graph Neural Networks (GNNs)
We consider standard message-passing graph neural networks (MPNNs) [19-21] defined as follows. A  $L$ -layer MPNN maps input  $X \in \mathbb{R}^{N \times d}$  to output  $Y \in \mathbb{R}^{N \times k}$  following an iterative scheme: At initialization,  $\mathbf{h}^{(0)} = X$ ; At each iteration  $l$ , the embedding for node  $i$  is updated to
$$
\mathbf {h} _ {i} ^ {(l)} = \phi \left(\mathbf {h} _ {i} ^ {(l - 1)}, \sum_ {j \in \mathcal {N} (i)} \psi \left(\mathbf {h} _ {i} ^ {(l - 1)}, \mathbf {h} _ {j} ^ {(l - 1)}, A _ {[ i, j ]}\right)\right), \tag {60}
$$
where  $\phi, \psi$  are the update and message functions,  $\mathcal{N}(i)$  denotes the neighbors of node  $i$ , and  $A_{[i,j]}$  represents the  $(i,j)$ -edge weight. MPNNs typically have two key design features: (1)  $\phi, \psi$  are shared across all nodes in the graph, typically chosen to be a linear transformation or a multi-layer perceptions (MLPs), known as global weight sharing; (2) the graph  $A$  is used for (spatial) convolution.
D.2 Application: Image Inpainting
We provide additional details of the data, model, and optimization procedure used in the experiment.
Data. We consider the datasets MNIST [74] and FashionMNIST [75]. For each dataset, we take 100 training samples and 1000 test samples via stratified random sampling. The input and output graph signals are  $(m_i\odot x_i,x_i)$  ( $\odot$  is entrywise multiplication). Here  $x_{i}\in \mathbb{R}^{28\times 28}\equiv \mathbb{R}^{784}$  denotes the image signals and  $m_{i}$  denotes a random mask (size  $14\times 14$  for MNIST and  $20\times 20$  for FashionMNIST). For experiments with reflection symmetry on the coarsened graph, we further transform each image in the FashionMNIST subset using horizontal flip with probability 0.5 (FashionMNIST+hflip). We remark that it is possible to model the dihedral group  $\mathcal{D}_4$  on the square as the coarsened graph symmetry, yet we only consider the reflection symmetry  $\mathcal{S}_2$  to simplify the parameterization (since it is Abelien while  $\mathcal{D}_4$  is not) while capturing the most relevant symmetries.
Model. We consider a 2-layer  $\mathcal{G}_{G\to G^{\prime}}$  -equivariant networks  $\mathcal{G}$  -Net, a composition of  $f_{\mathrm{out}}\circ \mathrm{ReLU}\circ f_{\mathrm{in}}$  , where  $f_{\mathrm{in}},f_{\mathrm{out}}$  denote the input/output equivariant linear layers. The input and output feature dimension is 1 (since the signals are grayscale images), and the hidden dimension is set as 28. For comparison, we also consider a simple 1-layer  $\mathcal{G}_{G\to G^{\prime}}$  -equivariant networks  $\mathcal{G}$  Net.
We train the models with ADAM (learning rate 0.01, no weight decay, at most 1000 epochs). We report the best test accuracy at the model checkpoint selected by the best validation accuracy (with a 80/20 training-validation split).
We supplement Figure 2 with Table 7 for further numerical details.
MSE (×1e-2) ↓	S282=Sn	(S142)4 = (Sn/4)4	(S72)16 = (Sn/16)16	(S42)49 = (Sn/49)49	(S22)196 = (Sn/196)196	Trivial = (Sn/784)784
MNIST	41.56 ± 0.16	40.53 ± 0.26	36.06 ± 0.24	34.68 ± 0.5	33.67 ± 0.07	33.92 ± 0.04
Fashion	23.48 ± 0.14	22.26 ± 0.02	16.94 ± 0.08	15.16 ± 0.1	14.47 ± 0.11	14.75 ± 0.11
Table 7: Image inpainting using  $\mathcal{G}$ -Net with different levels of coarsening. Table shows mean squared error (MSE) across 3 runs on the test set, supplementing Figure 2 (Left, blue curves).
D.3 Application: Traffic Flow Prediction