SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f = \left[ \begin{array}{c c c c} f _ {1 1} & f _ {1 2} & \dots & f _ {1 M} \\ f _ {2 1} & f _ {2 2} & \dots & f _ {2 M} \\ & & \ddots & \\ f _ {M 1} & f _ {M 2} & \dots & f _ {M M} \end{array} \right], f _ {k k} = a _ {k} I _ {c _ {k}} + b _ {k} \mathbb {1} _ {c _ {k}} \mathbb {1} _ {c _ {k}} ^ {\top}, f _ {k l} = e _ {k l} \mathbb {1} _ {c _ {k}} \mathbb {1} _ {c _ {l}} ^ {\top} \text {f o r} k \neq l, \tag {20}
$$
where  $f_{kl}$  are block matrices,  $I_k$  is a size- $k$  identity matrix,  $\mathbb{1}_k$  is a length- $k$  vector of all ones, and  $a_k, b_k, e_{kl}$  are scalars. The subscript  $c_k$  denotes the size of the  $k$ -th cluster. Figure 5 illustrates the block structure of  $f$ .
We provide a proof sketch of our claim. Since  $\overline{\mathcal{A}}_{G'}$  is trivial by assumption, it remains to show any function that is equivariant to the product of permutations  $\left(S_1 \times S_2 \dots \times S_M\right)$  admits the form in eqn. 20. We justify as follows:
1. (within-cluster)  $f_{kk}$  is a linear permutation-equivariant function if and only if its diagonal elements are the same and its off-diagonal elements are the same ([34, Lemma 3.]);
2. (across-cluster)  $f_{kl}$  for  $k \neq l$  is a constant matrix since nodes within a cluster are indistinguishable; moreover,  $e_{kl}$  are unconstrained since the coarsened symmetry  $\overline{\mathcal{A}}_{G'}$  is trivial.
Figure 5: The block structure of equivariant linear function  $f: \mathbb{R}^n \to \mathbb{R}^n$  with respect to  $\mathcal{G}_{G \to G'}$  (where  $G, G'$  are asymmetric): Each diagonal block  $f_{kk}$  is diagonally constant and off-diagonally constant; Each off-diagonal block  $f_{kl}$  is a constant matrix.
As an example, we illustrate the equivariant linear layer for two-cluster graph coarsening. Without loss of generality, assume that the node signals  $X$  are ordered according to the cluster assignment (e.g.,  $X_{[1:|V_1|]}$  are node features for the first cluster, etc). Let  $X_{(1)}, X_{(2)}$  denote the node features for the first and second cluster,  $W_{(1)}^s$ ,  $W_{(2)}^s$  denote the weights on the block diagonal for self-feature transformation,  $W_{(1)}^n$ ,  $W_{(2)}^n$  denote the weights on the block diagonal for within-cluster neighbors, and  $W_{(12)}^n$ ,  $W_{(21)}^n$  denote the weights off the block diagonal for across-cluster neighbors. Let  $I$  denote the identity matrix,  $\mathbf{1}_{(1)}$ ,  $\mathbf{1}_{(2)}$  denote the all-ones matrices with the same size as the corresponding
cluster, and  $\mathbf{1}_{(12)}$ ,  $\mathbf{1}_{(21)}$  denote the all-ones matrices mapping across clusters. Recall  $\odot$  denotes the element-wise multiplication of two matrices. Then the equivariant linear layer is parameterized as
$$
I \odot \left[ \begin{array}{c} X _ {(1)} W _ {(1)} ^ {s} \\ X _ {(2)} W _ {(2)} ^ {s} \end{array} \right] + \Big (\left[ \begin{array}{c c} \mathbf {1} _ {(1)} & 0 \\ 0 & \mathbf {1} _ {(2)} \end{array} \right] - I \Big) \odot \left[ \begin{array}{c} X _ {(1)} W _ {(1)} ^ {n} \\ X _ {(2)} W _ {(2)} ^ {n} \end{array} \right] + \left[ \begin{array}{c c} 0 & \mathbf {1} _ {(1 2)} \\ \mathbf {1} _ {(2 1)} & 0 \end{array} \right] \odot \left[ \begin{array}{c} X _ {(1)} W _ {(1 2)} ^ {n} \\ X _ {(2)} W _ {(2 1)} ^ {n} \end{array} \right]. \tag {21}
$$
For cases where  $\overline{\mathcal{A}}_{G'}$  is nontrivial, if  $\overline{\mathcal{A}}_{G'}$  is abelian, we can use a construction by Serre ([83] Section 8.2). Observe that the symmetry of  $\overline{\mathcal{A}}_{G'}$  effectively constrains the patterns of  $e_{kl}$  in equation 20. We provide an example where  $\overline{\mathcal{A}}_{G'} = S_2$  in our image inpainting experiment (see Section 5.1), and defer the investigation of general cases for future work.
B Proofs of Our Theoretical Results
B.1 Proofs of Generalization with Exact Symmetries
Lemma 1 (Risk Gap). Let  $\mathcal{X} = \mathbb{R}^{N\times d}$ ,  $\mathcal{Y} = \mathbb{R}^{N\times k}$  be the input and output graph signal spaces on a fixed graph  $G$ . Let  $X\sim \mu$  where  $\mu$  is a  $\mathcal{S}_N$ -invariant distribution on  $\mathcal{X}$ . Let  $Y = f^{*}(X) + \xi$ , where  $\xi \in \mathbb{R}^{N\times k}$  is random, independent of  $X$  with zero mean and finite variance and  $f^{*}:\mathcal{X}\to \mathcal{Y}$  is  $\mathcal{A}_G$ -equivariant. Then, for any  $f\in V$  and for any compact group  $\mathcal{G}\subseteq \mathcal{S}_N$ , we can decompose  $f$  as
where  $\bar{f}_{\mathcal{G}} = \mathcal{Q}_{\mathcal{G}}f, f_{\mathcal{G}}^{\perp} = f - \bar{f}_{\mathcal{G}}$ . Moreover, the risk gap satisfies
Lemma 1 is a straightforward extension of Lemma 6 in [8], which makes use of Lemma 1 in [8].
Lemma 1 in [8]. Let  $U$  be any subspace of  $V$  that is closed under  $\mathcal{Q}$ . Define the subspaces  $S$  and  $A$  of, respectively, the  $\mathcal{G}$ -symmetric and  $\mathcal{G}$ -anti-symmetric functions in  $U: S = \{f \in U : f \text{ is } \mathcal{G} \text{-equivariant}\}$  and  $A = \{f \in U : \mathcal{Q}f = 0\}$ . Then  $U$  admits an orthogonal decomposition into symmetric and anti-symmetric parts
$$
U = S \oplus A
$$
Proof of Lemma 1. The first part of Lemma 1  $f = \bar{f}_{\mathcal{G}} + f_{\mathcal{G}}^{\perp}$  follows from Lemma 1 in [8]. For the second part, by the assumption that the noise  $\xi$  is independent of  $X$  with zero mean and finite variance, we can simplify the risk gap as
$$
\begin{array}{l} \Delta (f, \bar {f} _ {\mathcal {G}}) := \mathbb {E} \left[ \| Y - f (X) \| _ {F} ^ {2} \right] - \mathbb {E} \left[ \| Y - \bar {f} _ {\mathcal {G}} (X) \| _ {F} ^ {2} \right] \\ = \mathbb {E} \left[ \| f ^ {*} (X) - f (X) \| _ {F} ^ {2} \right] - \mathbb {E} \left[ \| f ^ {*} (X) - \bar {f} _ {\mathcal {G}} (X) \| _ {F} ^ {2} \right]. \tag {22} \\ \end{array}
$$
Substituting  $f = \bar{f}_{\mathcal{G}} + f_{\mathcal{G}}^{\perp}$  yields
$$
\begin{array}{l} \mathbb {E} \left[ \| f ^ {*} (X) - \bar {f} _ {\mathcal {G}} (X) - f _ {\mathcal {G}} ^ {\perp} (X) \| _ {F} ^ {2} \right] - \mathbb {E} \left[ \| f ^ {*} (X) - \bar {f} _ {\mathcal {G}} (X) \| _ {F} ^ {2} \right] \\ = - 2 \langle f ^ {*} (X) - \bar {f} _ {\mathcal {G}} (X), f _ {\mathcal {G}} ^ {\perp} (X) \rangle_ {\mu} + \mathbb {E} \left[ \| f _ {\mathcal {G}} ^ {\perp} (X) \| _ {F} ^ {2} \right] \\ = - 2 \left\langle f ^ {*}, f _ {\mathcal {G}} ^ {\perp} \right\rangle_ {\mu} + \left\| f _ {\mathcal {G}} ^ {\perp} \right\| _ {\mu} ^ {2}. \tag {23} \\ \end{array}
$$
We remark that Lemma 6 in [8] assumes that  $f^{*}$  is  $\mathcal{G}$ -equivariant, so the first term in (23) vanishes. We are motivated from the symmetry model selection problem, and thereby relax the assumption of the chosen symmetry group  $\mathcal{G}$  can differ from the target symmetry group  $\mathcal{A}_{\mathcal{G}}$ .
Theorem 2 (Bias-Variance-Tradeoff). Let  $\mathcal{X} = \mathbb{R}^{N\times d}$ ,  $\mathcal{Y} = \mathbb{R}^{N\times k}$  be the graph signals spaces on a fixed graph  $G$ . Let  $S_N$  act on  $\mathcal{X}$  and  $\mathcal{Y}$  by permuting the rows with representations  $\phi$  and  $\psi$ . Let  $\mathcal{G}$  be a subgroup of  $S_N$  acting with restricted representations  $\phi|_{\mathcal{G}}$  on  $\mathcal{X}$  and  $\psi|_{\mathcal{G}}$  on  $\mathcal{Y}$ .
Let  $X_{[i,j]}\stackrel {i.i.d.}{\sim}\mathcal{N}\left(0,\sigma_X^2\right)$  and  $Y = f^{*}(X) + \xi$  where  $f^{*}(x) = \Theta^{\top}x$  is  $\mathcal{A}_G$ -equivariant and  $\Theta \in \mathbb{R}^{Nd\times Nk}$ . Assume  $\xi_{[i,j]}$  is random, independent of  $X$ , with mean 0 and  $\mathbb{E}\left[\xi \xi^{\top}\right] = \sigma_{\xi}^{2}I < \infty$ . Let  $\hat{\Theta}$  be the least-squares estimate of  $\Theta$  from  $n$  i.i.d. examples  $\{(X_i,Y_i):i = 1,\dots ,n\}$ ,  $\Psi_{\mathcal{G}}(\hat{\Theta})$  be its equivariant version with respect to  $\mathcal{G}$ . Let  $(\chi_{\psi |\mathcal{G}}|\chi_{\phi |\mathcal{G}}) = \int_{\mathcal{G}}\chi_{\psi |\mathcal{G}}(g)\chi_{\phi |\mathcal{G}}(g)\mathrm{d}\lambda (g)$  denote the inner product of the characters. If  $n > Nd + 1$  the risk gap is
$$
\mathbb {E} \left[ \Delta \left(f _ {\hat {\Theta}}, f _ {\Psi_ {\mathcal {G}} (\hat {\Theta})}\right) \right] = \underbrace {- \sigma_ {X} ^ {2} \| \Psi_ {\mathcal {G}} ^ {\perp} (\Theta) \| _ {F} ^ {2}} _ {b i a s} + \underbrace {\sigma_ {\xi} ^ {2} \frac {N ^ {2} d k - \left(\chi_ {\psi | _ {\mathcal {G}}} \mid \chi_ {\phi | _ {\mathcal {G}}}\right)}{n - N d - 1}} _ {v a r i a n c e}.
$$
Theorem 2 presents the risk gap in expectation, which follows from Lemma 1, by taking  $f$  as the least-squares estimator and using assumptions in the linear regression setting. To this end, we denote  $X \in \mathbb{R}^{n \times Nd}$ ,  $Y \in \mathbb{R}^{n \times Nk}$ ,  $\xi \in \mathbb{R}^{n \times Nk}$  as the training data arranged in matrix form, where  $Y = f^{*}(X) + \xi$ . Recall that the least-squares estimator of  $\Theta$  in the classic regime ( $n > Nd$ ) is given by
$$
\hat {\boldsymbol {\Theta}} := \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {\dagger} \boldsymbol {X} ^ {\top} \boldsymbol {Y} \stackrel {{a. e.}} {{=}} \boldsymbol {\Theta} + \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1} \boldsymbol {X} ^ {\top} \boldsymbol {\xi}, \tag {24}
$$
while its equivariant map is
$$
\Psi_ {\mathcal {G}} (\hat {\Theta}) = \int_ {\mathcal {G}} \phi (g) \hat {\Theta} \psi \left(g ^ {- 1}\right) \mathrm {d} \lambda (g). \tag {25}
$$
Our proof makes use of the following results in [8], which we restate adapted versions here for our setting.
Proposition 11 in [8]. Let  $V = \{f_W : f_W(x) = W^\top x, W \in \mathbb{R}^{d \times k}, x \in \mathbb{R}^d\}$  denote the space of linear functions. Let  $X \sim \mu$  with  $\mathbb{E}[XX^\top] = \Sigma$ . For any linear functions  $f_{W_1}, f_{W_2} \in V$ , the inner product on  $V$  satisfies
$$
\left\langle f _ {W _ {1}}, f _ {W _ {2}} \right\rangle_ {\mu} = \operatorname {T r} \left(W _ {1} ^ {\top} \Sigma W _ {2}\right). \tag {26}
$$
Theorem 13 in [8] (Simplified, Adapted). Consider the same setting as Theorem 2. For  $n > Nd + 1$
$$
\sigma_ {X} ^ {2} \mathbb {E} \left[ \left\| \Psi_ {\mathcal {G}} ^ {\perp} \left(\left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {+} \boldsymbol {X} ^ {\top} \boldsymbol {\xi}\right) \right\| _ {F} ^ {2} \right] = \sigma_ {\xi} ^ {2} \frac {N ^ {2} d k - \left(\chi_ {\psi | _ {\mathcal {G}}} \mid \chi_ {\phi | _ {\mathcal {G}}}\right)}{n - N d - 1}.
$$
Proof of Theorem 2. We first plug in the least-squares expressions  $\hat{\Theta}, \Psi_{\mathcal{G}}(\hat{\Theta})$  to Lemma 1 and treat the mismatch term and constraint term separately; We complete the proof by collecting common terms together.
For the mismatch term, our goal is to compute
$$
- 2 \mathbb {E} \left[ \langle \Theta , \hat {\Theta} - \Psi_ {\mathcal {G}} (\hat {\Theta}) \rangle_ {\mu} \right], \tag {27}
$$
where the expectation is taken over the test point  $X$  and the training data  $\pmb{X},\pmb{\xi}$
To that end, we write
$$
\left(\hat {\Theta} - \Psi_ {\mathcal {G}} (\hat {\Theta})\right) x \stackrel {{a. e.}} {{=}} \Theta^ {\top} x + \boldsymbol {\xi} ^ {\top} \boldsymbol {X} \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1} x - \int_ {\mathcal {G}} \psi (g ^ {- 1}) \left(\Theta^ {\top} + \boldsymbol {\xi} ^ {\top} \boldsymbol {X} \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1}\right) \phi (g) x d \lambda (g). \tag {28}
$$
Taking expectation yields
$$
\begin{array}{l} \mathbb {E} _ {X, X, \boldsymbol {\xi}} \left[ \langle \Theta , \hat {\Theta} - \Psi_ {\mathcal {G}} (\hat {\Theta}) \rangle_ {\mu} \right] = \| \Theta \| _ {\mu} ^ {2} + \mathbb {E} _ {X, X, \boldsymbol {\xi}} \left[ \langle \Theta^ {\top} X, \boldsymbol {\xi} ^ {\top} X (X ^ {\top} X) ^ {- 1} x \rangle \right] \\ \left. - \mathbb {E} _ {X, \boldsymbol {X}, \boldsymbol {\xi}} \left[ \left\langle \Theta^ {\top} x, \int_ {\mathcal {G}} \psi (g ^ {- 1}) \left(\Theta^ {\top} + \boldsymbol {\xi} ^ {\top} \boldsymbol {X} \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1}\right) \phi (g) x \mathrm {d} \lambda (g) \right\rangle \right]. \right. \tag {29} \\ \end{array}
$$
Note that  $\xi$  is independent with  $X$  and mean 0, so the second term in (29) vanishes. Similarly, the part  $\mathbb{E}_{X,X,\xi}\int_{\mathcal{G}}\psi(g^{-1})\left(\boldsymbol{\xi}^\top\boldsymbol{X}(\boldsymbol{X}^\top\boldsymbol{X})^{-1}\right)\phi(g)x\mathrm{d}\lambda(g)$  also vanishes (by first taking conditional
expectation of  $\xi$  conditioned on  $X$ ). Thus, we arrive at
$$
\begin{array}{l} \mathbb {E} \left[ \langle \Theta , \hat {\Theta} - \Psi_ {\mathcal {G}} (\hat {\Theta}) \rangle_ {\mu} \right] = \| \Theta \| _ {\mu} ^ {2} - \mathbb {E} _ {x} \left[ \langle \Theta^ {\top} x, \int_ {\mathcal {G}} \psi (g ^ {- 1}) \Theta^ {\top} \phi (g) x d \lambda (g) \rangle \right] \\ = \| \Theta \| _ {\mu} ^ {2} - \langle \Theta , \Psi_ {\mathcal {G}} (\Theta) \rangle_ {\mu} \\ = \| \Psi_ {\mathcal {G}} ^ {\perp} (\Theta) \| _ {\mu} ^ {2} \\ = - 2 \sigma_ {X} ^ {2} \| \Psi_ {\mathcal {G}} ^ {\perp} (\Theta) \| _ {F} ^ {2}, \tag {30} \\ \end{array}
$$
where the last equality follows from Proposition 11 in [8] with the assumption that  $\Sigma = \sigma_X^2$ . This finishes the computation for the mismatch term.
Now for the constraint term, we have
$$
\begin{array}{l} \left\| f _ {\mathcal {G}} ^ {\perp} \right\| _ {\mu} ^ {2} = \left\| \Psi_ {\mathcal {G}} ^ {\perp} (\hat {\Theta}) \right\| _ {\mu} ^ {2} (31) \\ = \sigma_ {X} ^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {\xi}} \| \Psi_ {\mathcal {G}} ^ {\perp} (\Theta + (\boldsymbol {X} ^ {\top} \boldsymbol {X}) ^ {- 1} \boldsymbol {X} ^ {\top} \boldsymbol {\xi}) \| ^ {2} (32) \\ = \sigma_ {X} ^ {2} \| \Psi_ {\mathcal {G}} ^ {\perp} (\Theta) \| _ {F} ^ {2} + \sigma_ {X} ^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {\xi}} \| \Psi_ {\mathcal {G}} ^ {\perp} ((\boldsymbol {X} ^ {\top} \boldsymbol {X}) ^ {- 1} \boldsymbol {X} ^ {\top} \boldsymbol {\xi}) \| ^ {2}, (33) \\ \end{array}
$$
where the last equality follows from linearity of expectation,  $\mathbb{E}[\pmb {\xi}] = 0$  and  $\pmb{\xi}$  independent of  $x$ .
Combining the mismatch term in (30) with the constraint term in (33), the risk gap becomes
$$
\mathbb {E} \left[ \Delta \left(f _ {\hat {\Theta}}, f _ {\Psi_ {\mathcal {G}} (\hat {\Theta})}\right) \right] = - \sigma_ {X} ^ {2} \| \Psi_ {\mathcal {G} _ {L}} ^ {\perp} (\Theta) \| ^ {2} + \sigma_ {X} ^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {\xi}} \| \Psi_ {\mathcal {G} _ {L}} ^ {\perp} \left((\boldsymbol {X} ^ {\top} \boldsymbol {X}) ^ {- 1} \boldsymbol {X} ^ {\top} \boldsymbol {\xi}\right) \| ^ {2}, \tag {34}
$$
Applying Theorem 13 in [8], the second term in (34) reduces to
$$
\sigma_ {X} ^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {\xi}} \| \Psi_ {\mathcal {G} _ {L}} ^ {\perp} \left((\mathbf {X} ^ {\top} \mathbf {X}) ^ {- 1} \mathbf {X} ^ {\top} \boldsymbol {\xi}\right) \| ^ {2} = \sigma_ {\xi} ^ {2} \frac {N ^ {2} d k - \left(\chi_ {\psi | \mathcal {G}} \mid \chi_ {\phi | \mathcal {G}}\right)}{n - N d - 1}, \tag {35}
$$
from which the theorem follows immediately.
Finally, we state a well-known result for the risk of (Ordinary) Least-Squares Estimator (see [84, 85] and references therein).
Lemma 6 (Risk of Least-Squares Estimator). Consider the same set-up as Theorem 2. For  $n > Nd + 1$ ,
$$
\mathbb {E} \left[ \| Y - \hat {\Theta} ^ {\top} X \| _ {F} ^ {2} \right] = \sigma_ {\xi} ^ {2} \frac {N d}{n - N d - 1} + \sigma_ {\xi} ^ {2}.
$$
Proof of Lemma 6. Recall  $X, Y$  denote the test sample. We denote the risk of the least-squares estimator conditional on the training data  $\mathbf{X} \in \mathbb{R}^{n \times N_d}$  as  $\mathcal{R}(\hat{\Theta} \mid \mathbf{X})$ , which has the following bias-variance decomposition:
$$
\begin{array}{l} \mathcal {R} (\hat {\Theta} \mid \boldsymbol {X}) = \mathbb {E} \left[ \| Y - \hat {\Theta} ^ {\top} X \| _ {F} ^ {2} \mid \boldsymbol {X} \right] (36) \\ = \mathbb {E} \left[ \| \Theta^ {\top} X + \xi - \hat {\Theta} ^ {\top} X \| _ {F} ^ {2} \mid X \right] (37) \\ = \mathbb {E} \left[ \| (\Theta - \hat {\Theta}) ^ {\top} X \| _ {F} ^ {2} \mid X \right] + \sigma_ {\xi} ^ {2}, (38) \\ \end{array}
$$
where the last equality follows from  $\xi$  being zero mean and independent with  $X$ . The second term  $\sigma_{\xi}^{2}$  is also known as irreducible error. We decompose the first term into
$$
\mathbb {E} \left[ \| (\Theta - \hat {\Theta}) ^ {\top} X \| _ {F} ^ {2} \mid \boldsymbol {X} \right] = \mathbb {E} \left[ \| (\Theta - \mathbb {E} [ \hat {\Theta} ]) ^ {\top} X \| _ {F} ^ {2} + \| (\mathbb {E} [ \hat {\Theta} ] - \hat {\Theta}) ^ {\top} X \| _ {F} ^ {2} \mid \boldsymbol {X} \right]. \tag {39}
$$
Recall that  $\hat{\Theta} \stackrel{a.e.}{=} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} = (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} (\mathbf{X} \Theta + \xi) = \Theta + (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \xi$ . Thus  $\mathbb{E}[\hat{\Theta}] = \Theta$  and (39) simplifies to  $\mathbb{E}\left[\|(\mathbb{E}[\hat{\Theta}] - \hat{\Theta})^{\top} X\|_F^2 \mid \mathbf{X}\right]$ .
We finish computing the risk by taking expectation over  $X$ , and using  $\mathbb{E}[\hat{\Theta} ] - \hat{\Theta} = (X^{\top}X)^{-1}X^{\top}\xi$
$$
\begin{array}{l} \mathbb {E} \left[ \| Y - \hat {\Theta} ^ {\top} X \| _ {F} ^ {2} \right] = \mathbb {E} \left[ \mathcal {R} (\hat {\Theta} \mid \boldsymbol {X}) \right] (40) \\ = \mathbb {E} _ {\boldsymbol {X}} \left[ \mathbb {E} _ {X, \xi} \left[ \| (\mathbb {E} [ \hat {\Theta} ] - \hat {\Theta}) ^ {\top} X \| _ {F} ^ {2} \mid \boldsymbol {X} \right] \right] + \sigma_ {\xi} ^ {2} (41) \\ = \mathbb {E} \left[ \left\| \left(\left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1} \boldsymbol {X} ^ {\top} \xi\right) ^ {\top} X \right\| _ {F} ^ {2} \right] + \sigma_ {\xi} ^ {2} (42) \\ = \sigma_ {\xi} ^ {2} \operatorname {t r} \left(\mathbb {E} \left[ \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {- 1} \right] \sigma_ {X} ^ {2} I\right) + \sigma_ {\xi} ^ {2}. (43) \\ \end{array}
$$
By [86, Lemma 2.3], for  $n > Nd + 1$ ,  $\mathbb{E}[(\boldsymbol{X}^\top \boldsymbol{X})^{-1}] = \frac{Nd}{n - Nd - 1} I$ . Putting this in (43) completes the proof.
B.2 Proofs of Generalization with Approximate Symmetries
In Definition 3, we construct the symmetry group of  $G$  induced by the coarsening  $G'$  via a semidirect product,
$$
\mathcal {G} _ {G \rightarrow G ^ {\prime}} = \left(\mathcal {S} _ {c _ {1}} \times \mathcal {S} _ {c _ {2}} \dots \times \mathcal {S} _ {c _ {M}}\right) \rtimes \overline {{\mathcal {A}}} _ {G ^ {\prime}}.
$$
We explain the construction in more details here. We first recall the definition of semidirect product. Given two groups  $\mathcal{G}_1, \mathcal{G}_2$  and a group homomorphism  $\varphi: \mathcal{G}_2 \to \operatorname{aut}(\mathcal{G}_1)$ , we can construct a new group  $\mathcal{G}_1 \rtimes_{\varphi} \mathcal{G}_2$ , called the semidirect product of  $\mathcal{G}_1, \mathcal{G}_2$  with respect to  $\varphi$  as follows:
1. The underlying set is the Cartesian product  $\mathcal{G}_1\times \mathcal{G}_2$
2. The group operation  $\circ$  is determined by the homomorphism  $\varphi$ , such that
$$
\begin{array}{l} \circ : \left(\mathcal {G} _ {1} \rtimes_ {\varphi} \mathcal {G} _ {2}\right) \times \left(\mathcal {G} _ {1} \rtimes_ {\varphi} \mathcal {G} _ {2}\right)\rightarrow \left(\mathcal {G} _ {1} \rtimes_ {\varphi} \mathcal {G} _ {2}\right) \\ (g _ {1}, g _ {2}) \circ (g _ {1} ^ {\prime}, g _ {2} ^ {\prime}) = (g _ {1} \varphi_ {g _ {2}} (g _ {1} ^ {\prime}), g _ {2} g _ {2} ^ {\prime}), \quad g _ {1}, g _ {1} ^ {\prime} \in \mathcal {G} _ {1}; g _ {2}, g _ {2} ^ {\prime} \in \mathcal {G} _ {2}, \\ \end{array}
$$
Take  $\mathcal{G}_1 = \left(\mathcal{S}_{c_1} \times \mathcal{S}_{c_2} \dots \times \mathcal{S}_{c_M}\right)$ ,  $\mathcal{G}_2 = \overline{\mathcal{A}}_{G'}$ . Note that  $\mathcal{G}_1$  is a normal subgroup in  $\mathcal{G}_{G \to G'}$ ; Namely, for all  $s \in \mathcal{G}_{G \to G'}$ ,  $g_1 \in \mathcal{G}_1$ , we have  $sg_1 s^{-1} \in \mathcal{G}_1$ . Thus, the map  $g_1 \mapsto sg_1 s^{-1}$  is an automorphism of  $\mathcal{G}_1$ . In particular, the homomorphism  $\varphi_{g_2}(g_1) = g_2 g_1 g_2^{-1}$  for  $g_2 \in \mathcal{G}_2$  describes the action of  $\mathcal{G}_2$  on  $\mathcal{G}_1$  by conjugation. In the context of a graph  $G$  with  $N$  nodes and its coarsening  $G'$  with  $M$  clusters, the homomorphism  $\varphi$  describes how across-cluster permutations in  $G'$  act on the original graph  $G$ . Note that in the special case where  $\mathcal{A}_{G'}$  acts transitively on the coarsened nodes, we recover the wreath product — a special kind of semidirect product. Our construction of  $\mathcal{G}_{G \to G'}$  can be seen as a natural generalization of the wreath product.
$$
\Delta (f, \bar {f} _ {\mathcal {G} _ {G \to G ^ {\prime}}}) = \underbrace {- 2 \langle f ^ {*} , f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} \rangle_ {\mu}} _ {\text {m i s m a t c h}} + \underbrace {\left\| f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} \right\| _ {\mu} ^ {2}} _ {\text {c o n s t r a i n t}} \geq - 2 \kappa (\epsilon) \left\| f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} \right\| _ {\mu} + \left\| f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} \right\| _ {\mu} ^ {2}
$$
In our context, it is sensible to write  $\varphi_{g_2}(g_1) = g_2g_1g_2^{-1}$  given that  $\mathcal{G}_1,\mathcal{G}_2$  are originally inside a common group  $S_N$ . Yet semidirect product applies to two arbitrary groups — not necessarily inside a common group initially, where  $\varphi_{g_2}(g_1)$  is an abstraction of  $g_{2}g_{1}g_{2}^{-1}$ .
Proof of Corollary 3. We start by simplifying the mismatch term in Lemma 1,
$$
\begin{array}{l} - 2 \mathbb {E} \left[ \langle f ^ {*} (x), f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} (x) \rangle \right] = - 2 \mathbb {E} \left[ \langle f ^ {*} (x) - f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {*} (x) + f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {*} (x), f _ {\mathcal {G} _ {G \to G ^ {\prime}}} ^ {\perp} (x) \rangle \right] \\ = - 2 \mathbb {E} \left[ \underbrace {\left\langle \underbrace {f ^ {*} (x) - f _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} ^ {*} (x)} , \underbrace {f _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} ^ {\perp} (x)} \right\rangle} _ {\mathcal {G} _ {L} \text {- a n t i - s y m m e t r i c p a r t o f} f ^ {*}} \right. \\ \geq - 2 \left\| f ^ {*} - f _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} ^ {*} \right\| _ {\mu} \left\| f _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} ^ {\perp} \right\| _ {\mu} \quad (\text {B y C a u c h y S h e w z I n e q .}) \\ \geq - 2 \kappa (\epsilon) \| f _ {\mathcal {G} _ {G \rightarrow G ^ {\prime}}} ^ {\perp} \| _ {\mu}. \quad (B y D e f i n t i o n 4 A p p r o x. E q u i v. M a p) \\ \end{array}
$$
Putting this together with the constraint term completes the proof.
Corollary 4 (Bias-Variance-Tradeoff via Graph Coarsening). Consider the same linear regression setting in Theorem 2, except now  $f^{*}$  is an approximately equivariant mapping with equivariance rate  $\kappa$ , and  $\mathcal{G} = \mathcal{G}_{G\to G^{\prime}}$  is controlled by  $G^{\prime}$  that coarsens  $G$  up to error  $\epsilon$ . Denote the canonical permutation representations of  $\mathcal{G}_{G\to G^{\prime}}$  on  $\mathcal{X}, \mathcal{Y}$  as  $\phi^{\prime}, \psi^{\prime}$ , respectively. Let  $(\chi_{\psi^{\prime}}\mid \chi_{\phi^{\prime}}) = \int_{\mathcal{G}_{G\to G^{\prime}}}\chi_{\psi^{\prime}}(g)\chi_{\phi^{\prime}}(g)\mathrm{d}\lambda (g)$  denote the inner product of the characters. If  $n > Nd + 1$  the risk gap is bounded by
$$
\mathbb {E} \left[ \Delta \left(f _ {\hat {\Theta}}, f _ {\Psi_ {\mathcal {G} _ {G \to G ^ {\prime}} (\hat {\Theta})}}\right) \right] \geq - 2 \kappa (\epsilon) \sqrt {\sigma_ {\xi} ^ {2} \frac {N ^ {2} d k - (\chi_ {\psi^ {\prime}} \mid \chi_ {\psi^ {\prime}})}{n - N d - 1}} + \sigma_ {\xi} ^ {2} \frac {N ^ {2} d k - (\chi_ {\psi^ {\prime}} \mid \chi_ {\psi^ {\prime}})}{n - N d - 1}.
$$