SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma E.1 (Adaptive coordinate selection). Given  $k, m \in \mathbb{N}$ , consider a ReLU CNN model with depth  $L = \log_2(4d)$  and the channel numbers satisfying  $C_l = 2k$  for all  $l = 1, \ldots, L - 1$  and  $C_L = m$ . Then for any  $\mathcal{I} = (i_1, \dots, i_k) \subset [d]$  with  $1 \leqslant i_1 < \dots < i_k \leqslant d$ ,  $\mathbf{u}_1, \dots, \mathbf{u}_m \in \mathbb{R}^k$ , and  $c_1, \dots, c_m \in \mathbb{R}$ , there exists  $\theta \in \Theta$  such that the CNN  $h_\theta$  outputs:
$$
\mathbf {z} ^ {(L)} (\mathbf {x}) = \left(\sigma \left(\mathbf {u} _ {1} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {1}\right), \dots , \sigma \left(\mathbf {u} _ {m} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {m}\right)\right) \in \mathbb {R} ^ {1 \times m}.
$$
Furthermore, for this CNN, the number of parameters is  $\mathcal{O}(k^2\log d + km)$  and the parameter norm satisfies
$$
\| \theta \| _ {\mathcal {P}} \lesssim \sqrt {k} \log d + \sqrt {\sum_ {i = 1} ^ {m} \| \mathbf {u} _ {i} \| _ {2} ^ {2}} + \sqrt {\sum_ {i = 1} ^ {m} c _ {i} ^ {2}} + \| W _ {o} \| _ {2}
$$
In this case, the proof is the same as that of linear CNNs (Lemma 4.2) since we only need to double the channels: every two channels form a group, storing  $\{\sigma(x_i), \sigma(-x_i)\}_{i=1}^k$ ; different groups of channels proceed still independently like the case of linear CNNs. Specifically, for  $l = 1, 2, \ldots, L-1$ , we follow a similar idea to set weights and biases such that
$$
\mathbf {z} ^ {(L - 1)} (\mathbf {x}) = \left( \begin{array}{c} \sigma (x _ {t _ {1}}), \sigma (- x _ {t _ {1}}), \sigma (x _ {t _ {2}}), \sigma (- x _ {t _ {2}}), \ldots , \sigma (x _ {t _ {k}}), \sigma (- x _ {t _ {k}}) \\ \sigma (x _ {s _ {1}}), \sigma (- x _ {s _ {1}}), \sigma (x _ {s _ {2}}), \sigma (- x _ {s _ {2}}), \ldots , \sigma (x _ {s _ {k}}), \sigma (- x _ {s _ {k}}) \end{array} \right) \in \mathbb {R} ^ {2 \times 2 k},
$$
where for any  $j \in [k]$ , either  $t_j = i_j$  or  $s_j = i_j$ . Then, we set the weights and bias of  $L$ -th layer according to  $\{\mathbf{u}_i\}_{i=1}^m$  and  $\{c_i\}_{i=1}^m$  such that the outputs are  $\{\sigma(\mathbf{u}_i^\top \mathbf{x}_{\mathcal{I}} + c_i)\}_{i=1}^m$ . We refer to the following proof for details.
Proof. For any  $j \in [k]$ , we write the  $i_j$ 's binary representation as follows
$$
i _ {j} - 1 = \sum_ {l = 0} ^ {L - 1} a _ {j, l} 2 ^ {l} \tag {77}
$$
where  $a_{j,l} \in \{0,1\}$ . Here, “ $-1$ ” is used to ensure the  $i_j \in [0,2^L - 1]$ . Next, we set the weights and bias according to the binary representation (77).
Set weights and bias adaptively:
- For  $l = 1$ ,  $W^{(1)} \in \mathbb{R}^{1 \times 2k \times 2}$ . For  $j \in [k]$ , set
$$
W _ {2 j - 1, 1,:} ^ {(1)} = - W _ {2 j, 1,:} ^ {(1)} = (1 - a _ {j, 0}, a _ {j, 0}).
$$
- For  $l = 2, \ldots, L - 1$ ,  $W^{(l)} \in \mathbb{R}^{2k \times 2k \times 2}$ . For  $j \in [k]$ , set
$$
W _ {2 j - 1, 2 j - 1,:} ^ {(l)} = W _ {2 j, 2 j,:} ^ {(l)} = - W _ {2 j - 1, 2 j,:} ^ {(l)} = - W _ {2 j, 2 j - 1,:} ^ {(l)} = (1 - a _ {j, l - 1}, a _ {j, l - 1}).
$$
- For any  $l \in [L - 1]$ , set all entries in  $W^{(l)}$  unmentioned above and  $\mathbf{b}^{(l)}$  to zeros.
Note that  $a_{j,l} \in \{0,1\}$  for all  $j \in [k]$  and  $l \in [L]$ . Thus, under the above construction, all nonzero filters are taken from  $\{(1,0), (-1,0), (0,1), (0,-1)\}$ .
The forward selection process. Let us compute  $\mathcal{T}_l\mathbf{z}^{(l - 1)}$  for  $l = 1,\dots ,L - 1$ . Let  $s_{j,l} = 1 + \sum_{p = 0}^{l - 1}a_{j,p}2^p$ . We claim that for  $l\in [L - 1]$
$$
(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}) _ {:, j} = \left\{ \begin{array}{l l} (x _ {s _ {j ^ {\prime}}, l}, x _ {2 ^ {l} + s _ {j ^ {\prime}, l}}, \dots , x _ {4 d - 2 ^ {l} + s _ {j ^ {\prime}, l}}) & \text {i f j i s o d d}, \\ - (\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}) _ {:, j - 1} & \text {i f j i s e v e n} \end{array} \right.
$$
We will verify this by induction.
For  $l = 1$
$$
(\mathcal {T} _ {1} \mathbf {z} ^ {(0)}) _ {:, j} = (\mathcal {T} _ {1} (\mathbf {x})) _ {:, j} = \mathbf {x} * _ {s} W _ {j, 1,:} ^ {(1)}
$$
So when  $j$  is even  $(\mathcal{T}_1\mathbf{z}^{(0)})_{:,j} = -(\mathcal{T}_1\mathbf{z}^{(0)})_{:,j-1}$  and when  $j$  is odd we have
$$
\left(\mathcal {T} _ {1} \mathbf {z} ^ {(0)}\right) _ {:, j} = \left(x _ {1 + a _ {j ^ {\prime}, 0}}, x _ {3 + a _ {j ^ {\prime}, 0}}, \dots , x _ {4 d - 1 + a _ {j ^ {\prime}, 0}}\right)
$$
where  $j' = (j + 1) / 2$  as above.  $l = 1$  has been verified.
Then, assume that the case of  $1,\dots ,l - 1$  hold, we compute  $(\mathcal{T}_l\mathbf{z}^{(l - 1)})_{:,j}$ . Without loss of generality, we only consider the case where  $j$  is odd. Observing that
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} ^ {(l - 1)} * _ {s} W _ {j, i,:} ^ {(l)} + b _ {j} ^ {(l)} \\ = \mathbf {z} _ {:, j} ^ {(l - 1)} * _ {s} W _ {j, j,:} ^ {(l)} + \mathbf {z} _ {:, j + 1} ^ {(l - 1)} * _ {s} W _ {j, j + 1,:} ^ {(l)} \\ \end{array}
$$
$$
\begin{array}{l} (\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}) _ {i, j} = (1 - a _ {j ^ {\prime}, l - 1}) \sigma (x _ {2 ^ {l - 1} (2 i - 2) + s _ {j ^ {\prime}, l - 1}}) + a _ {j ^ {\prime}, l - 1} \sigma (x _ {2 ^ {l - 1} (2 i - 1) + s _ {j ^ {\prime}, l - 1}}) \\ - \left[ (1 - a _ {j ^ {\prime}, l - 1}) \sigma (- x _ {2 ^ {l - 1} (2 i - 2) + s _ {j ^ {\prime}, l - 1}}) + a _ {j ^ {\prime}, l - 1} \sigma (- x _ {2 ^ {l - 1} (2 i - 1) + s _ {j ^ {\prime}, l - 1}}) \right]. \\ = x _ {2 ^ {l}} (i - 1) + s _ {j ^ {\prime}, l - 1} + a _ {j ^ {\prime}, l - 1} 2 ^ {l - 1} \\ = x _ {(i - 1) 2 ^ {l} + s _ {j ^ {\prime}, l}} \\ \end{array}
$$
Thus, the claim is verified and after  $L - 1$  layer,
$$
\mathbf {z} _ {1,:} ^ {(L - 1)} = \left(\sigma \left(x _ {s _ {1}, L - 1}\right), \sigma \left(- x _ {s _ {1}, L - 1}\right), \dots , \sigma \left(x _ {s _ {k}, L - 1}\right), \sigma \left(- x _ {s _ {k}, L - 1}\right)\right)
$$
$$
\mathbf {z} _ {2,:} ^ {(L - 1)} = \left(\sigma \left(x _ {2 ^ {L - 1} + s _ {1, L - 1}}\right), \sigma \left(- x _ {2 ^ {L - 1} + s _ {1, L - 1}}\right), \dots , \sigma \left(x _ {2 ^ {L - 1} + s _ {k, L - 1}}\right), \sigma \left(- x _ {2 ^ {L - 1} + s _ {k, L - 1}}\right)\right)
$$
For the  $L$ -th layer, for  $j = 1,2,\ldots ,k$  set
$$
W _ {2 i - 1, j,:} ^ {(L)} = - W _ {2 i, j,:} ^ {(L)} = \left(\mathbf {u} _ {j}\right) _ {i} \cdot \left(1 - a _ {i, L - 1}, a _ {i, L - 1}\right)
$$
$$
b _ {j} ^ {(L)} = c _ {j}
$$
$$
\begin{array}{l} \left(\mathcal {T} _ {L} \mathbf {z} ^ {(L - 1)}\right) _ {:, j} = \sum_ {i = 1} ^ {2 k} \mathbf {z} _ {:, i} ^ {(L - 1)} * _ {s} W _ {j, i,:} ^ {(L)} + b _ {j} ^ {(L)} \\ = \sum_ {p = 1} ^ {k} (\mathbf {u} _ {j}) _ {p} (\sigma (x _ {i _ {p}}) - \sigma (- x _ {i _ {p}})) + c _ {j} \\ = \mathbf {u} _ {j} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {j}. \\ \end{array}
$$
Thus we complete the proof of the first part.
Bounding the parameter norm. Under the above construction, we have:
- When  $l = 1,2,\ldots ,L - 1$ , there exists an absolute constant  $C > 0$  such that  $\| W^{(l)}\|_F \leqslant C\sqrt{k}$  and  $\mathbf{b}^{(l)} = 0$ . Thus, by equation (52) we have
$$
\left\| W ^ {(l)} \right\| _ {F} + \sqrt {\frac {d}{2 ^ {l - 2}}} \left\| \mathbf {b} ^ {(l)} \right\| _ {2} \leqslant C \sqrt {k};
$$
- Furthermore, the parameter norm in  $L$ -th layer satisfies
$$
\left\| W ^ {(L)} \right\| _ {F} + \left\| \mathbf {b} ^ {(L)} \right\| _ {2} \lesssim \sqrt {\sum_ {i = 1} ^ {m} \| \mathbf {u} _ {i} \| _ {2} ^ {2}} + \sqrt {\sum_ {i = 1} ^ {m} c _ {i} ^ {2}}.
$$
Combining them, we complete the proof.
E.2 Proof of Theorem 4.6
We will need following approximation result of two-layer ReLU networks.
Lemma E.2. (A restatement of (E et al., 2022, Theorem 4)) For any  $f \in \mathcal{B}$ , any probability distribution  $P$  on  $\mathcal{X} = [0,1]^d$  and integer  $m \geqslant 1$ , there exists a two-layer neural network  $f_{m}(\mathbf{x};\theta) = \frac{1}{m}\sum_{k=1}^{m}a_{k}\sigma\left(\mathbf{u}_{k}^{\top}\mathbf{x} + c_{k}\right)$  where  $\theta$  denotes the parameters  $\{(a_{k},\mathbf{u}_{k},c_{k}), k \in [m]\}$  in the neural network, such that
$$
\left\| f - f _ {m} (\cdot ; \theta) \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {3 \| f \| _ {\mathcal {B}} ^ {2}}{m}, \tag {78}
$$
$$
\frac {1}{m} \sum_ {j = 1} ^ {m} | a _ {j} | \left(\left\| \mathbf {u} _ {j} \right\| _ {1} + \left| c _ {j} \right|\right) \leqslant 2 \| f \| _ {\mathcal {B}} \tag {79}
$$
Proposition E.3. Let  $\mathcal{X} = [0,1]^{4d}$ . Suppose  $h^*(\mathbf{x}) = g^*(\mathbf{x}_{\mathcal{I}})$  with  $|\mathcal{I}| = k$ . Consider deep ReLU CNNs with  $L = \log_2(4d)$  and  $C_l = 2k$  for  $l \in [L - 1]$  and  $C_L = m$ . Then, for any  $g^* \in \mathcal{B}$ , there exist  $\theta^*$  such that
$$
\left\| h _ {\theta^ {*}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {3 \| g ^ {*} \| _ {\mathcal {B}} ^ {2}}{m} \tag {80}
$$
with the number of parameters being  $\mathcal{O}(k^2\log d + km)$  and
$$
\left\| \theta^ {*} \right\| _ {\mathcal {P}} \lesssim \sqrt {k} \log d + m + \left\| g ^ {*} \right\| _ {\mathcal {B}} \tag {81}
$$
Proof. Using lemma E.2, there exists a two layer ReLU network  $f_{m}(\mathbf{x}_{\mathcal{I}}) = \frac{1}{m}\sum_{k=1}^{m}a_{k}\sigma\left(\mathbf{u}_{k}^{\top}\mathbf{x}_{\mathcal{I}} + c_{k}\right)$  such that
$$
\begin{array}{l} \left\| h ^ {*} (\mathbf {x}) - f _ {m} (\mathbf {x} _ {\mathcal {I}}) \right\| _ {L ^ {2} (\mu)} ^ {2} = \left\| g ^ {*} (\mathbf {x} _ {\mathcal {I}}) - f _ {m} (\mathbf {x} _ {\mathcal {I}}) \right\| _ {L ^ {2} (\mu)} ^ {2} \leqslant \frac {3 \left\| g ^ {*} \right\| _ {\mathcal {B}} ^ {2}}{m} \\ \left\| \mathbf {u} _ {k} \right\| _ {1} + \left| c _ {k} \right| = 1, \forall k \tag {82} \\ \frac {1}{m} \sum_ {k = 1} ^ {m} | a _ {k} | \leqslant 2 \| g ^ {*} \| _ {\mathcal {B}} \\ \end{array}
$$
By Lemma 4.4, we can choose the weights and bias such that the feature map of  $L$ -th layer is
$$
\mathbf {z} ^ {(L)} (\mathbf {x}) = \left(\sigma \left(\mathbf {u} _ {1} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {1}\right), \dots , \sigma \left(\mathbf {u} _ {m} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {m}\right)\right) ^ {\top} \tag {83}
$$
By setting the parameters of output layer as  $W_{o} = \frac{1}{m} (a_{1},\ldots ,a_{m})$ , it is easy to check that  $h_{\theta^{*}}(\mathbf{x}) = \mathcal{M}_{o}\circ \mathbf{z}^{(L)}(\mathbf{x}) = f_{m}(\mathbf{x}_{\mathcal{I}})$  and the parameter number  $N_{\mathrm{cnn}} = \mathcal{O}(k^2\log d + km)$ .
Next, we turn to bound the parameter norm. Under the above parameter setting, we have
$$
\begin{array}{l} \left\| W _ {o} \right\| _ {2} \leqslant \left\| W _ {o} \right\| _ {1} \leqslant \frac {1}{m} \sum_ {k = 1} ^ {m} \left| a _ {k} \right| \leqslant 2 \left\| g ^ {*} \right\| _ {\mathcal {B}} \\ \sqrt {\sum_ {i = 1} ^ {m} \left\| \mathbf {u} _ {i} \right\| _ {2} ^ {2}} + \sqrt {\sum_ {i = 1} ^ {m} c _ {i} ^ {2}} \leqslant \sum_ {i = 1} ^ {m} \left(\left\| \mathbf {u} _ {i} \right\| _ {1} + \left| c _ {i} \right|\right) = m. \\ \end{array}
$$
Applying Lemma 4.4 gives
$$
\begin{array}{l} \left\| \theta^ {*} \right\| _ {\mathcal {P}} \lesssim \left\| g ^ {*} \right\| _ {\mathcal {B}} + \sqrt {k} \log d + \sqrt {\sum_ {i = 1} ^ {m} \left\| \mathbf {u} _ {i} \right\| _ {2} ^ {2}} + \sqrt {\sum_ {i = 1} ^ {m} c _ {i} ^ {2}} \tag {84} \\ \lesssim \left\| g ^ {*} \right\| _ {\mathcal {B}} + \sqrt {k} \log d + m \\ \end{array}
$$
Proof of Theorem 4.6. Let  $\mathcal{H}_J^{\mathrm{CNN}} = \{h_\theta : \| \theta \| \leqslant J\}$ . By proposition B.14 and the covering number satisfies
$$
\mathcal {N} \left(\mathcal {H} _ {J} ^ {\mathrm {C N N}}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {\gamma} _ {n} (J)}{t}\right) ^ {N _ {\mathrm {c n n}}} \tag {85}
$$
where  $\hat{\gamma}_n(J) = \hat{M}_n J(1 + J)^L$  and  $N_{\mathrm{conn}} = \mathcal{O}(k^2\log d + km)$ . Here we use the fact that  $\bar{Q}_{\sigma}(\mathbf{x})\leqslant 2^{L}$  for all  $\mathbf{x}$  and  $L = \log_2d + 2$  for deep ReLU CNNs. In addition, note that
$$
\begin{array}{l} \gamma (J) = \mathbb {E} [ \hat {\gamma} _ {n} (J) ] \leqslant 2 ^ {L} J (1 + J) ^ {L} \mathbb {E} \left[ \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} (\| \mathbf {x} _ {i} \| + 1) ^ {2}} \right] \\ \leqslant 2 ^ {L} J (1 + J) ^ {L} \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ (\| \mathbf {x} _ {i} \| + 1) ^ {2} ]} \lesssim \sqrt {d} 2 ^ {L} J (1 + J) ^ {L} \tag {86} \\ \end{array}
$$
where the second inequality follows from the Jensen's inequality since  $\sqrt{\cdot}$  is concave.
Recalling we have  $\alpha_{\mathcal{H}} = 1$  from equation (53) and applying Proposition B.14, we have
$$
\left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon_ {*} \lesssim \frac {\sigma^ {3} B}{(B - 2 A) ^ {2}} e ^ {- \frac {(B - 2 A) ^ {2}}{2 \sigma^ {2}}} + \lambda M _ {*} + B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}} + B ^ {2} \sqrt {\frac {N _ {\mathrm {c n n}} \log (B \gamma (U _ {\lambda}))}{n}},
$$
$$
\epsilon_ {*} = \frac {3 \| g ^ {*} \| _ {\mathcal {B}} ^ {2}}{m}, \qquad M _ {*} \lesssim \sqrt {k} \log d + m + \| g ^ {*} \| _ {\mathcal {B}},
$$
Taking  $A = 1, B = 2 + \sigma \sqrt{\log n}, \lambda = \epsilon / n^2$ ,  $m = \frac{6\|g^{*}\|_{B}^{2}}{\epsilon}$  and applying Lemma B.13, we have
$$
\begin{array}{l} \left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon / 2 \lesssim \frac {\sigma^ {2}}{\sqrt {n}} + \epsilon \frac {\sqrt {k} \log d + \frac {\| g ^ {*} \| _ {\mathcal {B}} ^ {2}}{\epsilon} + \| g ^ {*} \| _ {\mathcal {B}}}{n ^ {2}} + B ^ {2} \sqrt {\frac {\log (1 / \delta)}{n}} \\ + B ^ {2} \sqrt {\frac {\left(k ^ {2} \log d + k \frac {\| g ^ {*} \| _ {\mathbf {B}} ^ {2}}{\epsilon}\right) (\log (B) + \log (\gamma (U _ {\lambda})))}{n}}, \tag {87} \\ \end{array}
$$
$$
\log (\gamma (U _ {\lambda})) \lesssim \log (d) \log \left(\frac {1}{2 \lambda} \left(1 + \sigma^ {2} + B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}}\right) + \sqrt {k} \log d + \frac {\| g ^ {*} \| _ {\mathcal {B}} ^ {2}}{\epsilon} + \| g ^ {*} \| _ {\mathcal {B}}\right).
$$
Simplify the right side of equation (87) becomes
$$
\left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon / 2 \lesssim
$$
$$
\mathrm {p o l y} (\| g ^ {*} \| _ {\mathcal {B}}, k, \sigma , \log (1 / \delta), \log \log n, \log (1 / \epsilon)) \frac {\log n}{\sqrt {n}} \sqrt {\log (d) \left(\log d + \frac {1}{\epsilon}\right) (\log n + \log \log d)}
$$
This implies that, for a target accuracy  $\epsilon$  and failure probability  $\delta$ ,
$$
n \geqslant \mathrm {p o l y} (\| g ^ {*} \| _ {\mathcal {B}}, k, \sigma , \log \frac {1}{\delta}, \log \frac {1}{\epsilon}) \left(\frac {\log (d) (\log \log d) ^ {3}}{\epsilon^ {3}} + \frac {\log^ {2} (d) (\log \log d) ^ {3}}{\epsilon^ {2}}\right).
$$
is enough.
F Symmetries of Learning Algorithm and Proofs of Lemma 5.3 and 5.6
F.1 Learning Algorithm and Group Equivariance
(Li et al., 2020, Appendix C) established a framework to verify the group equivariance of an iterative algorithm, which however considers neither stochastic nor adaptive algorithms such as Adam. In this
section, we provide an extension of (Li et al., 2020, Appendix C) to include these situations. It should be stressed that the extension is mostly straightforward compared with (Li et al., 2020, Appendix C) and we provide the extension here only for clarity and completeness.
In the following, denote  $A \stackrel{\mathrm{d}}{=} B$  as  $\operatorname{Law}(A) = \operatorname{Law}(B)$ . Let  $G_{\mathcal{X}}$  be a group acting on  $\mathcal{X}$ . Then, for any  $\tau \in G_{\mathcal{X}}$  and  $S_n = \{\mathbf{x}_i, y_i\}_{i=1}^n$ , let  $\tau(S_n) = \{(\tau(\mathbf{x}_i), y_i)\}_{i=1}^n$ . We also occasionally write the group action  $\tau\theta = \tau(\theta)$  for simplicity when it is clear from the context.
Recall that  $h_\theta$  is our parametric model and  $\theta \in \Theta = \mathbb{R}^p$ . We make the following assumption.
Assumption 3. Given a group  $G_{\mathcal{X}}$  acting on  $\mathcal{X}$ . We assume that there exist a group  $G_{\Theta}$  acting on  $\Theta$  and a group isomorphism  $Q: G_{\mathcal{X}} \to G_{\Theta}$  such that for all  $\tau \in G_{\mathcal{X}}$ ,
$$
h _ {Q (\tau) (\theta)} \circ \tau = h _ {\theta}.
$$
The above assumption is satisfied by both FCNs and LCNs and we will verify it later. Here, we take linear regression as an example to gain some intuition. In such a case,  $h_{\theta}(\mathbf{x}) = \langle \mathbf{w}, \mathbf{x} \rangle$ . Given the group  $G_{\mathcal{X}} = O(d)$  acting on the input domain, we can set  $Q = \mathrm{id}$  and  $G_{\Theta} = O(d)$ . Then, for any  $U \in O(d)$ , we have  $h_{Q(U)(\theta)} \circ U(\mathbf{x}) = \langle U\mathbf{w}, U\mathbf{x} \rangle = \langle \mathbf{w}, \mathbf{x} \rangle = h_{\theta}(\mathbf{x})$ .
Suppose Assumption 3 holds. Given a learning algorithm  $\mathbb{A}:(\mathcal{X}\times \mathcal{Y})^n\mapsto \mathcal{M}(\Theta)$ , we say  $\mathbb{A}$  is  $G_{\mathcal{X}}$ -equivariant if  $\forall S_n\in (\mathcal{X}\times \mathcal{Y})^n$  and  $\tau \in G_{\mathcal{X}}$ :
$$
\mathbb {A} (\tau (S _ {n})) \stackrel {\mathrm {d}} {=} Q (\tau) \circ \mathbb {A} (S _ {n}). \tag {88}
$$
We can also informally define the equivariance in the model space as follows
$$
h _ {\mathbb {A} \left(\tau \left(S _ {n}\right)\right)} \circ \tau \stackrel {{\mathrm {d}}} {{=}} h _ {\mathbb {A} \left(S _ {n}\right)}, \tag {89}
$$
where  $h_{\mathbb{A}(S_n)}$  and  $h_{\mathbb{A}(\tau(S_n))}$  should be understood as random variables taking values in the space of all the algorithms  $\mathcal{A}$ . The definition (89) is intuitive and has been adopted in Abbe and Boix-Adserà (2022); Li et al. (2020) but it should be stressed that (89) is not rigorous as it is generally unclear how to deal with the measurability issue in  $\mathcal{A}$ , the space of algorithm. We thus will adopt (88) for its rigorous nature.
Iterative algorithm. In the following, we will focus on the learning algorithm given by
$$
\theta_ {0} \sim P _ {\mathrm {i n i t}}
$$
$$
\theta_ {t + 1} = F _ {t + 1} \left(\theta_ {t}, \dots , \theta_ {0}, S _ {n}, \xi_ {t + 1}\right) \text {f o r} t = 0, 1, \dots , T - 1, \tag {90}
$$
where  $F_{t}:(\Theta)^{t}\times (\mathcal{X}\times \mathcal{Y})^{n}\times \Omega \mapsto \Theta$  is a deterministic update map and  $\{\xi_t\}_{t = 1}^T$  are i.i.d. freshly generated random variables taking values in  $\Omega$ , which encode the algorithm randomness and  $\xi_{t}$  is independent of  $S_{n}$  and  $\{\theta_0,\theta_1,\dots ,\theta_t\}$ .
Take SGD as a concrete example. Let  $\Omega = \{0,1\}^n$  denote the set of minibatch selection masks and the corresponding algorithm map is given by
$$
F _ {t + 1} \left(\theta_ {t}, \dots , \theta_ {0}, S _ {n}, \xi_ {t + 1}\right) = \theta_ {t} - \eta_ {t} \nabla_ {\theta} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\xi_ {t + 1}\right) _ {i} \ell \left(h _ {\theta_ {t}} \left(\mathbf {x} _ {i}\right), y _ {i}\right) + \lambda r \left(\left\| \theta_ {t} \right\| _ {p}\right)\right).
$$
Note that one should not confuse  $\xi_{t}$  with the SGD noise, where latter is state-dependent.
Assumption 4.  $P_{\mathrm{init}}$  is  $G_{\Theta}$ -invariant.
$$
\cdot \forall \tau \in G _ {\mathcal {X}}, \theta \in \Theta , t \in \mathbb {N}, S _ {n} \in (\mathcal {X} \times \mathcal {Y}) ^ {n}, a n d \xi \in \Omega :
$$
$$
Q (\tau) \circ F _ {t + 1} \left(\theta_ {t}, \dots , \theta_ {0}, S _ {n}, \xi\right) = F _ {t + 1} \left(Q (\tau) \theta_ {t}, \dots , Q (\tau) \theta_ {0}, \tau \left(S _ {n}\right), \xi\right). \tag {91}
$$
The second assumption can be thought as that the update map  $F_{t}(\cdot)$  is equivariant under the joint group action  $(\tau, Q(\tau))$ . Still taking the linear regression as an example, let  $X = (\mathbf{x}_1, \ldots, \mathbf{x}_n) \in \mathbb{R}^{d \times n}$  be  $n$  inputs and  $\mathbf{y} \in \mathbb{R}^n$  be the labels. Then, the GD update for minimizing  $\hat{L}(\mathbf{w}) := \frac{1}{2} \|X^\top \mathbf{w} - \mathbf{y}\|_2^2$  would be
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} - \eta X (X ^ {\top} \mathbf {w} _ {t} - \mathbf {y}) =: F _ {t + 1} (\mathbf {w} _ {t}, S _ {n}).
$$
Let  $G_{\mathcal{X}} = G_{\Theta} = G_{\mathrm{ort}}(d)$ ,  $Q = \mathrm{id}$ . We thus have for any  $U \in O(d)$ ,  $Q(U) = U$  and
$$
\begin{array}{l} Q (U) F _ {t + 1} \left(\mathbf {w} _ {t}, S _ {n}\right) = U \left(\mathbf {w} _ {t} - \eta X \left(X ^ {\top} \mathbf {w} _ {t} - \mathbf {y}\right)\right) \\ = U \mathbf {w} _ {t} - \eta (U X) \left((U X) ^ {\top} (U \mathbf {w} _ {t}) - \mathbf {y}\right) = F _ {t + 1} (Q (U) \mathbf {w} _ {t}, U (S _ {n})), \\ \end{array}
$$
which verifies the equivariance (91).
Proposition F.1. Let  $\mathbb{A}_T$  denote the iterative algorithm (90). Under Assumption 3 and 4, we have for any  $T\in \mathbb{N}$ ,  $\mathbb{A}_T$  is  $G_{\mathcal{X}}$ -equivariant.
Proof. Fix  $S_{n}\in (\mathcal{X}\times \mathcal{Y})^{n}$  and  $\tau \in G_{\mathcal{X}}$ . Let  $\{\theta_t\}_{t = 0}^{\top}$  and  $\{\widetilde{\theta}_t\}_{t = 0}^{\top}$  be the trajectories independently generated by the iterative algorithm by using the data  $S_{n}$  and  $\tau (S_n)$ , respectively. By the definition (88), it suffices to prove that for any  $t\in \mathbb{N}$ ,
$$
\left(\widetilde {\theta} _ {t}, \dots , \widetilde {\theta} _ {0}\right) \stackrel {{\mathrm {d}}} {{=}} \left(Q (\tau) \theta_ {t}, \dots , Q (\tau) \theta_ {0}\right). \tag {92}
$$
Next we prove it by induction.
When  $T = 0$ ,  $\theta_0, \widetilde{\theta}_0 \sim P_{\mathrm{init}}$ . For any  $\tau \in G_{\mathcal{X}}$ , we have  $Q(\tau) \in G_{\Theta}$ . The assumption that  $P_{\mathrm{init}}$  is  $G_{\Theta}$ -invariant implies  $\widetilde{\theta}_0 \stackrel{\mathrm{d}}{=} Q(\tau)\theta_0$ .
Assume (92) holds for  $T = 0,1,\dots ,t$ . Let  $\{\xi_t\}$  and  $\{\widetilde{\xi}_t\}$  be the "noise" used to generate  $\{\theta_t\}$  and  $\{\widetilde{\theta}_t\}$ , respectively. Then, we have
$$
\begin{array}{l} \left(Q (\tau) \theta_ {t + 1}, \dots , Q (\tau) \theta_ {0}\right) \stackrel {{\mathrm {d}}} {{=}} \left(Q (\tau) F _ {t + 1} \left(\theta_ {t}, \dots , \theta_ {0}, S _ {n}, \xi_ {t + 1}\right), Q (\tau) \theta_ {t}, \dots , Q (\tau) \theta_ {0}\right) \\ \stackrel {{\mathrm {d}}} {{=}} \left(F _ {t + 1} (Q (\tau) \theta_ {t}, \dots , Q (\tau) \theta_ {0}, \tau \left(S _ {n}\right), \xi_ {t + 1}), Q (\tau) \theta_ {t}, \dots , Q (\tau) \theta_ {0}\right) \\ \stackrel {\mathrm {d}} {=} \left(F _ {t + 1} \left(\widetilde {\theta} _ {t}, \dots , \widetilde {\theta} _ {0}, \tau \left(S _ {n}\right), \widetilde {\xi} _ {t + 1}\right), \widetilde {\theta} _ {t}, \dots , \widetilde {\theta} _ {0}\right) \\ \stackrel {{\mathrm {d}}} {{=}} \left(\widetilde {\theta} _ {t + 1}, \dots , \widetilde {\theta} _ {0}\right), \\ \end{array}
$$
where the second step follows from Assumption (4); the third step follows from the assumption that (92) holds for  $T = t$ . Thus, we prove that (92) holds for  $T = t + 1$ .
By induction, we complete the proof.
F.2 Verifying the Group Equivariance of SGD and Adam
Let  $\ell(\cdot, \cdot)$  be a general loss function. Consider a regularized empirical risk
$$
\hat {L} _ {n} (\theta) := \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(h _ {\theta} \left(\mathbf {x} _ {i}\right), y _ {i}\right) + r \left(\left\| \theta \right\| _ {p}\right)
$$
where  $p\geqslant 1$  and and  $r:[0, + \infty)\to [0, + \infty)$
At the  $t$ -th step, let  $S_{t} = \{i_{1},\ldots ,i_{k}\}$  be  $k$  i.i.d. indices uniformly drawn from  $[n]$ . Define the minibatch risk by
$$
\hat {L} _ {S _ {t}} (\theta) := \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \ell (h _ {\theta} (\mathbf {x} _ {i}), y _ {i}) + r (\| \theta \| _ {p}).
$$
In each step, stochastic gradient descent (SGD) updates as follows
$$
\theta_ {t + 1} = \theta_ {t} - \eta_ {t} \nabla_ {\theta} \hat {L} _ {S _ {t}} (\theta_ {t}).
$$
Adam optimizer (Kingma and Ba, 2014) updates as follows
$$
\mathbf {v} _ {t + 1} = \alpha \mathbf {v} _ {t} + (1 - \alpha) (\nabla_ {\theta} \hat {L} _ {S _ {t}} (\theta_ {t})) ^ {2}
$$