SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {m} _ {t + 1} = \beta \mathbf {m} _ {t} + (1 - \beta) \nabla_ {\theta} \hat {L} _ {S _ {t}} (\theta_ {t}) \tag {93}
$$
$$
\theta_ {t + 1} = \theta_ {t} - \eta_ {t} \frac {\mathbf {m} _ {t + 1} / \left(1 - \beta^ {t + 1}\right)}{\sqrt {\mathbf {v} _ {t + 1} / \left(1 - \alpha^ {t + 1}\right)} + \epsilon \mathbf {1}},
$$
where  $\alpha, \beta \in (0,1), \epsilon > 0$ , and the square and division should be understood in an element-wise manner. We just consider initialization  $\mathbf{v}_0 = \mathbf{m}_0 = \mathbf{0}$  for simplicity. For both SGD and Adam,  $\eta_t$  is the learning rate at the  $t$ -th step.
FCNs: Proof of Lemma 5.6 For  $h_{\theta}^{\mathrm{fcn}}$ , write  $\theta = (\mathbf{w}_1, \dots, \mathbf{w}_m, \bar{\theta}) \in \mathbb{R}^p$ , where  $m$  is the width of the first layer,  $\mathbf{w}_j \in \mathbb{R}^d$  for  $j = 1, \dots, m$  denotes the weights of the first layer, and  $\bar{\theta}$  denotes other parameters. Then, we can write  $h_{\theta}^{\mathrm{fcn}}(\mathbf{x}) = g_{\bar{\theta}}(\mathbf{w}_1^\top \mathbf{x}, \dots, \mathbf{w}_m^\top \mathbf{x})$  for some  $g: \mathbb{R}^m \times \mathbb{R}^{p - md} \mapsto \mathbb{R}$ .
Consider  $G_{\mathcal{X}} = G_{\mathrm{ort}}(d)$  and
$$
Q: U \rightarrow \operatorname {d i a g} \{U, U, \dots , U, I _ {p - m d} \}. \tag {94}
$$
Let  $G_{\Theta} \coloneqq \{Q(U) : U \in G_{\mathcal{X}}\}$ . Then, it is not hard to verify that  $G_{\Theta}$  is a group under matrix product and  $Q: G_{\mathcal{X}} \mapsto G_{\Theta}$  is a group isomorphism.
First, it is not hard to verify that the Gaussian initialization  $P_{\mathrm{init}}$  is invariant under  $G_{\Theta}$ . To verify (91) for SGD, we can simply consider the case where  $n = 1$  and  $S_{n} = \{(\mathbf{x},y)\}$  without loss of generality. First, we can show that there exist functions  $\{s_i(\cdot)\}_{i=1}^m$  and  $O(\cdot)$  such that
$$
F _ {t + 1} (\theta_ {t}, \ldots , \theta_ {0}, S _ {n}, \xi_ {t + 1}) = \theta_ {t} - \eta_ {t} \nabla_ {\theta} \ell (h _ {\theta_ {t}} ^ {\mathrm {f c n}} (\mathbf {x}), y) = \theta_ {t} - \eta_ {t} \left( \begin{array}{c} s _ {1} (W ^ {\top} \mathbf {x}, \bar {\theta} _ {t}) \mathbf {x} \\ s _ {2} (W ^ {\top} \mathbf {x}, \bar {\theta} _ {t}) \mathbf {x} \\ \vdots \\ s _ {m} (W ^ {\top} \mathbf {x}, \bar {\theta} _ {t}) \mathbf {x} \\ O (W ^ {\top} \mathbf {x}, \bar {\theta} _ {t}) \end{array} \right),
$$
where  $W = (\mathbf{w}_1,\mathbf{w}_2,\dots ,\mathbf{w}_m)\in \mathbb{R}^{d\times m}$ . It is not hard to verify that (91) holds for the above update map. Then, Lemma 5.6 follows trivially from Proposition F.1.
LCNs: Proof of Lemma 5.3 Here we only prove Lemma 5.3 for the Adam optimizer since the SGD case is similar to the proof above.
WLOG, we let  $C_1 = 1$  in the LCN model. Denote by  $\bar{\theta} = W_{1,1,:}^{(1)}$  the filter weights of the first layer and  $\bar{\theta}^c$  all the other parameters. Then,  $\theta = \{\bar{\theta},\bar{\theta}^c\}$  and  $W_{1,1,:}^{(1)}\in \mathbb{R}^{D_0}$ . Let  $G_{\chi} = G_{\mathrm{loc}}$  and define
$$
Q: U \rightarrow \operatorname {d i a g} \{U, I \}. \tag {95}
$$
Let  $G_{\Theta} = \{Q(U):U\in G_{\mathcal{X}}\}$ . It is not hard to verify that  $G_{\Theta}$  is a group under the matrix multiplication and  $Q$  is the group isomorphism.
First, it is obvious that under Assumption 1,  $P_{\mathrm{init}}$  is  $G_{\mathrm{loc}}$ -invariant. To verify (91) for Adam, we can simply consider the case where  $r(\cdot) = 0$ ,  $n = 1$  and  $S_{n} = \{(\mathbf{x},y)\}$  without loss of generality. In this case, for the first step, we have
$$
F _ {1} \left(\theta_ {0}, S _ {n}, \xi_ {1}\right) = \theta_ {0} - c _ {\alpha , \beta} \eta_ {0} \frac {\nabla_ {\theta} \ell \left(h _ {\theta_ {0}} (\mathbf {x}) , y\right)}{\sqrt {\left| \nabla_ {\theta} \ell \left(h _ {\theta_ {0}} (\mathbf {x}) , y\right) \right| ^ {2} + \epsilon \mathbf {1}}} = \theta_ {0} - \eta_ {0} \left( \begin{array}{c} s _ {1} \left(\bar {\theta} _ {0} \star_ {s} \mathbf {x}, \bar {\theta} _ {0} ^ {c}\right) \mathbf {x} _ {I _ {1}} \\ s _ {2} \left(\bar {\theta} _ {0} \star_ {s} \mathbf {x}, \bar {\theta} _ {0} ^ {c}\right) \mathbf {x} _ {I _ {2}} \\ \vdots \\ s _ {D _ {1}} \left(\bar {\theta} _ {0} \star_ {s} \mathbf {x}, \bar {\theta} _ {0} ^ {c}\right) \mathbf {x} _ {I _ {D _ {1}}} \\ O \left(\bar {\theta} _ {0} \star_ {s} \mathbf {x}, \bar {\theta} _ {0} ^ {c}\right) \end{array} \right) \tag {96}
$$
for some function  $\{s_i(\cdot)\}_{i=1}^{D_1}$  and  $O(\cdot)$ . Here we recall that in our paper  $s = 2$  and the local linear operator  $\star_s: \mathbb{R}^{ks} \times \mathbb{R}^{ks} \mapsto \mathbb{R}^k$  is defined by  $\mathbf{v} \star_s \mathbf{w} = (\mathbf{v}_{I_1}^\top \mathbf{w}_{I_1}, \mathbf{v}_{I_2}^\top \mathbf{w}_{I_2}, \ldots, \mathbf{v}_{I_k}^\top \mathbf{w}_{I_k})$ , where  $I_j = [(j - 1)s + 1, js]$  denotes the indices of  $j$ -th patch.
Noting  $(U\bar{\theta}_0)\star_s(U\mathbf{x}) = \bar{\theta}_0\star_s\mathbf{x}$  for any  $U\in G_{\mathrm{loc}}$  , it is therefore not hard to verify that (91) holds for the update map (96). Thus we show the  $G_{\mathrm{loc}}$  -equivariance of Adam for training LCNs with  $T = 1$  The case of  $T > 1$  can be shown in the same way.
Remark F.2. Note that the case of FCNs was established in (Li et al., 2020, Corollary C.2) but the proof there is not fully rigorous. We here provide a completely rigorous proof. The case of LCNs under Gaussian initialization was studied in Xiao and Pennington (2022), where the equivariance group is  $G_{\text{ort}}(2) \otimes I_{2d}$ . We instead show that under much milder condition on initialization, the equivariance group becomes the local permutation group (10). It should be stressed that the major contribution of this section is providing a rigorous/unified framework to verify these group equivariance instead of yielding new insights.
G CNNs vs. LCNs: Proofs of Section 5.2
G.1 Proof of Theorem 5.4
By Lemma 5.1, we only need to lower bound the minimax error of learning the enlarged class  $\{\bar{h}^*\} \circ G_{\mathrm{loc}}$ . To this end, we need to find a proper packing of this enlarged class and then apply the Fano's method, i.e., Proposition B.20. To this end, we will prove
Lemma G.1. There exist absolute positive constants  $C, c_1, c_2 > 0, c_3 > 1$  such that if  $d \geqslant C$  and  $A_0 \geqslant C$ , there exists  $h_1, h_2, \ldots, h_M \in \{\bar{h}^*\} \circ G_{\mathrm{loc}}$  satisfying that
$$
\sup  _ {i, j} \| h _ {i} - h _ {j} \| _ {L ^ {2} (P)} ^ {2} \leqslant c _ {2}, \inf  _ {i \neq j} \| h _ {i} - h _ {j} \| _ {L ^ {2} (P)} ^ {2} \geqslant \frac {1}{4} c _ {1}, a n d M \geqslant c _ {3} ^ {d}.
$$
Assuming Lemma G.1 is right for now and applying the above packing to Fano's inequality (Proposition B.20), we have
$$
\begin{array}{l} \inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {h \in \{\bar {h} ^ {*} \} \circ G _ {\mathrm {l o c}}} \mathbb {E} \| h _ {\mathbb {A} (S _ {n})} ^ {\mathrm {l c n}} - h \| _ {L ^ {2} (P)} ^ {2} \geqslant A \left(1 - \frac {n}{2 \sigma^ {2} M ^ {2} \log M} \sum_ {j, j ^ {\prime} = 1} ^ {M} \| h _ {j} - h _ {j ^ {\prime}} \| _ {L ^ {2} (P)} ^ {2} - \frac {\log 2}{\log M}\right) \\ \geqslant \frac {1}{1 6} c _ {1} \left(1 - \frac {c _ {2} n}{2 \sigma^ {2} d \log c _ {3}} - \frac {\log 2}{d \log c _ {3}}\right) \tag {97} \\ \end{array}
$$
Taking  $n$  such that the RHS of the above inequality satisfies
$$
\frac {1}{1 6} c _ {1} \left(1 - \frac {c _ {2} n}{2 \sigma^ {2} d \log c _ {3}} - \frac {\log 2}{d \log c _ {3}}\right) \leqslant \frac {c _ {1}}{3 2} =: \epsilon_ {0} \tag {98}
$$
gives  $n = \Omega(\sigma^2 d)$ . Thus, we can conclude that  $\mathcal{C}(\{\bar{h}^*\} \circ G_{\mathrm{loc}}, \epsilon_0) = \Omega(\sigma^2 d)$ . Thus, we complete the proof of Theorem 5.4.
G.1.1 Proof of Lemma G.1
We now show how to construct a packing of  $\{\bar{h}^*\} \circ G_{\mathrm{loc}}$  to satisfy the condition in Lemma G.1. The key idea is to reduce the problem to pack the equivariance group  $G_{\mathrm{loc}}$ . For  $U = \operatorname{diag}(U_1, U_2, \ldots, U_{2d}) \in G_{\mathrm{loc}}$ , denote by  $U_i \in \mathbb{R}^{2 \times 2}$  the  $i$ -th block matrix. Define a metric over  $G_{\mathrm{loc}}$ : for any  $U, U' \in G_{\mathrm{loc}}$ ,
$$
\rho_ {\text {l o c}} \left(U, U ^ {\prime}\right) := \sharp \left\{i: U _ {i} \neq U _ {i} ^ {\prime} \right\}. \tag {99}
$$
In addition, we consider a subgroup of  $G_{\mathrm{loc}}$ :
$$
G _ {\text {s e m i l o c}} = \left\{\operatorname {d i a g} \left(U _ {1}, U _ {2}, \dots , U _ {2 d}\right) \in G _ {\text {l o c}}: U _ {i} = I _ {2} \text {f o r} i > d \right\} \tag {100}
$$
Clearly, a packing of  $\{\bar{h}^*\} \circ G_{\mathrm{semiloc}}$  is also a packing of  $\{\bar{h}^*\} \circ G_{\mathrm{loc}}$ . For  $G_{\mathrm{semiloc}}$ , we have
Lemma G.2. There exist absolute constants  $C, c_1, c_2 > 0$  such that if  $d, A_0 \geqslant C$ , for any  $\tau, \tau' \in G_{\mathrm{semiloc}}$ , we have
$$
c _ {1} \frac {\rho_ {\mathrm {l o c}} \left(\tau , \tau^ {\prime}\right)}{d} \leqslant \left\| \bar {h} ^ {*} \circ \tau - \bar {h} ^ {*} \circ \tau^ {\prime} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant c _ {2} \frac {\rho_ {\mathrm {l o c}} \left(\tau , \tau^ {\prime}\right)}{d}. \tag {101}
$$
Proof idea. The proof of this lemma is quite technically lengthy and is thus deferred to Appendix I.1. Recall that  $\bar{h}^{*} = \pi_{A_{0}}\circ h^{*}$ . If there is no truncation, i.e.,  $A_0 = \infty$ , the proof is straightforward in the sense that a direct calculation gives  $\| h^{\ast}\circ \tau -h^{\ast}\circ \tau^{\prime}\|_{L^{2}(P)}^{2} = c\rho_{\mathrm{loc}}(\tau ,\tau^{\prime}) / d$  for some absolute constant  $c$ . When  $A_0$  is finite but large enough, by concentration inequalities, we can expect that the truncated target function works in a way similar to the original one. Indeed the lengthy part in the proof is to estimate the influence of truncation.
In the following, we show that  $G_{\mathrm{semiloc}}$  is isometric to the Hamming space:
Definition G.3 (Hamming Space). The Hamming cube  $\{0,1\}^n$  consists of all binary strings of length  $n$ . The Hamming distance  $d_H(x,y)$  between two binary strings is defined as the number of bits where  $x$  and  $y$  disagree, i.e.
$$
d _ {H} (x, y) := \# \{i: x (i) \neq y (i) \}, \quad x, y \in \{0, 1 \} ^ {n}.
$$
We call  $\left(\{0,1\} ^n,d_H\right)$  the Hamming space.
Specifically, it is obvious that
-  $(G_{\mathrm{loc}},\rho_{\mathrm{loc}})$  and  $(\{0,1\}^{2d},d_H)$  are isometric.
-  $(G_{\mathrm{semiloc}}, \rho_{\mathrm{loc}})$  and  $(\{0,1\}^{d}, d_{H})$  are isometric.
For the Hamming space, we have the following bounds for packing and covering numbers.
Lemma G.4. (Vershynin, 2018, Exercise 4.2.16) Let  $K = \{0,1\}^n$ . For every  $0 < m \leqslant n$ , we have
$$
\mathcal {P} \left(K, d _ {H}, m\right) \geqslant \mathcal {N} \left(K, d _ {H}, m\right) \geqslant \frac {2 ^ {n}}{\sum_ {k = 0} ^ {\lfloor m \rfloor} \binom {n} {k}}
$$
Proof. Without loss of generality we set  $m$  as an integer. The first inequality follows trivially from the definition and thus, we only need to prove the second one.
Set  $B_{m,x} = \{y \in K : d_H(x,y) \leqslant m\}$ . For any  $x \in K$ , any  $k \leqslant m$ , there are  $\binom{n}{k}$  elements satisfying  $d_H(x,y) = k$ . This is because the ways of choosing  $k$  elements from  $n$  elements are  $\binom{n}{k}$ , and  $d_H(x,y) = k$  if and only if  $x$  and  $y$  have  $k$  different coordinates. By taking a union bound we have  $|B_{m,x}| \leqslant \sum_{k=0}^{m} \binom{n}{k}$ , where  $|\cdot|$  denotes the cardinality. The ball of radius  $m$  and center  $x$  can only cover at most  $|B_{m,x}|$  elements, However there are  $2^n$  elements in  $K$ . So we have
$$
\mathcal {N} (K, d _ {H}, m) \geqslant \frac {2 ^ {n}}{\sup  _ {x} | B _ {m , x} |} \geqslant \frac {2 ^ {n}}{\sum_ {k = 0} ^ {m} \binom {n} {k}} \tag {102}
$$
By the isometries and Lemma G.4, we trivially have
Lemma G.5. For any  $0 < m \leqslant d$ ,
$$
\mathcal {P} \left(G _ {\text {l o c}}, \rho_ {\text {l o c}}, m\right) \geqslant \mathcal {N} \left(G _ {\text {l o c}}, \rho_ {\text {l o c}}, m\right) \geqslant \frac {4 ^ {d}}{\sum_ {k = 0} ^ {\lfloor m \rfloor} \binom {2 d} {k}}
$$
$$
\mathcal {P} (G _ {\text {s e m i l o c}}, \rho_ {\text {l o c}}, m) \geqslant \mathcal {N} (G _ {\text {s e m i l o c}}, \rho_ {\text {l o c}}, m) \geqslant \frac {2 ^ {d}}{\sum_ {k = 0} ^ {\lfloor m \rfloor} \binom {d} {k}}
$$
To further simplify the lower bounds, we need
Lemma G.6. (Vershynin, 2018, Exercise 0.0.5) Given  $m \in [n]$ ,  $\sum_{k=0}^{m} \binom{n}{k} \leqslant \left(\frac{en}{m}\right)^m$ .
Proof. To prove this we only need to prove  $\left(\frac{m}{n}\right)^m\sum_{k = 0}^{m}\binom{n}{k}\leqslant e^m$ , and we observe that
$$
\left(\frac {m}{n}\right) ^ {m} \sum_ {k = 0} ^ {m} \binom {n} {k} \leqslant \sum_ {k = 0} ^ {m} \left(\frac {m}{n}\right) ^ {k} \binom {n} {k} \leqslant \sum_ {k = 0} ^ {m} \frac {m ^ {k}}{k !} \leqslant \sum_ {k = 0} ^ {+ \infty} \frac {m ^ {k}}{k !} = e ^ {m} \tag {103}
$$
Proof of Lemma G.1. Note that for any  $\tau, \tau' \in G_{\mathrm{semiloc}}, \rho_{\mathrm{loc}}(\tau, \tau') \leqslant d$ . Then, applying Lemma G.2, we have for any  $h = \bar{h}^* \circ \tau$ ,  $h' = \bar{h}^* \circ \tau' \in \{\bar{h}^*\} \circ G_{\mathrm{semiloc}}$  that
$$
\left\| h - h ^ {\prime} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {c _ {2} \rho_ {\mathrm {l o c}} \left(\tau , \tau^ {\prime}\right)}{d} \leqslant c _ {2}. \tag {104}
$$
In addition, Lemma G.2 implies  $\| h - h' \|_{L^2(P)}^2 \geqslant \frac{c_1}{d} \rho_{\mathrm{loc}}(\tau, \tau')$ . Thus, for  $0 < \delta \leqslant \frac{1}{2} \sqrt{c_1}$ :
$$
\begin{array}{l} \mathcal {P} \left(\left\{\bar {h} ^ {*} \right\} \circ G _ {\text {s e m i l o c}}, L ^ {2} (P), \delta\right) \geqslant \mathcal {N} \left(\left\{\bar {h} ^ {*} \right\} \circ G _ {\text {s e m i l o c}}, L ^ {2} (P), \delta\right) \\ \geqslant \mathcal {N} \left(G _ {\text {s e m i l o c}}, \rho_ {\text {l o c}}, d \delta^ {2} / c _ {1}\right) \\ \geqslant \frac {2 ^ {d}}{\sum_ {m = 0} ^ {\lfloor \frac {\delta^ {2} d}{c _ {1}} \rfloor} \left( \begin{array}{c} d \\ m \end{array} \right)}, \\ \end{array}
$$
where the last step follows from Lemma G.5. By Lemma G.6 and taking  $\delta^{*} = \frac{1}{2}\sqrt{c_{1}}$ , we have
$$
\mathcal {P} \left(\left\{\bar {h} ^ {*} \right\} \circ G _ {\text {s e m i l o c}}, L ^ {2} (P), \frac {1}{2} \sqrt {c _ {1}}\right) \geqslant \frac {2 ^ {d}}{\sum_ {m = 0} ^ {\lfloor d / 4 \rfloor} \binom {d} {m}} \geqslant \left(\frac {2}{(5 e) ^ {1 / 4}}\right) ^ {d}. \tag {105}
$$
Let  $c_{3} = \frac{2}{(5e)^{1/4}}$ , which is greater than 1.
Combining equation (104) and (105), we complete the proof.
G.2 Proof of Theorem 5.2
Let  $\sigma(z) \coloneqq \mathrm{ReLU}(z)$  in this subsection for notation simplicity. We first recall the architecture of the CNN model:  $L = \log_2(4d)$ , channel number  $C_1 = C_L = 4$ ,  $C_l = 2$  for  $l = 2, \ldots, L-1$ , and activation functions  $\sigma_1(z) = \sigma_L(z) = \sigma^2(z)$ ,  $\sigma_l(z) = \sigma(z)$  for  $l = 2, \ldots, L-1$ .
Step I: Representing the target using CNNs.
Lemma G.7. There is a parametrization  $\theta^{*}$  such that  $h_{\theta^{*}}^{\mathrm{conn}} = h^{*}$  and  $\| \theta^{*}\|_{\mathcal{P}}\lesssim \log d$
Proof. We provide a constructive proof of this lemma. We fist set all the bias to zeros. Then we set the parameter of the filters  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1} \times 2}$  as follows.
- For  $l = 1$ ,  $W_{1,1,:}^{(1)} = -W_{2,1,:}^{(1)} = (1,0)$ , and  $W_{3,1,:}^{(1)} = -W_{4,1,:}^{(1)} = (0,1)$ .
- For  $l = 2$ ,  $W_{1,1,:}^{(2)} = -W_{1,2,:}^{(2)} = W_{1,3,:}^{(2)} = -W_{1,4,:}^{(2)} = (1,1)$  and  $W_{2,i,:}^{(2)} = -W_{1,i,:}^{(2)}$  for  $i = 1,2,3,4$ .
- For  $l = 3, \ldots, L - 1$ ,  $W_{1,1,:}^{(l)} = -W_{1,2,:}^{(l)} = (1,1)$  and  $W_{2,i,:}^{(l)} = -W_{1,i,:}^{(l)}$  for  $i = 1,2$ .
- For  $l = L$ ,  $W_{1,1,:}^{(L)} = -W_{2,1,:}^{(L)} = (1,1)$ ,  $W_{3,1,:}^{(L)} = -W_{4,1,:}^{(L)} = (1, -1)$  and  $W_{j,2,:}^{(L)} = -W_{j,1,:}^{(L)}$  for  $j = 1,2,3,4$ .
- For the final linear layer,  $W_{o} = \frac{1}{4d} (1, 1, -1, -1)$ .
First, it is easy to check the output of the first two layer by direct calculation. Concretely, we have
$$
\left(\mathbf {z} ^ {(1)}\right) ^ {\top} = \left( \begin{array}{c c c c} \sigma^ {2} (x _ {1}) & \sigma^ {2} (x _ {3}) & \dots & \sigma^ {2} \left(x _ {4 d - 1}\right) \\ \sigma^ {2} (- x _ {1}) & \sigma^ {2} (- x _ {3}) & \dots & \sigma^ {2} (- x _ {4 d - 1}) \\ \sigma^ {2} (x _ {2}) & \sigma^ {2} (x _ {4}) & \dots & \sigma^ {2} (x _ {4 d}) \\ \sigma^ {2} (- x _ {2}) & \sigma^ {2} (- x _ {4}) & \dots & \sigma^ {2} (- x _ {4 d})) \end{array} \right) \in \mathbb {R} ^ {4 \times 2 d} \tag {106}
$$
$$
\left(\mathbf {z} ^ {(2)}\right) ^ {\top} = \left( \begin{array}{l l l} \sigma \left(x _ {1} ^ {2} - x _ {2} ^ {2} + x _ {3} ^ {2} - x _ {4} ^ {2}\right) & \dots & \sigma \left(x _ {4 d - 3} ^ {2} - x _ {4 d - 2} ^ {2} + x _ {4 d - 1} ^ {2} - x _ {4 d} ^ {2}\right) \\ \sigma \left(- x _ {1} ^ {2} + x _ {2} ^ {2} - x _ {3} ^ {2} + x _ {4} ^ {2}\right) & \dots & \sigma \left(- x _ {4 d - 3} ^ {2} + x _ {4 d - 2} ^ {2} - x _ {4 d - 1} ^ {2} + x _ {4 d} ^ {2}\right) \end{array} \right) \in \mathbb {R} ^ {2 \times d} \tag {107}
$$
Next, we are going to prove the following conclusion for  $l = 2,\ldots ,L - 1$  by induction
$$
\left(\mathbf {z} ^ {(l)}\right) ^ {\top} = \left( \begin{array}{c c c} \sigma \left(\sum_ {k = 1} ^ {2 ^ {l - 1}} \left(x _ {2 k - 1} ^ {2} - x _ {2 k} ^ {2}\right)\right) & \dots & \sigma \left(\sum_ {k = 1} ^ {2 ^ {l - 1}} \left(x _ {4 d - 2 k + 3} ^ {2} - x _ {4 d - 2 k + 2} ^ {2}\right)\right) \\ \sigma \left(- \sum_ {k = 1} ^ {2 ^ {l - 1}} \left(x _ {2 k - 1} ^ {2} - x _ {2 k} ^ {2}\right)\right) & \dots & \sigma \left(- \sum_ {k = 1} ^ {2 ^ {l - 1}} \left(x _ {4 d - 2 k + 3} ^ {2} - x _ {4 d - 2 k + 2} ^ {2}\right)\right) \end{array} \right) \in \mathbb {R} ^ {2 \times d / 2 ^ {l - 2}} \tag {108}
$$
First, by (107), it holds for  $l = 2$ . Next we assume (108) holds for the case of  $l - 1$  and verify it holds also for the case of  $l$ .
By the definition of our CNN model, we have
$$
\left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {i, j} = \mathbf {z} _ {2 i - 1, 1} ^ {(l - 1)} W _ {j, 1, 1} ^ {(l)} + \mathbf {z} _ {2 i - 1, 2} ^ {(l - 1)} W _ {j, 2, 1} ^ {(l)} + \mathbf {z} _ {2 i, 1} ^ {(l - 1)} W _ {j, 1, 2} ^ {(l)} + \mathbf {z} _ {2 i, 2} ^ {(l - 1)} W _ {j, 2, 2} ^ {(l)} \tag {109}
$$
when  $j = 1$
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {i, 1} = \mathbf {z} _ {2 i - 1, 1} ^ {(l - 1)} - \mathbf {z} _ {2 i - 1, 2} ^ {(l - 1)} + \mathbf {z} _ {2 i, 1} ^ {(l - 1)} - \mathbf {z} _ {2 i, 2} ^ {(l - 1)} \\ = \sum_ {k = 1} ^ {2 ^ {l - 2}} \left(x _ {2 k - 1 + (2 i - 2) 2 ^ {l - 1}} ^ {2} - x _ {2 k + (2 i - 2) 2 ^ {l - 1}} ^ {2}\right) + \sum_ {k = 1} ^ {2 ^ {l - 2}} \left(x _ {2 k - 1 + (2 i - 1) 2 ^ {l - 1}} ^ {2} - x _ {2 k + (2 i - 1) 2 ^ {l - 1}} ^ {2}\right) \\ = \sum_ {k = 1} ^ {2 ^ {l - 1}} \left(x _ {2 k - 1 + (i - 1) 2 ^ {l}} ^ {2} - x _ {2 k + (i - 1) 2 ^ {l}} ^ {2}\right), \\ \end{array}
$$
which verifies (108) for  $j = 1$ . The case of  $j = 2$  can be verified in the same way.
So by now we have verified that the output of the  $L - 1$  layer is
$$
\left(\mathbf {z} ^ {(L - 1)}\right) ^ {\top} = \left( \begin{array}{c c} \sigma \left(\sum_ {i = 1} ^ {d} \left(x _ {2 i - 1} ^ {2} - x _ {2 i} ^ {2}\right)\right) & \sigma \left(\sum_ {i = 1} ^ {d} \left(x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2}\right)\right) \\ \sigma \left(- \sum_ {i = 1} ^ {d} \left(x _ {2 i - 1} ^ {2} - x _ {2 i} ^ {2}\right)\right) & \sigma \left(- \sum_ {i = 1} ^ {d} \left(x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2}\right)\right) \end{array} \right) \in \mathbb {R} ^ {2 \times 2} \tag {110}
$$
Noting  $\sigma(x) - \sigma(-x) = x$ , we can recover  $\sum_{i=1}^{d}(x_{2i-1}^2 - x_{2i}^2)$  and  $\sum_{i=1}^{d}(x_{2i+2d-1}^2 - x_{2i+2d}^2)$  by using the feature stored in  $\mathbf{z}^{(L-1)}$ .
Recall that in our last layer, our activation function is  $\sigma_L(x) = \sigma^2 (x)$ , and we have  $\sigma^2 (x) + \sigma^2 (-x) = x^2$ . Noting that  $4\alpha \beta = (\alpha +\beta)^{2} - (\alpha -\beta)^{2}$  and viewing  $\sum_{i = 1}^{d}(x_{2i - 1}^{2} - x_{2i}^{2})$ ,  $\sum_{i = 1}^{d}(x_{2i + 2d - 1}^{2} - x_{2i + 2d}^{2})$  as  $\alpha$  and  $\beta$ , respectively, a simple calculation would lead to  $h_{\theta^*}^{\mathrm{cnn}}(\mathbf{x}) = h^* (\mathbf{x})$ .
Note that for all  $l \in [L]$ ,  $\mathbf{b}^{(l)} = 0$ ,  $C_l \leqslant 4$ . Then, it is obvious that there exists an absolute constant  $C > 0$  such that  $\| W^{(l)}\|_F \leqslant C$  for any  $l \in [L]$  and  $\| W_o\|_2 \leqslant C$ . Thus, we have  $\| \theta^*\|_{\mathcal{P}} = \| W_o\|_2 + \sum_{l=1}^L \| W^{(l)}\|_F \lesssim L \lesssim \log d$ .
Step II: Estimating the sample complexity.
Recall the input distribution  $P = \mathcal{N}(0, I_{4d})$ . Let  $\mathcal{H}_J^{\mathrm{CNN}} = \{h_\theta : \| \theta \| \leqslant J\}$ , where we recall  $\| \cdot \|$  is defined in Appendix C. By Proposition B.14, the covering number satisfies
$$
\mathcal {N} \left(\mathcal {H} _ {J} ^ {\mathrm {C N N}}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {\gamma} _ {n} (J)}{t}\right) ^ {N _ {\mathrm {c n n}}} \tag {111}
$$
where  $\hat{\gamma}_n(J) = \hat{M}_n J(1 + J)^L$ ,  $\hat{M}_n = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \left( \bar{Q}_{\sigma}(\mathbf{x}_i) \right)^2 \left( \| \mathbf{x}_i \|_2 + 1 \right)^2}$  and  $N_{\mathrm{cnn}} = \mathcal{O}(\log d)$ . We will use the fact that  $L = \log_2 d + 2$ .
First, we turn to the estimate of  $\bar{Q}_{\sigma}(\mathbf{x}_i)$ . To this end, we give a simple lemma to estimate the local Lipschitz constant of our activation functions.
Lemma G.8. If for  $|x|, |y| \leqslant M$ , we have  $|\beta(x) - \beta(y)| \leqslant K|x - y|$ , then we will have
$$
\left\| \beta (\mathbf {x}) - \beta (\mathbf {y}) \right\| _ {2} \leqslant K \left\| \mathbf {x} - \mathbf {y} \right\| _ {2}
$$
for any  $\| \mathbf{x}\| _2,\| \mathbf{y}\| _2\leqslant M$
$$
\text {P r o o f .} \| \beta (\mathbf {x}) - \beta (\mathbf {y}) \| _ {2} ^ {2} = \sum_ {i} (\beta (x _ {i}) - \beta (y _ {i})) ^ {2} \leqslant \sum_ {i} K ^ {2} (x _ {i} - y _ {i}) ^ {2} = K ^ {2} \| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2}
$$
Our  $\sigma_{1}(\cdot) = \sigma_{L}(\cdot) = \sigma^{2}(\cdot)$  activation functions are  $2K$  Lipschitz continuous when restricted in  $[-K, K]$ . So for  $Q_{1}(\mathbf{x}_{i})$ , since  $\| \mathcal{T}_1(\mathbf{x}_i)\|_2 \leqslant \| \mathcal{T}_1\| (\|\mathbf{x}_i\|_2 + 1) \leqslant J(\|\mathbf{x}_i\|_2 + 1)$ , we have  $Q_{1}(\mathbf{x}_{i}) \leqslant 2J(\|\mathbf{x}_{i}\|_{2} + 1)$ . For  $Q_{L}(\mathbf{x}_{i})$ , we similarly have
$$
\left\| \mathcal {T} _ {L} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x} _ {i}) \right\| _ {2} \leqslant (1 + J) ^ {L + 1} \left(\left\| \mathbf {x} _ {i} \right\| _ {2} + 1\right) ^ {2},
$$
so  $Q_{L}(\mathbf{x}_{i}) \leqslant 2(1 + J)^{L + 1}(\| \mathbf{x}_{i}\|_{2} + 1)^{2}$  is straightforward. Thus,  $\bar{Q}_{\sigma}(\mathbf{x}_i) \leqslant 2^{L + 2}(1 + J)^{L + 2}(\| \mathbf{x}_i\|_2 + 1)^3$ . Therefore, we will have
$$
\begin{array}{l} \gamma (J) = \mathbb {E} \left[ \hat {\gamma} _ {n} (J) \right] \lesssim J (1 + J) ^ {L} \mathbb {E} [ \hat {M} _ {n} ] \\ \lesssim J (1 + J) ^ {2 L + 2} 2 ^ {L + 2} \sqrt {\mathbb {E} [ (\| \mathbf {x} \| _ {2} + 1) ^ {8} ]} \lesssim d ^ {2} J (1 + J) ^ {2 L + 2} 2 ^ {L + 2} \tag {112} \\ \end{array}
$$
where the third step follows from the Jensen's inequality and the fact that  $z \mapsto \sqrt{z}$  is concave; the last step uses the higher-order moment bound of sub-exponential random variables (Vershynin, 2018, Proposition 2.7.1).
Recalling we have  $\alpha_{\mathcal{H}} = 1$  from equation (53) and applying Proposition B.14, we have  $w.p.$  at least  $1 - 2\delta$
$$
\epsilon_ {*} = 0, \quad M _ {*} \lesssim \log d
$$
Taking  $A = A_0$ ,  $B = 2A + \sigma \sqrt{\log n}$ ,  $\lambda = 1 / \sqrt{n}$  and applying Lemma B.13, we have