SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem C.2 (Covering number of CNN). Let  $h_\theta$  denote the CNN described in Section 2.1 and define  $\mathcal{H}_J^{\mathrm{CNN}} = \{h_\theta : \| \theta \| \leqslant J\}$ . Given  $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n \in \mathcal{X}$ , denote  $\hat{M}_n = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\bar{Q}_\sigma(\mathbf{x}_i))^2 (\|\mathbf{x}_i\|_2 + 1)^2}$ . Then, we have
$$
\mathcal {N} \left(\mathcal {H} _ {J} ^ {\mathrm {C N N}}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {M} _ {n} J (1 + J) ^ {L}}{t}\right) ^ {N _ {\mathrm {c n n}}}, \tag {57}
$$
$$
\operatorname {g i v e n} 0 <   t \leqslant \hat {M} _ {n} J (1 + J) ^ {L}.
$$
Proof. The conclusion follows trivially from Lemma C.1 and Lemma B.8.  $\square$
C.2 LCNs
Recall that for LCNs, the linear transform  $\mathcal{T}_l: \mathbb{R}^{D_{l-1} \times C_{l-1}} \mapsto \mathbb{R}^{D_l \times C_l}$  is parameterized by  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1} \times D_{l-1}}$  and  $\mathbf{b}^{(l)} \in \mathbb{R}^{D_l \times C_l}$  as follows
$$
\left(\mathcal {T} _ {l} (\mathbf {z})\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} \star_ {s} W _ {j, i,:} ^ {(l)} + \mathbf {b} _ {:, j} ^ {(l)}, \quad \text {f o r} j = 1, \dots , C _ {l}.
$$
We further point out that the number of parameters in our LCN model is
$$
N _ {\mathrm {l c n}} = C _ {L} D _ {L} + \sum_ {l = 0} ^ {L - 1} \left(2 C _ {l} + 1\right) C _ {l + 1} D _ {l + 1} \tag {58}
$$
A patch-based reformulation of LCNs. We reformulate deep LCNs in the same way as CNNs. Let  $\widetilde{\mathbf{z}}^{(l)}\in \mathbb{R}^{D_lC_l}$  be the vectorized features; then the matrix form of  $\mathcal{T}_l:\mathbb{R}^{D_{l - 1}C_{l - 1}}\to \mathbb{R}^{D_lC_l}$  is given by
$$
\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)} = K ^ {(l)} \widetilde {\mathbf {z}} ^ {(l - 1)} + \mathbf {s} ^ {(l)} \tag {59}
$$
where  $K^{(l)} = \mathrm{diag}\{\widetilde{W}_1^{(l)},\dots ,\widetilde{W}_{D_l}^{(l)}\}$  and  $\widetilde{W}_i^{(l)}\in \mathbb{R}^{C_l\times (sC_{l - 1})}$ , where
$$
\widetilde {W} ^ {(l)} = \left( \begin{array}{c} \operatorname {v e c} \left(W _ {1,: I _ {1}}\right) ^ {\top} \\ \operatorname {v e c} \left(W _ {2,: I _ {2}}\right) ^ {\top} \\ \vdots \\ \operatorname {v e c} \left(W _ {C _ {l,:}, I _ {C _ {l}}}\right) ^ {\top} \end{array} \right) \in \mathbb {R} ^ {C _ {l} \times (s C _ {l - 1})}, \quad \mathbf {s} ^ {(l)} = \left( \begin{array}{c} \mathbf {b} _ {1,:} ^ {\top} \\ \mathbf {b} _ {2,:} ^ {\top} \\ \vdots \\ \mathbf {b} _ {D _ {l,:}} ^ {\top} \end{array} \right) \in \mathbb {R} ^ {D _ {l} C _ {l}}, \tag {60}
$$
where  $I_{j} = [(j - 1)s + 1, js]$ .
Parameter norm. By (59), it is obvious that
$$
\begin{array}{l} \| \mathcal {T} _ {l} \| \leqslant \| K ^ {(l)} \| _ {2} + \| \mathbf {s} ^ {(l)} \| _ {2} \leqslant \max  _ {j \in [ D _ {l} ]} \| \widetilde {W} _ {j} ^ {(l)} \| + \| \mathbf {b} ^ {(l)} \| _ {F} \\ \leqslant \| W ^ {(l)} \| _ {F} + \| \mathbf {b} ^ {(l)} \| _ {F}. \\ \end{array}
$$
$$
\left\| \theta \right\| = \left\| W _ {o} \right\| + \sum_ {l = 1} ^ {L} \left\| \mathcal {T} _ {l} \right\| \leqslant \left\| W _ {o} \right\| + \sum_ {l = 1} ^ {L} \left(\left\| W ^ {(l)} \right\| _ {F} + \left\| \mathbf {b} ^ {(l)} \right\| _ {F}\right) = \| \theta \| _ {\mathcal {P}}. \tag {61}
$$
In the following, we provide an upper bound of the covering number of deep LCNs, whose proof is nearly the same as that of deep CNNs.
Lemma C.3. For any  $\theta, \theta'$  satisfy that  $\| \theta \| \leqslant J$  and  $\| \theta' \| \leqslant J$  and any  $\mathbf{x} \in \mathcal{X}$ , assume  $\sigma_l$  satisfies  $\sigma_l(0) = 0$  and locally Lipschitz condition  $\| \sigma_l(\mathbf{y}) - \sigma_l(\mathbf{z}) \|_2 \leqslant Q_l(\mathbf{x}) \| \mathbf{y} - \mathbf{z} \|_2$  for any  $\mathbf{y}, \mathbf{z} \in \{\mathcal{T}_l \circ \dots \circ \sigma_1 \circ \mathcal{T}_1(\mathbf{x}) : \| \theta \| \leqslant J\}$ . We further denote  $\bar{Q}_{\sigma}(\mathbf{x}) = \prod_{l=1}^{L} (Q_l(\mathbf{x}) + 1)$ . Then, we have
$$
\left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| \leqslant \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \| \theta - \theta^ {\prime} \| \tag {62}
$$
The proof is exactly the same as that of Lemma C.1, i.e., the case of CNNs.
$$
\begin{array}{l} \left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| = \left| \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) - \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \widetilde {\mathcal {T}} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \right| \\ \leqslant \operatorname {L i p} \left(\mathcal {M} _ {o} \circ \dots \circ \sigma_ {l}\right) \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \left(\| \sigma_ {l - 1} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \| _ {2} + 1\right) \\ \leqslant \operatorname {L i p} \left(\mathcal {M} _ {o} \circ \dots \circ \sigma_ {l}\right) \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| (\| \mathbf {x} \| _ {2} + 1) (1 + J) ^ {l - 1} \left(\prod_ {j = 1} ^ {l - 1} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) \\ \leqslant \left(\prod_ {j = l} ^ {L} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) (1 + J) ^ {L - l + 1} \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {l - 1} \left(\prod_ {j = 1} ^ {l - 1} \left(Q _ {j} (\mathbf {x}) + 1\right)\right) \\ = \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \left\| \mathcal {T} _ {l} - \widetilde {\mathcal {T}} _ {l} \right\| \tag {63} \\ \end{array}
$$
$$
\begin{array}{l} \left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right| = \left| \mathcal {M} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) - \widetilde {\mathcal {M}} _ {o} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x}) \right| \\ \leqslant \left\| W _ {o} - \widetilde {W _ {o}} \right\| _ {2} \| (\sigma_ {L} \circ \dots \circ \sigma_ {l} \circ \mathcal {T} _ {l} \circ \dots \circ \sigma_ {1} \circ \mathcal {T} _ {1} (\mathbf {x})) \| _ {2} \\ \leqslant \left\| W _ {o} - \widetilde {W} _ {o} \right\| _ {2} \bar {Q} _ {\sigma} (\mathbf {x}) \left(\| \mathbf {x} \| _ {2} + 1\right) (1 + J) ^ {L} \tag {64} \\ \end{array}
$$
-  $\| \theta_l\| \leqslant J$  for all  $l\in [L + 1]$
-  $\theta' = \theta_{L+1}$ ,  $\theta$  and  $\theta_1$  only differ in one layer;
for any  $l$ ,  $\theta_{l}$  and  $\theta_{l + 1}$  only differ in one layer.
Theorem C.4 (Covering number of LCNs). Let  $h_\theta$  denote the LCN model described in Section 2.1 and define  $\mathcal{H}_J^{\mathrm{LCN}} = \{h_\theta : \| \theta \| \leqslant J\}$ . Given  $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n \in \mathcal{X}$ , denote  $\hat{M}_n = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\bar{Q}_\sigma(\mathbf{x}_i))^2 (\|\mathbf{x}_i\|_2 + 1)^2}$ . Then, we have
$$
\mathcal {N} \left(\mathcal {H} _ {J} ^ {\mathrm {L C N}}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {M} _ {n} J (1 + J) ^ {L}}{t}\right) ^ {N _ {\mathrm {l c n}}}, \tag {65}
$$
given  $0 < t \leqslant \hat{M}_n J(1 + J)^L$ .
Proof. The conclusion follows trivially from Lemma C.3 and Lemma B.8.  $\square$
Remark C.5. Note that the number of parameters of a LCN is much larger than that of its CNN counterpart due to the lack of weight sharing. As a result, the covering number of LCNs is accordingly much larger than that of CNNs.
C.3 Comparison with Long and Sedghi (2019).
We acknowledge that our proofs follow a similar approach developed in Long and Sedghi (2019) but we improve Long and Sedghi (2019) in various aspects.
- We provide an upper bound of the covering number of LCNs while Long and Sedghi (2019) only considered CNNs. Moreover, our bounds apply to activation functions that are not Lipschitz continuous such as the squared ReLU:  $\sigma(z) = \max(0, z)^2$ . This is activation function has become popular recently in solving scientific computing problems (Li and Yu, 2019; E and Yu, 2018) and training large NLP pre-trained models (So et al., 2021).
- The bound in Long and Sedghi (2019) depends on the parameter norm  $J$  exponentially. In contrast, our bound depends on  $J$  only polynomially. This improvement is critical for obtaining a sharp generalization bound for the case where the parameter norm is relatively large. For instance, by applying Long and Sedghi (2019) to the setting of learning the separation task in Theorem 5.5, the corresponding sample complexity bound of LCNs will become  $\widetilde{\mathcal{O}}(d^2)$ . In contrast, our bound is  $\widetilde{\mathcal{O}}(d)$ . This improvement is very critical since the lower bound of learning with FCNs is  $\Omega(d^2)$  and therefore, without this improvement, we are unable to establish the provable separation between LCNs and FCNs.
D Universal Approximation: Proofs in Section 3
We first recall a well-known universality result of two-layer ReLU networks.
Lemma D.1. (Leshno et al., 1993) Let  $\Omega$  be any compact set in  $\mathbb{R}^d$  and  $\sigma$  be the ReLU function. For any  $h\in C(\Omega)$  and any  $\epsilon >0$ , there exists a two-layer neural network  $f_{m}(\mathbf{x};\theta^{\prime}) = \sum_{j = 1}^{m}a_{j}\sigma (\mathbf{u}_{j}^{\top}\mathbf{x} + c_{j})$  such that
$$
\sup  _ {\mathbf {x} \in \Omega} \left| f _ {m} (\mathbf {x}; \theta^ {\prime}) - h (\mathbf {x}) \right| \leqslant \epsilon . \tag {66}
$$
Lemma D.2 (Feature extraction). With the choice of architecture in Theorem 3.1, the exist parameters such that  $\mathbf{z}^{(L - 1)}(\mathbf{x})\in \mathbb{R}^{2\times (4d)}$  satisfies
$$
\mathbf {z} ^ {(L - 1)} (\mathbf {x}) = \left( \begin{array}{c c c c c c c} \sigma (x _ {1}) & \sigma (- x _ {1}) & \sigma (x _ {2}) & \sigma (- x _ {2}) & \dots & \sigma (x _ {2 d}) & \sigma (- x _ {2 d}) \\ \sigma (x _ {2 d + 1}) & \sigma (- x _ {2 d + 1}) & \sigma (x _ {2 d + 2}) & \sigma (- x _ {2 d + 2}) & \dots & \sigma (x _ {4 d}) & \sigma (- x _ {4 d}) \end{array} \right) \in \mathbb {R} ^ {2 \times 4 d}.
$$
Remark D.3. This lemma shows that  $\mathbf{z}^{(L - 1)}(\mathbf{x})$  strictly stores spatial information into different channels. Note that 4d is the number of channels and 2 is the spatial dimension. The stored information information is in the form of  $\{\sigma (x_i),\sigma (-x_i)\}_{i = 1}^{4d}$ . Noticing  $t = \sigma (t) - \sigma (-t)$ , we can conclude that there is no information loss since for any  $h:\mathbb{R}^{4d}\mapsto \mathbb{R}$ ,  $h(x_{1},x_{2},\ldots ,x_{4d})$  can be represented using the stored features as follows
$$
h \left(\sigma \left(x _ {1}\right) - \sigma \left(- x _ {1}\right), \sigma \left(x _ {2}\right) - \sigma \left(- x _ {2}\right), \dots , \sigma \left(x _ {4 d}\right) - \sigma \left(- x _ {4 d}\right)\right).
$$
It is worth noting that obviously, the weights and bias in  $\mathcal{T}_1,\ldots ,\mathcal{T}_{L - 1}$  do not depend on the target function  $h^*$ .
Proof. We provide a constructive proof to this lemma. First, we set all bias to be zero, i.e.,  $\mathbf{b}^{(l)} = \mathbf{0}$  for all  $l\in [L - 1]$ . Next we state how to set the weights for different layers.
- When  $l = 0$ ,  $\mathbf{z}^{(0)} = \mathbf{x} = (x_1, x_2, \ldots, x_{4d})$ .
- When  $l = 1$ , set  $W^{(1)} \in \mathbb{R}^{1 \times 4 \times 2}$  as follows  $W_{1,1,:}^{(1)} = (1,0)^{\top}, W_{1,2,:}^{(1)} = (-1,0)^{\top}, W_{1,3,:}^{(1)} = (0,1)^{\top}, W_{1,4,:}^{(1)} = (0,-1)^{\top}$ . Under this construction, it is easy to verify that
$$
\left(\mathbf {z} ^ {(1)}\right) ^ {\top} = \left( \begin{array}{c c c c} \sigma \left(x _ {1}\right) & \sigma \left(x _ {3}\right) & \dots & \sigma \left(x _ {4 d - 1}\right) \\ \sigma \left(- x _ {1}\right) & \sigma \left(- x _ {3}\right) & \dots & \sigma \left(- x _ {4 d - 1}\right) \\ \sigma \left(x _ {2}\right) & \sigma \left(x _ {4}\right) & \dots & \sigma \left(x _ {4 d}\right) \\ \sigma \left(- x _ {2}\right) & \sigma \left(- x _ {4}\right) & \dots & \sigma \left(- x _ {4 d}\right) \end{array} \right) \in \mathbb {R} ^ {4 \times 2 d} \tag {67}
$$
- Similarly, for  $l = 2$ , we can set  $W^{(2)} \in \mathbb{R}^{4 \times 8 \times 2}$  such that
$$
(\mathbf {z} ^ {(2)}) ^ {\top} = \left( \begin{array}{l l l l} \sigma (x _ {1}) & \sigma (x _ {5}) & \ldots & \sigma (x _ {4 d - 3}) \\ \sigma (- x _ {1}) & \sigma (- x _ {5}) & \ldots & \sigma (- x _ {4 d - 3}) \\ \sigma (x _ {2}) & \sigma (x _ {6}) & \ldots & \sigma (x _ {4 d - 2}) \\ \sigma (- x _ {2}) & \sigma (- x _ {6}) & \ldots & \sigma (- x _ {4 d - 2})) \\ \sigma (x _ {3}) & \sigma (x _ {7}) & \ldots & \sigma (x _ {4 d - 1})) \\ \sigma (- x _ {3}) & \sigma (- x _ {7}) & \ldots & \sigma (- x _ {4 d - 1})) \\ \sigma (x _ {4}) & \sigma (x _ {8}) & \ldots & \sigma (x _ {4 d})) \\ \sigma (- x _ {4}) & \sigma (- x _ {8}) & \ldots & \sigma (- x _ {4 d})) \end{array} \right) \in \mathbb {R} ^ {8 \times d}
$$
- By induction, when  $l = L - 1$ , we have
$$
(\mathbf {z} ^ {(L - 1)}) ^ {\top} = \left( \begin{array}{c c} \sigma (x _ {1}) & \sigma (x _ {2 d + 1}) \\ \sigma (- x _ {1}) & \sigma (- x _ {2 d + 1}) \\ \sigma (x _ {2}) & \sigma (x _ {2 d + 2}) \\ \sigma (- x _ {2}) & \sigma (- x _ {2 d + 2}) \\ \vdots & \vdots \\ \sigma (x _ {2 d - 1}) & \sigma (x _ {4 d - 1}) \\ \sigma (- x _ {2 d - 1}) & \sigma (- x _ {4 d - 1}) \\ \sigma (x _ {2 d}) & \sigma (x _ {4 d}) \\ \sigma (- x _ {2 d}) & \sigma (- x _ {4 d}) \end{array} \right) \in \mathbb {R} ^ {4 d \times 2}
$$
Specifically, the above result is achieved by setting the weights of  $l = 2,\dots ,L - 1$  as follows. For  $1\leqslant j\leqslant C_{l - 1}$  and  $j$  is odd, we set
$$
W _ {j, j,:} ^ {(l)} = - W _ {j + 1, j,:} ^ {(l)} = W _ {j + 1, j + 1,:} ^ {(l)} = - W _ {j, j + 1,:} ^ {(l)} = (1, 0) ^ {\top}
$$
$$
W _ {j, j + C _ {l - 1,:}} ^ {(l)} = - W _ {j + 1, j + C _ {l - 1,:}} ^ {(l)} = W _ {j + 1, j + 1 + C _ {l - 1,:}} ^ {(l)} = - W _ {j, j + 1 + C _ {l - 1,:}} ^ {(l)} = (0, 1) ^ {\top}
$$
Recall that  $W_{i,j,\cdot}^{(l)}$  represent the filter used to extract the information from the  $i$ -th input channel to the  $j$ -th output channel. Let  $j' = (j + 1)/2$  if  $j$  is odd. To prove the above result, we only show that for any  $l \in [L - 1]$ ,
$$
\mathbf {z} _ {:, j} ^ {(l)} = \left\{ \begin{array}{l l} \left(\sigma \left(x _ {j ^ {\prime}}\right), \sigma \left(x _ {j ^ {\prime} + 2 ^ {l}}\right), \dots , \sigma \left(x _ {j ^ {\prime} + \left(D _ {l} - 1\right) 2 ^ {l}}\right)\right) & \text {i f} j \text {i s o d d} \\ \left(\sigma \left(- x _ {j ^ {\prime}}\right), \sigma \left(- x _ {j ^ {\prime} + 2 ^ {l}}\right), \dots , \sigma \left(- x _ {j ^ {\prime} + \left(D _ {l} - 1\right) 2 ^ {l}}\right)\right) & \text {i f} j \text {i s e v e n .} \end{array} \right. \tag {68}
$$
We verify this below by induction.
First, the case of  $l = 1$  holds due to (68). Assume (68) holds for  $1, \ldots, l - 1$ , let us compute  $(\mathcal{T}_l(\mathbf{z}^{(l-1)}))_{\cdot,j}$ . Without loss of generality, we only consider the case where  $j$  is odd.
- When  $j \leqslant C_{l - 1}$ ,
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} ^ {(l - 1)} * _ {s} W _ {j, i,:} ^ {(l)} + b _ {j} ^ {(l)} \\ = \mathbf {z} _ {:, j} ^ {(l - 1)} * _ {s} W _ {j, j,:} ^ {(l)} + \mathbf {z} _ {:, j + 1} ^ {(l - 1)} * _ {s} W _ {j + 1, j,:} ^ {(l)} \\ \end{array}
$$
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {i, j} = \mathbf {z} _ {2 i - 1, j} ^ {(l - 1)} - \mathbf {z} _ {2 i - 1, j + 1} ^ {(l - 1)} \\ = \sigma \left(x _ {j ^ {\prime} + (2 i - 2) 2 ^ {l - 1}}\right) - \sigma \left(- x _ {j ^ {\prime} + (2 i - 2) 2 ^ {l - 1}}\right) = x _ {j ^ {\prime} + (i - 1) 2 ^ {l}} \\ \end{array}
$$
- When  $j \geqslant C_{l - 1}$  we similarly have
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} ^ {(l - 1)} * _ {s} W _ {j, i,:} ^ {(l)} + b _ {j} ^ {(l)} \\ = \mathbf {z} _ {:, j - C _ {l - 1}} ^ {(l - 1)} * _ {s} W _ {j - C _ {l - 1}, j,:} ^ {(l)} + \mathbf {z} _ {:, j + 1 - C _ {l - 1}} ^ {(l - 1)} * _ {s} W _ {j + 1 - C _ {l - 1}, j,:} ^ {(l)} \\ \end{array}
$$
2The following verification is rigorous but not intuitive.
$$
\begin{array}{l} \left(\mathcal {T} _ {l} \mathbf {z} ^ {(l - 1)}\right) _ {i, j} = \mathbf {z} _ {2 i, j - C _ {l - 1}} ^ {(l - 1)} - \mathbf {z} _ {2 i, j + 1 - C _ {l - 1}} ^ {(l - 1)} \\ = \sigma \left(x _ {j ^ {\prime} - C _ {l - 1} / 2 + (2 i - 1) 2 ^ {l - 1}}\right) - \sigma \left(- x _ {j ^ {\prime} - C _ {l - 1} / 2 + (2 i - 1) 2 ^ {l - 1}}\right) = x _ {j ^ {\prime} + (i - 1) 2 ^ {l}} \\ \end{array}
$$
Thus, the case of  $l + 1$  also holds.
Proof of Theorem 3.1. By setting the weights and biases of the first  $L - 1$  layers according to Lemma D.2, we have  $h_{\theta}(\mathbf{x}) = \mathcal{M}_o\circ \sigma \circ \mathcal{T}_L(\mathbf{z}^{(L - 1)}(\mathbf{x}))$ , where
$$
\mathbf {z} ^ {(L - 1)} (\mathbf {x}) = \left( \begin{array}{c c c c c c c} \sigma (x _ {1}) & \sigma (- x _ {1}) & \sigma (x _ {2}) & \sigma (- x _ {2}) & \dots & \sigma (x _ {2 d}) & \sigma (- x _ {2 d}) \\ \sigma (x _ {2 d + 1}) & \sigma (- x _ {2 d + 1}) & \sigma (x _ {2 d + 2}) & \sigma (- x _ {2 d + 2}) & \dots & \sigma (x _ {4 d}) & \sigma (- x _ {4 d}) \end{array} \right)
$$
By Lemma D.1, for any  $h^* \in C(\Omega)$ , there exists a two-layer neural network  $f_m(\mathbf{x};\theta') = \sum_{j=1}^{m} a_j \sigma(\mathbf{u}_j^\top \mathbf{x} + c_j)$  such that  $\sup_{\mathbf{x} \in \Omega} |f_m(\mathbf{x};\theta') - h^*(\mathbf{x})| \leqslant \epsilon$ . Then our proof is completed by showing that we can construct appropriate  $\mathcal{T}_L$  and  $\mathcal{M}_o$  such that the deep CNN  $h_\theta$  can simulate the two-layer neural network  $f_m(\cdot; \theta')$ .
1. set  $C_L = m$ ,  $b_j^{(L)} = c_j$ . For odd  $i$ , let  $i' = (i + 1)/2$  and set  $W_{i,j,:}^{(L)} = ((\mathbf{u}_j)_{i'}, (\mathbf{u}_j)_{i' + 2d})$  and for even  $i$  we set  $W_{j,i,:}^{(L)} = -W_{j,i - 1,:}^{(L)}$ . Under this construction,
$$
\begin{array}{l} (\mathbf {z} ^ {(L)} (\mathbf {x})) _ {1, j} = \sigma \left(\sum_ {i = 1} ^ {C _ {L - 1}} \left(\mathbf {z} ^ {(L - 1)} (\mathbf {x})\right) _ {:, i} * _ {s} W _ {i, j,:} ^ {(L)} + b _ {j} ^ {(L)}\right) \\ = \sigma \left(\sum_ {i = 1} ^ {4 d} \left(\mathbf {u} _ {j}\right) _ {i} \left(\sigma \left(x _ {i}\right) - \sigma \left(- x _ {i}\right)\right) + c _ {j}\right) \\ = \sigma \left(\mathbf {u} _ {j} ^ {\top} \mathbf {x} + c _ {j}\right) \\ \end{array}
$$
2. Set  $W_{o} = (a_{1},\ldots ,a_{m})$ . Then, the CNN  $h_\theta$  represents exactly the same function as  $f_{m}(\cdot ;\theta^{\prime})$ :
$$
h _ {\theta} (\mathbf {x}) = \mathcal {M} _ {o} \circ \mathbf {z} ^ {(L)} (\mathbf {x}) = \sum_ {j = 1} ^ {m} a _ {j} (\mathbf {z} ^ {(L)} (\mathbf {x})) _ {1, j} = \sum_ {j = 1} ^ {m} a _ {j} \sigma (\mathbf {u} _ {j} ^ {\top} \mathbf {x} + c _ {j}) = f _ {m} (\mathbf {x}; \theta^ {\prime}).
$$
D.2 Proof of Proposition 3.2
Given two functions  $f, g$  over  $\mathcal{X}$ , let  $\rho_{\infty}(f,g) \coloneqq \sup_{\mathbf{x} \in \mathcal{X}} |f(\mathbf{x}) - g(\mathbf{x})|$ .
We prove this theorem by contradiction. First, any CNN  $h_\theta$  with the depth  $L \leqslant \log_2(d) + 1$  can be represented as
$$
h _ {\theta} (\mathbf {x}) = g _ {1} \left(x _ {1}, \dots , x _ {2 d}\right) + g _ {2} \left(x _ {2 d + 1}, \dots , x _ {4 d}\right) \tag {69}
$$
for some  $g_{1}$  and  $g_{2}$ . This is because that the size of receptive field is no greater than  $2d$ . Obviously,  $h_{\theta}$  cannot represent long-range functions like  $h^{*}(\mathbf{x}) \coloneqq x_{1}x_{2d + 1}$ . Specifically, we have
$$
\inf  _ {\theta} \rho_ {\infty} \left(h _ {\theta}, h ^ {*}\right) \geqslant \inf  _ {g _ {1}, g _ {2}} \sup  _ {\mathbf {x} \in \mathcal {X}} \left| g _ {1} \left(x _ {1}, \dots , x _ {2 d}\right) + g _ {2} \left(x _ {2 d + 1}, \dots , x _ {4 d}\right) - x _ {1} x _ {2 d + 1} \right| \tag {70}
$$
If there exist  $g_{1}$  and  $g_{2}$  such that
$$
\sup  _ {\mathbf {x} \in \mathcal {X}} \left| g _ {1} \left(x _ {1}, \dots , x _ {2 d}\right) + g _ {2} \left(x _ {2 d + 1}, \dots , x _ {4 d}\right) - x _ {1} x _ {2 d + 1} \right| \leqslant \frac {1}{8}, \tag {71}
$$
then taking special  $\mathbf{x}$  's gives
$$
\left| g _ {1} (0, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) \right| \leqslant \frac {1}{8}
$$
$$
\left| g _ {1} (0, 0, \dots , 0) + g _ {2} (1, 0, \dots , 0) \right| \leqslant \frac {1}{8} \tag {72}
$$
$$
\left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) \right| \leqslant \frac {8}{8} \tag {72}
$$
$$
\left| g _ {1} (1, 0, \dots , 0) + g _ {2} (1, 0, \dots , 0) - 1 \right| \leqslant \frac {1}{8}
$$
From the first three inequalities, we have
$$
\begin{array}{l} \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (1, 0, \dots , 0) \right| \\ = \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) - g _ {2} (0, 0, \dots , 0) - g _ {1} (0, 0, \dots , 0) + g _ {1} (0, 0, \dots , 0) + g _ {2} (1, 0, \dots , 0) \right| \\ \leqslant \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) \right| + \left| g _ {2} (0, 0, \dots , 0) + g _ {1} (0, 0, \dots , 0) \right| + \left| g _ {1} (0, 0, \dots , 0) + g _ {2} (1, 0, \dots , 0) \right| \\ \leqslant \frac {3}{8}, \\ \end{array}
$$
which is contradictory to the fourth inequality in (72). Thus (71) cannot hold and we complete the proof.  $\square$
D.3 Proof of Proposition 3.3
We first recall that  $\rho_{\infty}(f,g)\coloneqq \sup_{\mathbf{x}\in \mathcal{X}}|f(\mathbf{x}) - g(\mathbf{x})|$
We prove this theorem by contradiction. First, without downsampling, any CNN with depth  $L \leqslant 4d - 2$  can be represented as
$$
h _ {\theta} (\mathbf {x}) = g _ {1} \left(x _ {1}, x _ {2}, \dots , x _ {4 d - 1}\right) + g _ {2} \left(x _ {2}, x _ {3}, \dots , x _ {4 d}\right), \tag {73}
$$
for some  $g_1, g_2$ . In this form,  $x_{4d}$  and  $x_1$  do not have direct correlation and intuitively, it should be impossible to represent functions like  $h^*(\mathbf{x}) \coloneqq x_1 x_{4d}$  by using this CNN. This intuition can be made rigorously as follows. Note that
$$
\inf  _ {\theta} \rho_ {\infty} \left(h _ {\theta}, h ^ {*}\right) \geqslant \inf  _ {g _ {1}, g _ {2}} \sup  _ {\mathbf {x} \in \mathcal {X}} \left| g _ {1} \left(x _ {1}, \dots , x _ {4 d - 1}\right) + g _ {2} \left(x _ {2}, \dots , x _ {4 d}\right) - x _ {1} x _ {4 d} \right| \tag {74}
$$
If there is  $g_{1}, g_{2}$  such that
$$
\sup  _ {\mathbf {x} \in \mathcal {X}} \left| g _ {1} \left(x _ {1}, x _ {2}, \dots , x _ {4 d - 1}\right) + g _ {2} \left(x _ {2}, x _ {4}, \dots , x _ {4 d}\right) - x _ {1} x _ {4 d} \right| \leqslant \frac {1}{8}, \tag {75}
$$
$$
\left| g _ {1} (0, 0, \dots , 0) + g _ {2} (0, 0, \dots , 1) \right| \leqslant \frac {1}{8} \tag {76}
$$
$$
\left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) \right| \leqslant \frac {1}{8} \tag {76}
$$
$$
\left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 1) - 1 \right| \leqslant \frac {1}{8}
$$
$$
\begin{array}{l} \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 1) \right| \\ = \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) - g _ {2} (0, 0, \dots , 0) - g _ {1} (0, 0, \dots , 0) + g _ {1} (0, 0, \dots , 0) + g _ {2} (0, 0, \dots , 1) \right| \\ \leqslant \left| g _ {1} (1, 0, \dots , 0) + g _ {2} (0, 0, \dots , 0) \right| + \left| g _ {2} (0, 0, \dots , 0) + g _ {1} (0, 0, \dots , 0) \right| + \left| g _ {1} (0, 0, \dots , 0) + g _ {2} (0, 0, \dots , 1) \right| \\ \leqslant \frac {3}{8}, \\ \end{array}
$$
which is contradictory to the fourth inequalities in (76). Therefore, (75) cannot hold and we complete the proof.  $\square$
E Learning Sparse Functions: Proofs of Section 4
E.1 Proof of Lemma 4.4
Here we prove a complete version of Lemma 4.4.