SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: A diagram illustration of how CNNs select coordinates adaptively. In this case  $d = 8$ ,  $L = 3$ . The nonzero coordinate is  $i = 4$ , for which  $a_0 = 1$ ,  $a_1 = 1$ ,  $a_2 = 0$ . The values on edges represent the weights, which are set according to the proof of Lemma 4.2.
Remark 4.3. Lemma 4.2 indicates that deep CNNs are able to effectively identify any  $k$  critical coordinates with  $\mathcal{O}(\log d)$  depth and  $\mathcal{O}(k^2\log d)$  parameters, which is significantly smaller than  $d$ . The key to this achievement is the adaptivity of neural networks, in combination with weight sharing, multichanneling, downsampling, and depth, as demonstrated in the proof. Specifically, downsampling and increased depth allow for capturing long-range sparse correlations with only  $\mathcal{O}(\log d)$  depth, multichanneling facilitates the storage of information regarding different critical coordinates, and weight sharing ensures the number of parameters of each layer to be independent of  $d$ .
We now proceed to consider the learning of nonlinear sparse functions. We first need the following feature selection result for ReLU CNNs. The proof is similar to the linear case (Lemma 4.2) and deferred to Appendix E.1.
Lemma 4.4. Given  $k, m \in \mathbb{N}$ , consider a ReLU CNN with depth  $L = \log_2(4d)$  and the channel numbers  $C_l = 2k$  for all  $l \in [L - 1]$  and  $C_L = m$ . Then for any  $\mathcal{I} = (i_1, \ldots, i_k) \subset [4d]$ ,  $\mathbf{u}_1, \ldots, \mathbf{u}_m \in \mathbb{R}^k$ , and  $c_1, \ldots, c_m \in \mathbb{R}$ , there exists  $\theta \in \Theta$  such that the  $L$ -th layer outputs:
$$
\mathbf {z} ^ {(L)} (\mathbf {x}) = \left(\sigma \left(\mathbf {u} _ {1} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {1}\right), \dots , \sigma \left(\mathbf {u} _ {m} ^ {\top} \mathbf {x} _ {\mathcal {I}} + c _ {m}\right)\right) \in \mathbb {R} ^ {1 \times m}. \tag {4}
$$
Furthermore, this CNN has  $\mathcal{O}(k^2\log d + km)$  parameters.
According to this lemma, deep CNNs are capable of generating adaptive features of the form:  $\mathbf{x} \mapsto \sigma(\mathbf{u}^\top \mathbf{x}_{\mathcal{I}} + c)$  with only  $\mathcal{O}(\log d)$  depth and  $\mathcal{O}(k^2 \log d)$  parameters, where the features depend only on the  $k$  critical coordinates. Note that linear combinations of this type of features give two-layer networks. Therefore, functions of the form  $f(\mathbf{x}) \coloneqq g(\mathbf{x}_{\mathcal{I}})$  with  $g: \mathbb{R}^k \mapsto \mathbb{R}$  can be well approximated by deep CNNs, as long as  $g$  can be well approximated by two-layer ReLU networks. To measure the learnability of  $g$ , we adopt the Barron regularity as proposed in E et al. (2022).
Definition 4.5 (Barron space). Consider functions admitting the following integral representation  $g_{\rho}(\mathbf{x}) = \int_{\Omega}a\sigma \left(\mathbf{u}^{\top}\mathbf{x} + c\right)\rho (\mathrm{da},\mathrm{du},\mathrm{dc})$  for all  $\mathbf{x}\in \mathcal{X}$ , where  $\sigma = \operatorname {ReLU},\Omega = \mathbb{R}^{1}\times \mathbb{R}^{k}\times \mathbb{R}^{1}$  and  $\rho \in \mathcal{P}(\Omega)$ . For a function  $g:\mathcal{X}\mapsto \mathbb{R}$ , denote by  $A_{g} = \{\rho :g_{\rho}(\mathbf{x}) = g(\mathbf{x}) \text{ for all } \mathbf{x}\in \mathcal{X}\}$ . Then, we define
$$
\| g \| _ {\mathcal {B}} = \inf  _ {\rho \in A _ {f}} \mathbb {E} _ {(a, \mathbf {u}, c) \sim \rho} \left[ | a | \left(\| \mathbf {u} \| _ {1} + | c |\right) \right], \qquad \mathcal {B} = \left\{g: \| g \| _ {\mathcal {B}} <   \infty \right\}.
$$
More explicitly, the Fourier-analytic characterization (Barron, 1993; Klusowski and Barron, 2016) showed that  $\| g\|_{\mathcal{B}}\lesssim \int (1 + \| \xi \| _1)^2 |\hat{g} (\xi)|\mathrm{d}\xi$ , where  $\hat{g}$  denotes the Fourier transform of  $g$ . The analysis with Barron regularity is often much simpler (E et al., 2019; 2022) and can yield a better dependence on  $k$  than using traditional smoothness measures. Specifically, the Barron regularity allows obtaining generalization bounds with dimension-independent rates like  $\mathcal{O}(n^{-1 / 2})$ . In contrast, for traditional smoothness measures such as assuming  $g\in C^s ([0,1]^k)$ , the resulting error rate would scale like  $\mathcal{O}(n^{-s / k})$ , which depends on  $k$  exponentially.
Recall  $\ell_B(y,y') = \frac{1}{2} (y - y')^2\wedge \frac{1}{2} B^2$ . Consider the regularized estimator given by  $\hat{\theta}_n = \mathrm{argmin}_{\theta}\left(\frac{1}{n}\sum_{i = 1}^{n}\ell_B(\pi_A\circ h_\theta (\mathbf{x}_i),y_i) + \lambda \| \theta \|_{\mathcal{P}}\right)$ , where  $A,B,\lambda$  are hyperparameters to be tuned.
Theorem 4.6. Let  $\mathcal{X} = [0,1]^{4d}$  and  $h^*(\mathbf{x}) \coloneqq g(\mathbf{x}_{\mathcal{I}})$  for some  $\mathcal{I} \subset [4d]$  be the target function. Let  $k = |\mathcal{I}|$  and suppose  $g \in \mathcal{B}$ ,  $\sup_z |g(z)| \leqslant 1$ , and  $d \geqslant 3$ . For any  $\delta \in (0,1/2)$ ,  $\epsilon \in (0,1/2)$ , there is a choice of  $A, B$ , and  $\lambda$  such that w.p. at least  $1 - 2\delta$  over the sampling of training set we have  $\left\| \pi_A \circ h_{\hat{\theta}_n} - h^*\right\|_{L^2(P)}^2 \leqslant \epsilon$ , whenever
$$
n \geqslant \operatorname {p o l y} (\| g ^ {*} \| _ {\mathcal {B}}, k, \sigma , \log \frac {1}{\delta}, \log \frac {1}{\epsilon}) \left(\frac {\log (d) (\log \log d) ^ {3}}{\epsilon^ {3}} + \frac {\log^ {2} (d) (\log \log d) ^ {3}}{\epsilon^ {2}}\right).
$$
The proof is presented in Appendix E.2. This theorem shows that learning sparse functions with deep CNNs requires only  $\widetilde{\mathcal{O}} (\log^2 d)$  samples, which is nearly optimal from an information-theoretic perspective. This is because Han and Yuan (2020) proved that learning sparse functions requires at least  $\Omega (\log d)$  samples. It is also important to mention that Cagnetta et al. (2022) showed that using deep convolutional kernels to learn long-range sparse functions suffers the curse of dimensionality. The comparison with our results also highlights the significance of adaptivity in neural networks.
Experimental validation. In this experiment, both short-range and long-range sparse target functions are considered. We set the input dimension  $d = 4096$ , the sample size  $n = 400$  and the noise level  $\sigma$  to be zero. For the CNN architecture, the filter size is  $s = 4$ , resulting in a depth  $L = \log_4(d) = 6$ ; the number of channels is set to  $C = 4$  across all layers. The Adam optimizer is employed to our models, and importantly, no regularization is applied. As a comparison, we also examine two-layer fully-connected networks (FCNs) with a width 10, as well as the ordinary least linear regression (OLS). The results are shown in Figure 2. One can see clearly that even without any explicit sparsity regularization, CNN can still learn sparse interactions efficiently in both short-range and long-range scenarios. In contrast, FCN and OLS overfit the data and fail to generalize to test data.
Figure 2: CNN can learn sparse functions efficiently. Both short-range (left) and long-range (right) sparse target functions are considered in this experiment. The training is stopped when the training loss drops below  $10^{-5}$ .
5 Disentangle the Inductive Biases of Weight Sharing and Locality
To facilitate our statement, in this section, we denote by  $h_{\theta}^{\mathrm{cnn}}$ ,  $h_{\theta}^{\mathrm{lcn}}$ ,  $h_{\theta}^{\mathrm{fcn}}$  the CNN, LCN, and FCN model, respectively. We shall consider the following task.
Separation Task. Suppose the input distribution  $P = \mathcal{N}(0, I_{4d})$  and the target function  $\bar{h}^*(\mathbf{x}) = \pi_{A_0} \circ h^*(\mathbf{x})$  where  $A_0$  is a universal absolute constant to be specified later and
$$
h ^ {*} (\mathbf {x}) = \frac {1}{d} \left(\sum_ {i = 1} ^ {d} \left(x _ {2 i - 1} ^ {2} - x _ {2 i} ^ {2}\right)\right) \left(\sum_ {i = 1} ^ {d} \left(x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2}\right)\right). \tag {5}
$$
The truncation is employed to ensure boundedness of the output. However, we believe that a more refined analysis could eliminate the need for this constraint. This target function possesses the structures of both "weight sharing" and "locality" (the global sum can be computed hierarchically through local additions). We then consider the following comparisons.
- CNNs vs. LCNs. This comparison allows us to isolate the bias of weight sharing as both models have the same structures, but LCNs do not have weight sharing.
- LCNs vs. FCNs. This comparison allows us to isolate the bias of locality since the only difference between FCNs and LCNs is the presence or absence of locality.
By these comparisons, we can evaluate the effectiveness of each bias in contributing to the improved performance of CNNs over FCNs, and gain insights into how these biases interact with other structures such as depth, multichanneling, and downsampling.
To establish provable separations, we need both upper and lower bounds of the sample complexity of learning  $\bar{h}^*$ . For the upper bounds, we consider the regularized estimator given by
$$
\hat {\theta} _ {n} = \underset {\theta} {\operatorname {a r g m i n}} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \ell_ {B} \left(\pi_ {A} \circ h _ {\theta} (\mathbf {x} _ {i}), y _ {i}\right) \right] + \lambda \| \theta \| _ {\mathcal {P}}\right), \tag {6}
$$
where the model  $h_\theta$  and hyperparameters  $A, B$ , and  $\lambda$  will be specified for each comparison. We will use the covering number-based technique (see Appendix B.2) to upper bound the generalization error. See also Appendix C for how to bound the covering numbers of deep CNNs and deep LCNs.
Establishing lower bounds is often much more challenging. We will adopt a similar approach to Ng (2004); Li et al. (2020); Abbe and Boix-Adserà (2022), which involves understanding the learning hardness through the equivariance group of the learning algorithm.
5.1 Learning Algorithm, Group Equivariance, and Lower Bounds
A learning algorithm  $\mathbb{A}:(\mathcal{X}\times \mathcal{Y})^n\mapsto \mathcal{M}(\Theta)$  generates a random variable taking values in  $\Theta$  by using the training set  $S_{n}$ . This definition includes the situation where our models are returned by stochastic optimizers such as SGD and Adam. We will denote  $X\stackrel {\mathrm{d}}{=}Y$  as  $\operatorname {Law}(X) = \operatorname {Law}(Y)$ .
$G$ -equivariant/-invariant. Let  $G$  be a group acting on  $\mathcal{X}$ . A learning algorithm  $\mathbb{A}$  is said to be  $G$ -equivariant if  $\forall \{\mathbf{x}_i, y_i\}_{i=1}^n \in (\mathcal{X} \times \mathcal{Y})^n$  and  $\forall \tau \in G, \mathbf{x} \in \mathcal{X}$ :
$$
h _ {\mathbb {A} \left(\left\{\tau \left(\mathbf {x} _ {i}\right), y _ {i} \right\} _ {i = 1} ^ {n}\right)} \circ \tau \stackrel {{\mathrm {d}}} {{=}} h _ {\mathbb {A} \left(\left\{\mathbf {x} _ {i}, y _ {i} \right\} _ {i = 1} ^ {n}\right)}. \tag {7}
$$
A distribution  $\mu$  is said to be  $G$ -invariant if for any  $\tau \in G$ ,  $\tau(X) \sim \mu$  if  $X \sim \mu$ .
Sample Complexity. Given a target function class  $\mathcal{F}$  and a learning algorithm  $\mathbb{A}$ , for a target accuracy  $\epsilon > 0$ , the sample complexity  $\mathcal{C}(\mathbb{A}, \mathcal{F}, \epsilon)$  is defined as the smallest integer  $n$  such that
$$
\sup  _ {h ^ {*} \in \mathcal {F}} \mathbb {E} \left\| h _ {\mathbb {A} (S _ {n})} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \epsilon , \tag {8}
$$
where the expectation is taken over both the sampling of  $S_{n}$  and the randomness in  $\mathbb{A}$ . In addition, we define the minimax sample complexity by  $\bar{\mathcal{C}}(\mathcal{F},\epsilon)\coloneqq \inf_{\mathbb{A}}\mathcal{C}(\mathbb{A},\mathcal{F},\epsilon)$ , where the infimum is taken over all learning algorithms, i.e., all the mappings:  $(\mathcal{X}\times \mathcal{Y})^n\mapsto \mathcal{M}(\Theta)$ .
For a function class  $\mathcal{F}$ , denote by  $\mathcal{F} \circ G \coloneqq \{f \circ \tau : f \in \mathcal{F}, \tau \in G\}$  the  $G$ -enlarged class.
Lemma 5.1. (A restatement of (Li et al., 2020, Lemma D.1)) Let  $\mathcal{A}_G$  be the set of all  $G$ -equivariant algorithms. For any function class  $\mathcal{F}$  and  $\epsilon > 0$ , it holds that  $\inf_{\mathbb{A} \in \mathcal{A}_G} \mathcal{C}(\mathbb{A}, \mathcal{F}, \epsilon) \geqslant \bar{\mathcal{C}}(\mathcal{F} \circ G, \epsilon)$ .
This lemma shows that the sample complexity of learning  $\mathcal{F}$  with a  $G$ -equivalent algorithm can be lower bounded by the minimax complexity of learning the enlarged class:  $\mathcal{F} \circ G$ . The latter can be handled using standard approaches from minimax theory (see (Wainwright, 2019, Section 15)). Specifically, we will adopt Fano's method to lower-bound the minimax sample complexity, which reduces the problem to find a proper packing of the enlarged class  $\mathcal{F} \circ G$  (see Appendix B.3). In particular, when  $\mathcal{F} = \{f^{*}\}$ , the packing number of  $\mathcal{F} \circ G$  can be completely determined by the packing number of the symmetry group  $G$  if  $f^{*}$  is chosen to satisfy
$$
\left\| f ^ {*} \circ \tau_ {1} - f ^ {*} \circ \tau_ {2} \right\| _ {L ^ {2} (\mu)} ^ {2} \sim \left\| \tau_ {1} - \tau_ {2} \right\| _ {G} ^ {2}, \tag {9}
$$
where  $\| \cdot \|_G$  is a proper norm defined on  $G$ . Moreover, estimating the packing number of a symmetry group is often much easier.
<sup>1</sup>The definition here is intuitive but not fully rigorous. We refer to Appendix F.1 for a rigorous definition.
Note that the specific target function (5) in our separation task is designed to satisfies (9) for the symmetry groups associated with SGD algorithms and thus, the sample complexity can be completely governed by the size of symmetry groups. Specifically, for SGD algorithms, we have
- The symmetry group of FCNs is the orthogonal group  $G_{\mathrm{ort}}(4d)$ , whose degree of freedom is  $\Theta(d^2)$ . This explains the lower bound  $\Omega(d^2)$  for FCNs.
- The symmetry group of LCNs is the local permutation group  $G_{\mathrm{loc}}$ , whose degree of freedom is  $\Theta(d)$ . This explains the lower bound  $\Omega(d)$  for LCNs.
See the sections below for more details.
5.2 CNNs vs. LCNs
In this section, we will derive an upper bound of sample complexity for learning  $\bar{h}^*$  with CNNs and a lower bound for learning  $\bar{h}^*$  with LCNs. The two bounds together provide a quantification of the effect of weight sharing.
Consider the deep CNN  $h_{\theta}^{\mathrm{cnn}}$  with  $L = \log_2(4d)$ ,  $C_1 = C_L = 4$ ,  $C_l = 2$  for  $l \in [2, L - 1]$ , and  $\sigma_1(x) = \sigma_L(x) = \mathrm{ReLU}^2(x)$ ,  $\sigma_l(x) = \mathrm{ReLU}(x)$  for  $l \in [2, L - 1]$ .
Theorem 5.2 (Upper bound of CNNs). Suppose  $d \geqslant 3$  and  $A_0 \geqslant 0$  and consider the estimator  $\hat{\theta}_n$  given by (6). For any  $\epsilon \in (0,1/2)$  and  $\delta \in (0,1/2)$ , there is a choice of  $A, B,$  and  $\lambda$  such that  $\left\| \pi_A \circ h_{\hat{\theta}_n}^{\mathrm{cnn}} - \bar{h}^* \right\|_{L^2(P)}^2 \leqslant \epsilon$  holds w.p. at least  $1 - 2\delta$ , whenever
$$
n \geqslant \operatorname {p o l y} \left(A _ {0}, \sigma , \log (1 / \delta), \log (1 / \epsilon)\right) \epsilon^ {- 2} \left(\log^ {2} d\right) \left(\log \log d\right) ^ {3}.
$$
The proof is deferred to Appendix G.2. It is shown that CNNs need only  $\tilde{\mathcal{O}} (\log^2 d)$  samples to learn  $\bar{h}^{*}$ . The reason behind this upper bound is that  $\bar{h}^{*}$  can be accurately represented by deep CNNs with only  $\mathcal{O}(\log d)$  parameters and well-controlled norms due to the well matching between the weight sharing and locality in CNNs and the unique characteristics of  $\bar{h}^{*}$ .
Next we turn to establish a lower bound for LCNs. Let  $\ell(\cdot, \cdot)$  be a general differentiable loss function and  $\hat{L}^{\mathrm{lcn}}(\theta) \coloneqq \frac{1}{n}\sum_{i=1}^{n}\ell(h_{\theta}^{\mathrm{lcn}}(\mathbf{x}_i), y_i) + r\left(\|\theta\|_p\right)$ , where  $p \geqslant 1$  and  $r: [0, +\infty) \to [0, +\infty)$ .
Denote by  $\mathbb{A}_T^{\mathrm{lcn}}$  the algorithm that returns solutions by optimizing  $\hat{L}^{\mathrm{lcn}}(\cdot)$  for  $T$  steps using SGD or Adam. Under mild conditions, it can be shown that  $\mathbb{A}_T^{\mathrm{lcn}}$  is equivariant under the local permutation group:
$$
G _ {\mathrm {l o c}} = \left\{U \in \mathbb {R} ^ {4 d \times 4 d}: U = \operatorname {d i a g} \left(U _ {1}, U _ {2}, \dots , U _ {2 d}\right), U _ {i} \in \left\{\left( \begin{array}{c c} 1 & 0 \\ 0 & 1 \end{array} \right), \left( \begin{array}{c c} 0 & 1 \\ 1 & 0 \end{array} \right) \right\} \right\}. \tag {10}
$$
Obviously,  $G_{\mathrm{loc}}$  is a group under matrix product. To ensure the  $G_{\mathrm{loc}}$ -equivariance of  $\mathbb{A}_T^{\mathrm{len}}$ , we need the following assumption, which is satisfied by all popular initialization schemes used in practice.
Assumption 1. At initialization, the fist layer weight satisfies:  $W_{j,1,2k-1}^{(1)}$  and  $W_{j,1,2k}^{(1)}$  have the same marginal distribution for any  $j \in [C_1], k \in [2d]$ .
Lemma 5.3. Suppose that the input distribution  $P$  is  $G_{\mathrm{loc}}$ -invariant and Assumption 1 is satisfied. Then, for any  $T \in \mathbb{N}$ ,  $\mathbb{A}_T^{\mathrm{len}}$  is  $G_{\mathrm{loc}}$ -equivariant for both SGD and Adam.
The proof is deferred to Appendix F.2. Xiao and Pennington (2022) showed that under Gaussian initialization, the equivariance group of  $\mathbb{A}_T^{\mathrm{lcn}}$  is  $O(2)\otimes I_{2d}$ , which is slightly larger than  $G_{\mathrm{loc}}$  obtained in the current work. It should be stressed that  $G_{\mathrm{loc}}$  is obtained under a much milder assumption of initialization which holds for the popular uniform initialization scheme. Moreover, we show below that the  $G_{\mathrm{loc}}$  equivariance is sufficient for separating LCNs from CNNs.
Theorem 5.4 (Lower bound of LCNs). Under Assumption 1, there exists absolute constants  $C, c > 0$  such that  $\forall T \in \mathbb{N}, \epsilon_0 \in (0, c], d, A_0 \geqslant C$ :
$$
\mathcal {C} \left(\mathbb {A} _ {T} ^ {\mathrm {l c n}}, \{\bar {h} ^ {*}, \epsilon_ {0}\right) = \Omega (\sigma^ {2} d)
$$
The proof is deferred to Appendix G.1. By comparing Theorem 5.2 and 5.4, we see that the weight sharing in CNNs yields an  $\widetilde{\Theta}(d)$  improvement on the sample complexity for learning  $\bar{h}^*$ . This highlights the importance of exploiting the "weight sharing" in  $\bar{h}^*$  for efficient learning.
5.3 LCNs vs. FCNs
We will derive an upper bound of sample complexity for learning  $\bar{h}^*$  with LCNs and a lower bound for learning  $\bar{h}^*$  with FCNs. The two bounds together provide a quantification of the effect of locality.
We consider the deep LCN  $h_{\theta}^{\mathrm{lcn}}$  with  $L = \log_2(4d)$ ,  $C_1 = C_L = 4$ ,  $C_l = 2$  for  $l \in [2, L - 1]$ , and  $\sigma_1(x) = \sigma_L(x) = \mathrm{ReLU}^2(x)$ ,  $\sigma_l(x) = \mathrm{ReLU}(x)$  for  $l \in [2, L - 1]$ .
Theorem 5.5 (Upper bound of LCNs). Suppose  $d \geqslant 3$  and  $A_0 \geqslant 0$  and consider the estimator  $\hat{\theta}_n$  given by (6). Then, for any  $\epsilon \in (0,1/2)$  and  $\delta \in (0,1/2)$ , there is a choice of  $A, B$ , and  $\lambda$  such that  $\left\| \pi_A \circ h_{\hat{\theta}_n}^{\mathrm{lcn}} - \bar{h}^* \right\|_{L^2(P)}^2 \leqslant \epsilon$  holds w.p. at least  $1 - 2\delta$ , whenever
$$
n \geqslant \operatorname {p o l y} \left(A _ {0}, \sigma , \log (1 / \delta), \log (1 / \epsilon)\right) \epsilon^ {- 2} d \log^ {4} d
$$
The proof is deferred to Appendix H.2. It is shown that LCNs needs  $\widetilde{O}(d)$  samples to learn  $\bar{h}^*$ . The reason behind this upper bound is that  $\bar{h}^*$  can be well approximated by LCNs with  $\mathcal{O}(d)$  parameters and well-controlled norm. However, as opposed to CNNs, LCNs lack weight sharing and can only imitate the local features of  $\bar{h}^*$  with the "weight sharing" structure completely ignored.
Next we turn to establish a lower bound for FCNs. Let  $\ell(\cdot, \cdot)$  be a general differentiable loss function. Denote by  $\mathbb{A}_T^{\mathrm{fcn}}$  the algorithm: Run SGD for  $T$  steps by optimizing  $\hat{L}^{\mathrm{fcn}}(\theta) := \frac{1}{n} \sum_{i=1}^{n} \ell(h_{\theta}^{\mathrm{fcn}}(\mathbf{x}_i), y_i) + r(\|\theta\|_2)$ , where  $r: [0, +\infty) \to [0, +\infty)$  is a general penalty function. We refer to Appendix F.1 for a rigorous definition.
Lemma 5.6. Suppose that the input distribution  $P$  is  $G_{\text{ort}}(4d)$ -invariant and entries of  $W^{(1)}$  are i.i.d. initialized from  $\mathcal{N}(0, \beta^2)$  for some  $\beta > 0$ . Then, for any  $T \in \mathbb{N}$ ,  $\mathbb{A}_T^{\mathrm{fen}}$  is  $G_{\text{ort}}(4d)$ -equivariant.
This lemma was proved in (Li et al., 2020, Corollary C.2) and we state it here for completeness. It implies that training FCNs with SGD induces a larger equivariance group than that of LCNs since  $G_{\mathrm{loc}} \subset G_{\mathrm{ort}}(4d)$ . It is important to note that this lemma only holds for SGD, as Adam is only permutation invariant. In contrast, the result in Lemma 5.3 applies to both SGD and Adam.
Theorem 5.7 (Lower bound of FCNs). Suppose that the input distribution  $P$  is  $G_{\text{ort}}(4d)$ -invariant and entries of  $W^{(1)}$  are i.i.d. initialized from  $\mathcal{N}(0, \beta^2)$  for some  $\beta > 0$ . Then there exists absolute constants  $C, c > 0$  such that  $\forall T \in \mathbb{N}, \epsilon_0 \in (0, c], d, A_0 \geqslant C$ :
$$
\mathcal {C} \left(\mathbb {A} _ {T} ^ {\mathrm {f c n}}, \{\bar {h} ^ {*} \}, \epsilon_ {0}\right) = \Omega \left(\sigma^ {2} d ^ {2}\right)
$$
The proof is deferred to Appendix H.1. By comparing Theorem 5.5 and 5.7, we see that the locality itself yields another  $\widetilde{\Theta}(d)$  improvement on the sample complexity for learning  $\bar{h}^*$ . This highlights the importance of exploiting the locality in  $\bar{h}^*$  for efficient learning.
In this paper, we delve into the theoretical analysis of the inductive biases associated with the multichanneling, downsampling, weight sharing, and locality in deep CNN architectures with a focus on understanding the interplay between network depth and these biases. Our results highlight the critical role of multichanneling and downsampling in enabling deep CNNs to effectively capture long-range correlations. We also analyze the effects of weight sharing and locality on breaking the learning algorithm's symmetry, through which we make a clear distinction between the two biases: the global orthogonal equivariance vs. the local permutation equivariance. By leveraging these symmetries, we establish provable separations for FCNs vs. LCNs and LCNs vs. CNNs, providing a strong theoretical support for the unique nature of weight sharing and locality.
Lei Wu is supported in part by the National Key R&D Program of China (No. 2022YFA1008200).
Lechao Xiao and Jeffrey Pennington. Synergy and symmetry in deep learning: Interactions between the data, model, and inference algorithm. In International Conference on Machine Learning, 2022. 3, 9, 15, 38
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations, 2016. 15
Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks, 124:319-327, 2020a. 1, 2
Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis, 48(2):787-794, 2020b. 1, 2, 15
A Related Works 15
B Technical Background 15
B.1 Toolbox for Concentrations 15
B.2 A Generalization Bound for Learning with Unbounded Noise 17
B.3 Fano's Method for Random Estimators 21
C Capacity Control for CNNs and LCNs 23
C.1 CNNs 24
C.2 LCNs 26
C.3 Comparison with Long and Sedghi (2019). 27
D Universal Approximation: Proofs in Section 3 28
D.1 Proof of Theorem 3.1 28
D.2 Proof of Proposition 3.2 30
D.3 Proof of Proposition 3.3 31
E Learning Sparse Functions: Proofs of Section 4 32
E.1 Proof of Lemma 4.4 32
E.2 Proof of Theorem 4.6 34
F Symmetries of Learning Algorithm and Proofs of Lemma 5.3 and 5.6 35