SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihao Wang
zihaowang@stu.pku.edu.cn
Lei Wu
leiwu@math.pku.edu.cn
In this paper, we provide a theoretical analysis of the inductive biases in convolutional neural networks (CNNs). We start by examining the universality of CNNs, i.e., the ability to approximate any continuous functions. We prove that a depth of  $\mathcal{O}(\log d)$  suffices for deep CNNs to achieve this universality, where  $d$  in the input dimension. Additionally, we establish that learning sparse functions with CNNs requires only  $\widetilde{\mathcal{O}} (\log^2 d)$  samples, indicating that deep CNNs can efficiently capture long-range sparse correlations. These results are made possible through a novel combination of the multichanneling and downsampling when increasing the network depth. We also delve into the distinct roles of weight sharing and locality in CNNs. To this end, we compare the performance of CNNs, locally-connected networks (LCNs), and fully-connected networks (FCNs) on a simple regression task, where LCNs can be viewed as CNNs without weight sharing. On the one hand, we prove that LCNs require  $\Omega (d)$  samples while CNNs need only  $\widetilde{\mathcal{O}} (\log^2 d)$  samples, highlighting the critical role of weight sharing. On the other hand, we prove that FCNs require  $\Omega (d^{2})$  samples, whereas LCNs need only  $\widetilde{\mathcal{O}} (d)$  samples, underscoring the importance of locality. These provable separations quantify the difference between the two biases, and the major observation behind our proof is that weight sharing and locality break different symmetries in the learning process.
Convolutional neural networks (CNNs) (Fukushima, 1988; LeCun et al., 1998) are a fundamental model in deep learning, known for their exceptional performance in many tasks. In particular, CNNs consistently outperform the fully-connected neural network (FCN) counterparts in vision-related tasks (Krizhevsky et al., 2012; He et al., 2016; Huang et al., 2017). Uncovering the underlying mechanism behind the success of CNNs is thus of paramount importance in deep learning.
Zhou (2020a;b); Feng et al. (2022); He et al. (2022) studied the approximation capabilities of CNNs for target functions in spaces such as continuous functions and Sobolev spaces. Although these results are important, they cannot explain why CNNs perform better than FCNs. The primary reason for this limitation is that these works did not take into account the specific structures of CNNs, including multichanneling, downsampling, weight sharing, and locality. The locality refers to the use of small filters, e.g., filter sizes can be as small as 3 in popular VGG nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016). Comprehending the inductive biases of these architecture choices is critical to understand the exceptional performance of CNNs.
Li et al. (2020) designed a simple classification task to demonstrate the superiority of CNNs over FCNs. The authors prove that for this task, FCNs need  $\Omega(d^2)$  samples while CNNs need only  $\mathcal{O}(1)$  samples, thereby providing theoretical support for the superiority of using convolutions. However, this study neither examined the individual impact of weight sharing and locality nor considered the
inductive biases of multichanneling and downsampling. Additionally, the analysis was limited to shallow CNNs and did not examine the interaction between these structures and network depth.
In this work, we conduct a systematic analysis of the inductive biases associated with the specific structures of CNNs. Our main contributions are summarized as follows.
Universality. We establish the universality of deep CNNs with a depth of  $\mathcal{O}(\log d)$ . This is in contrast to existing works (Zhou, 2020a;b; He et al., 2022), where the universality requires a depth of at least  $\Omega(d)$ . The key to our improvement is an effective leveraging of the inductive biases of multichanneling and downsampling:
- Downsampling amplifies the size of the receptive field exponentially, thus explaining the need for logarithmic depth. Furthermore, we prove that if downsampling is not used, CNNs require a depth of at least  $\Omega(d)$ , demonstrating the cruciality of downsampling.
- Multichanneling serves as a mechanism for storing extracted information. By increasing the number of channels whenever the spatial dimension is reduced by downsampling, we ensure that no information is lost. This combination of multichanneling and downsampling is widely employed in practical CNNs, ranging from classical LeNet (LeCun et al., 1998) to modern VGG nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016).
It is worth mentioning that while studies like Poggio et al. (2017) and Cagnetta et al. (2022) have examined similar CNN architectures with  $O(\log d)$  depth, they did not explicitly establish universality as our work does. Specifically, Cagnetta et al. (2022) focused on the kernel associated with deep CNNs having infinitely many channels. In retrospect, one could potentially show that the deep-CNN kernel is universal by verifying that kernel has no zero eigenvalues.
Learning sparse functions. A function  $f: \mathbb{R}^d \mapsto \mathbb{R}$  is said to be sparse if it only depends on a few coordinates of the input, e.g.,  $f(\mathbf{x}) \coloneqq g(x_1, x_d)$  for some  $g: \mathbb{R}^2 \mapsto \mathbb{R}$ . We prove that learning sparse functions using CNNs requires only  $\widetilde{\mathcal{O}}(\log^2 d)$  samples, which is nearly optimal as the information-theoretic lower bound of learning such functions is  $\Omega(\log d)$  (Han and Yuan, 2020).
This result is surprising because it has been widely believed that CNNs struggle to capture long-range correlations. However, our findings suggest that CNNs can efficiently learn long-range sparse ones, which is a valuable attribute for many applications.
In addition, it is important to note that the near-optimal sample complexity of learning sparse functions using CNNs is achieved with only  $\mathcal{O}(\log d)$  depth and  $\mathcal{O}(k^2\log d)$  total parameters, where  $k$  is the number of critical coordinates in sparse functions. A lower bound is established to demonstrate the optimality of the depth requirement. The ability of CNNs to select any  $k$  coordinate with only  $\mathcal{O}(k^2\log d)$  total parameters is remarkable, especially considering that even in the linear case, LASSO (Tibshirani, 1996) requires  $\Omega(d)$  parameters. It is the synergy of increased depth, weight sharing, and multichannelling that gives CNNs this exceptional capability.
Disentangling the weight sharing and locality. We next study the inductive biases of locality and weight sharing by comparing the performance of CNNs, LCNs, and FCNs on a synthetic regression task. This allows us to separate the effects of weight sharing and locality.
- CNNs vs. LCNs. We prove that CNNs require only  $\widetilde{\mathcal{O}} (\log^2 d)$  samples to learn, while LCNs trained by SGD or Adam with standard initialization need  $\Omega (d)$  samples. This provides a separation between CNNs and LCNs and demonstrates the crucial role of weight sharing.
- LCNs vs. FCNs. We prove that LCNs requires only  $\widetilde{\mathcal{O}}(d)$  samples to learn, while FCNs trained by SGD with Gaussian initialization need  $\Omega(d^2)$  samples. This provably separates LCNs from FCNs and demonstrates the benefit of locality.
The difference in sample complexity can be attributed to the different symmetries encoded in the architecture biases. For instance, stochastic gradient descent (SGD) exhibits different symmetries for these models: a lack of equivariance for CNNs, a local permutation equivariance for LCNs, and a global orthogonal equivariance for FCNs. The size of the equivariance groups determines the
minimax sample complexity and distinguishes between different architectures. Similar ideas have been previously explored in Li et al. (2020); Xiao and Pennington (2022) to understand CNNs; but these works did not differentiate the roles of weight sharing and locality. For a detailed comparison with these works, see the related work section in Appendix A.
Technical contribution. The lower bounds for LCNs and FCNs are established using Fano's method from minimax theory (Wainwright, 2019, Section 15). However, different from the traditional statistical setup where the estimator is deterministic, the estimators produced by stochastic optimizers are random. To address this issue, we develop a variant of Fano's method for random estimators, which might be of independent interest. Further details can be found in Appendix B.3.
Related work. We refer to Appendix A for a detailed comparison with other related works.
We use  $\mathrm{poly}(z_1,\ldots ,z_p)$  to denote a quantity that depends on  $z_{1},\ldots ,z_{p}$  polynomially. We use  $X\lesssim Y$  to denote  $X\leqslant CY$  for some absolute constant  $C$  and  $X\gtrsim Y$  is defined analogously. We also use the standard big-O notations:  $\Theta (\cdot),\mathcal{O}(\cdot)$  and  $\Omega (\cdot)$ . In addition, we use  $\widetilde{\mathcal{O}}$  and  $\widetilde{\Omega}$  to hide higher-order terms, e.g.,  $\mathcal{O}((\log d)(\log \log d)^2) = \widetilde{\mathcal{O}} (\log d)$  and  $\mathcal{O}(d\log d) = \widetilde{\mathcal{O}} (d)$ . In addition, we use  $G_{\mathrm{ort}}(d)$  to denote the orthogonal group in dimension  $d$ :
$$
G _ {\mathrm {o r t}} (d) = \left\{Q \in \mathbb {R} ^ {d \times d}: Q Q ^ {\top} = Q ^ {\top} Q = I _ {d} \right\}.
$$
Let  $\mathcal{P}(\Omega)$  be the set of probability distributions over  $\Omega$  and  $\mathcal{M}(\Omega)$  be the set of random variables taking values in  $\Omega$ . Given two functions  $f, g$  over  $\mathcal{X}$  and  $\{\mathbf{x}_i\}_{i=1}^n$ , let  $\hat{\rho}_n(f, g) = \sqrt{\frac{1}{n} \sum_{i=1}^n |f(\mathbf{x}_i) - g(\mathbf{x}_i)|^2}$ . Let  $\mathbb{S}^{d-1} = \{x \in \mathbb{R}^d : \|x\|_2 = 1\}$ ,  $r\mathbb{S}^{d-1} = \{x : x/r \in \mathbb{S}^{d-1}\}$ . Let  $a \wedge b = \min(a, b)$ ,  $[k] = \{1, 2, \ldots, k\}$  for  $k \in \mathbb{N}$ , and  $[a, b] = \{a, a+1, \ldots, b\}$  for  $b, a \in \mathbb{N}$  and  $b > a$ . For a vector  $\mathbf{v}$ , denote by  $\|\mathbf{v}\|_p := (\sum_i |v_i|^p)^{1/p}$  the  $\ell^p$  norm. For a matrix  $A$ , let  $\|A\|$  and  $\|A\|_F$  be the spectral norm and Frobenius norm, respectively. Moreover, denote by  $A_{i,\cdot}$  and  $A_{\cdot,j}$  the  $i$ -th row and  $j$ -th column, respectively, and similar notations are also defined for tensors. Given  $\mathcal{I} = (i_1, i_2, \ldots, i_k)$ , let  $\mathbf{x}_{\mathcal{I}} = (x_{i_1}, \ldots, x_{i_k})$ . We use  $\sigma$  to denote both the activation function and standard deviation of label noise. To avoid ambiguity, we shall use  $\sigma(\cdot)$  and  $\sigma$  to distinguish them. When applying  $\sigma(\cdot)$  to a vector/matrix/tensor, it should be understood in an element-wise manner.
We consider the standard setup of supervised learning. Let  $\mathcal{X} \subset \mathbb{R}^{4d}$  and  $\mathcal{Y} \subset \mathbb{R}$  be the input and output domain, respectively. We are given the training set  $S_{n} = \{(x_{i},y_{i})\}_{i=1}^{n}$  with  $y_{i} = h^{*}(x_{i}) + \xi_{i}, x_{i} \stackrel{iid}{\sim} P$  and  $\xi_{i} \stackrel{iid}{\sim} \mathcal{N}(0,\sigma^{2})$ . Here  $P$  and  $h^{*}$  denote the input distribution and target function, respectively. We assume  $\log_2 d \in \mathbb{N}^+$  for simplicity. Let  $h: \mathcal{X} \times \Theta \mapsto \mathbb{R}$  be our parametric model with  $\Theta = \mathbb{R}^p$ , where  $p$  denotes the number of parameters; we often write  $h_{\theta} = h(\cdot;\theta)$  for short. Our task is to recover  $h^{*}$  from  $S_{n}$  by using  $h_{\theta}$ .
Given a threshold  $A > 0$ , we also consider the truncated model  $\pi_A \circ h_\theta$ , where the truncation operator  $\pi_A$  is defined by  $\pi_A \circ h(\mathbf{x}) = \max (\min (h(\mathbf{x}),A), - A)$ . In addition, we consider both the square loss  $\ell (y,y^{\prime}) = (y - y^{\prime})^{2} / 2$  and its truncated version  $\ell_B(y,y') = \frac{1}{2} (y - y')^2\wedge \frac{1}{2} B^2$ . This loss truncation is applied to handle the noise unboundedness; see Appendix B.2 for more details.
2.1 Network Architectures
A  $L$ -layer neural network is given by  $h_{\theta}(\mathbf{x}) = \mathcal{M}_o \circ \sigma_L \circ \mathcal{T}_L \circ \dots \circ \sigma_1 \circ \mathcal{T}_1(\mathbf{x})$ , where  $\sigma_l$  and  $\mathcal{T}_l$  denote the activation function and linear transform with bias at the  $l$ -th layer, respectively. Let  $\mathbf{z}^{(l)}$  denote the hidden state of  $l$ -th layer:  $\mathbf{z}^{(l)}(\mathbf{x}) = \sigma_l \circ \mathcal{T}_l \circ \dots \circ \sigma_1 \circ \mathcal{T}_1(\mathbf{x})$  for  $l \in [L]$  and  $\mathbf{z}^{(0)}(\mathbf{x}) = \mathbf{x}$ . When it is clear from context, we will write  $\mathbf{z}^{(l)} = \mathbf{z}^{(l)}(\mathbf{x})$  and  $\mathcal{T}_l\mathbf{z} = \mathcal{T}_l(\mathbf{z})$  for short. In different architectures,  $\{\mathcal{T}_l\}_{l=1}^L$  are parameterized in different ways.  $\mathcal{M}_o$  denotes the output layer, which performs a linear combination of the output features:  $\mathcal{M}_o \circ \mathbf{z}^{(L)}(\mathbf{x}) = W_o\mathrm{vec}(\mathbf{z}^{(L)}(\mathbf{x}))$ , where  $W_o$  is the weight used to parameterize  $\mathcal{M}_o$ .
FCNs.  $\mathcal{T}_l: \mathbb{R}^{C_l} \mapsto \mathbb{R}^{C_{l-1}}$  is a fully-connected transform parameterized by  $\mathcal{T}_l(\mathbf{z}) = W^{(l)}\mathbf{z} + \mathbf{b}^{(l)}$  with  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1}}$  and  $\mathbf{b}^{(l)} \in \mathbb{R}^{C_l}$ . Here  $C_l$  denotes the width of  $l$ -th layer.
CNNs. The  $l$ -th hidden state is a feature matrix:  $\mathbf{z}^{(l)}(\mathbf{x}) \in \mathbb{R}^{D_l \times C_l}$  with  $D_{l}$  and  $C_l$  denoting the spatial dimension and number of channels, respectively.  $\mathcal{T}_l: \mathbb{R}^{D_{l-1} \times C_{l-1}} \mapsto \mathbb{R}^{D_l \times C_l}$  is parameterized by a kernel  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1} \times s}$  and bias  $\mathbf{b}^{(l)} \in \mathbb{R}^{C_l}$  as follows
$$
\left(\mathcal {T} _ {l} (\mathbf {z})\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} * _ {s} W _ {j, i,:} ^ {(l)} + b _ {j} ^ {(l)} \mathbf {1}, \quad \text {f o r} j = 1, \dots , C _ {l} \tag {1}
$$
where  $*_{s} : \mathbb{R}^{sk} \times \mathbb{R}^{s} \mapsto \mathbb{R}^{k}$  denotes the convolution with stride given by
$$
\mathbf {v} * _ {s} \mathbf {w} = \left(\mathbf {v} _ {I _ {1}} ^ {\top} \mathbf {w}, \mathbf {v} _ {I _ {2}} ^ {\top} \mathbf {w}, \dots , \mathbf {v} _ {I _ {k}} ^ {\top} \mathbf {w}\right) \in \mathbb {R} ^ {k}, \tag {2}
$$
where  $I_{j} \coloneqq [(j - 1)s + 1, js]$  denotes the  $j$ -th patch, and  $\mathbf{v}$  and  $\mathbf{w}$  denote the signal and filter, respectively. As a comparison, we also consider the convolution without stride  $*: \mathbb{R}^{k} \times \mathbb{R}^{s} \mapsto \mathbb{R}^{k - s + 1}$  given by  $(\mathbf{v}*\mathbf{w})_i = \sum_{j = 1}^s v_{i + j - 1}w_j$ . Note that the stride plays a role of downsampling and in practice, it is also common to use other downsampling schemes such as max pooling and average pooling. All results in this paper hold regardless of which one is used and thus, we will focus on the stride case without loss of generality.
LCNs. A LCN has the same architecture as its CNN counterpart but lacks weight sharing. Consequently, LCNs have more parameters. Specifically, the linear transform  $\mathcal{T}_l: \mathbb{R}^{D_{l-1} \times C_{l-1}} \mapsto \mathbb{R}^{D_l \times C_l}$  is parameterized by  $W^{(l)} \in \mathbb{R}^{C_l \times C_{l-1} \times D_{l-1}}$  and  $\mathbf{b}^{(l)} \in \mathbb{R}^{D_l \times C_l}$  as follows
$$
\left(\mathcal {T} _ {l} (\mathbf {z})\right) _ {:, j} = \sum_ {i = 1} ^ {C _ {l - 1}} \mathbf {z} _ {:, i} \star_ {s} W _ {j, i,:} ^ {(l)} + \mathbf {b} _ {:, j} ^ {(l)}, \quad \text {f o r} j = 1, \dots , C _ {l}
$$
where the local linear operator  $\star_{s}:\mathbb{R}^{ks}\times \mathbb{R}^{ks}\mapsto \mathbb{R}^{k}$  is defined by  $\mathbf{v}\star_{s}\mathbf{w} = (\mathbf{v}_{I_1}^\top \mathbf{w}_{I_1},\mathbf{v}_{I_2}^\top \mathbf{w}_{I_2},\ldots ,\mathbf{v}_{I_k}^\top \mathbf{w}_{I_k})$  , where  $I_{j} = [(j - 1)s + 1,js]$  denotes the indices of  $j$  -th patch.
Throughout this paper, we always assume the filter size  $s = 2$  for technical simplicity and thus  $D_{l} = 4d / 2^{l} = D_{l - 1} / 2$  for both CNNs and LCNs.
Regularizer. To regularize CNNs and LCNs, we consider following  $\ell_2$ -type norm:
$$
\left\| \theta \right\| _ {\mathcal {P}} := \left\| W _ {o} \right\| _ {2} + \sum_ {l = 1} ^ {L} \left(\left\| W ^ {(l)} \right\| _ {F} + \alpha_ {l} \left\| \mathbf {b} ^ {(l)} \right\| _ {F}\right), \tag {3}
$$
where  $\alpha_{l} = \sqrt{D_{l}}$  for CNNs and  $\alpha_{l} = 1$  for LCNs. The factor  $\alpha_{l}$  is introduced such that  $\| \theta \|_{\mathcal{P}}$  can control the Lipschitz norm of  $h_\theta$ , thereby yielding effective capacity controls for CNNs and LCNs. See Appendix C for more details.
3 Universal Approximation
The following theorem shows that deep ReLU CNNs are universal as long as they are logarithmically deep with respect to the input dimension and the proof is deferred to Appendix D.1.
Theorem 3.1 (Universality). Consider CNNs with all activation functions to be ReLU. Suppose  $L = \log_2(4d)$  and  $C_l = 2^{l + 1}$  for  $l \in [L - 1]$  to be fixed and allow the number of channels of the last layer  $C_L$  to increase. Then, the CNNs are universal: for any  $\epsilon > 0$ , any compact set  $\Omega \subset \mathbb{R}^{4d}$ , and any  $h^* \in C(\Omega)$ , there exists a CNN  $h_\theta$  such that  $\sup_{\mathbf{x} \in \Omega} |h_\theta(\mathbf{x}) - h^*(\mathbf{x})| \leqslant \epsilon$ .
Proof idea. Write  $h_{\theta} = \mathcal{M}_{o}\circ \sigma \circ \mathcal{T}_{L}\circ \mathbf{z}^{(L - 1)}$ . First, we show there exists parameters (independent of  $h^*$ ) such that
$$
\mathbf {z} ^ {(L - 1)} (\mathbf {x}) = \left( \begin{array}{c c c c c c c} \sigma (x _ {1}) & \sigma (- x _ {1}) & \sigma (x _ {2}) & \sigma (- x _ {2}) & \dots & \sigma (x _ {2 d}) & \sigma (- x _ {2 d}) \\ \sigma (x _ {2 d + 1}) & \sigma (- x _ {2 d + 1}) & \sigma (x _ {2 d + 2}) & \sigma (- x _ {2 d + 2}) & \dots & \sigma (x _ {4 d}) & \sigma (- x _ {4 d}) \end{array} \right) \in \mathbb {R} ^ {2 \times (4 d)}.
$$
This implies that after  $L - 1$  layers, the spatial dimension is reduced to 2 but the spatial information is stored to different channels in the form of  $\{\sigma(x_i), \sigma(-x_i)\}$ . Comparing  $\mathbf{z}^{(L-1)}(\mathbf{x})$  with the input  $\mathbf{x}$ , there is no information loss since  $x_i = \sigma(x_i) - \sigma(-x_i)$  for any  $i \in [4d]$ . Then, the universality can be established by simply showing that  $\mathcal{M}_o \circ \sigma \circ \mathcal{T}_L$  can simulate any two-layer ReLU networks.
The synergy between multichanneling and downsampling. The key to achieving universality with a depth of  $\mathcal{O}(\log d)$  lies in a unique synergy between multichanneling and downsampling. Downsampling can expand the receptive field at an exponential rate, enabling CNNs to capture long-range correlations with only  $\mathcal{O}(\log d)$  depth. Meanwhile, multichanneling prevents information loss whenever downsampling operations reduce the spatial dimensions. It is worth noting that this specific collaboration between multichanneling and downsampling has been adopted in most practical CNNs, such as VGG Nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016). Our universality analysis provides theoretical support for this widespread architectural choice.
The following proposition further shows that the depth requirement in Theorem 3.1 is optimal, whose proof can be found in Appendix D.2.
Proposition 3.2. Let  $\mathcal{X} = [0,1]^{4d}$  and consider the target function  $h^*(\mathbf{x}) = x_1x_{2d+1}$ . If  $L \leqslant \log_2(4d) - 1$ , then for any  $C_l \in \mathbb{N}$  for  $l \in [L]$  and any activation functions  $\{\sigma_l\}_{l=1}^L$ , we have  $\inf_{\theta} \sup_{\mathbf{x} \in \mathcal{X}} |h_\theta(\mathbf{x}) - h^*(\mathbf{x})| \geqslant \frac{1}{8}$ .
The intuition behind this is straightforward: If the depth of the CNN is less than  $\log_2(4d)$ , the size of the receptive field will not exceed  $2d$ . Consequently, functions that encode longer-range correlations cannot be accurately approximated.
The cruciality of downsampling. The following proposition shows that without downsampling, a minimum depth of  $\Omega(d)$  is necessary for achieving universality. This highlights the importance of downsampling as it enables universality with logarithmic depth (Theorem 3.1).
Proposition 3.3. We temporarily use  $h_\theta$  to denote the CNN without downsampling. Let  $\mathcal{X} = [0,1]^{4d}$  and  $h^*(\mathbf{x}) = x_1x_{4d}$ . If  $L \leqslant 4d - 2$ , then for any  $C_l \in \mathbb{N}$  for  $l \in [L]$  and any activation functions  $\{\sigma_l\}_{l=1}^L$ , we have  $\inf_{\mathbf{x} \in \mathcal{X}} |h_\theta(\mathbf{x}) - h^*(\mathbf{x})| \geqslant \frac{1}{8}$ .
The proof is presented in Appendix D.3. The reason behind is simple: vanilla convolution (without stride) can only capture local correlations of length 2 (since our filter size is  $s = 2$ ). Stacking  $L$  layers of vanilla convolutions without downsampling will only allow the network to capture correlations of length  $L + 1$ .
4 Efficient Learning of Sparse Functions
Definition 4.1 (Sparse function). A function  $f: \mathbb{R}^d \mapsto \mathbb{R}$  is said to be sparse if  $f$  only depends on a few coordinates. Given  $k \in \mathbb{N}$  with  $k \ll d$ ,  $f$  is said to be  $k$ -sparse if there exist an index set  $\mathcal{I} = \{i_1, \ldots, i_k\} \subset [d]$  and  $g: \mathbb{R}^k \mapsto \mathbb{R}$  such that  $f(\mathbf{x}) = g(\mathbf{x}_{\mathcal{I}})$ , where  $\mathbf{x}_{\mathcal{I}} = (x_{i_1}, \ldots, x_{i_k}) \in \mathbb{R}^k$ .
The class of sparse functions includes functions of both functions with short-range correlations, such as  $f(\mathbf{x}) = g(x_{1},x_{2})$ , and those with long-range correlations, like  $f(\mathbf{x}) = g(x_{1},x_{d})$ . It is widely held that it is difficult for CNNs to capture long-range correlations due to locality bias. Consequently, it might seem that CNNs are not well-suited to learning sparse functions like  $\mathbf{x} \mapsto g(x_1,x_d)$ . However, for CNNs with downsampling, increasing depth can expand the receptive field exponentially, providing the opportunity to learn long-range correlations. Indeed this has been proven in the universality analysis.
In this section, we further show that deep CNNs are not only capable of, but also efficient at learning long-range sparse functions. The term "efficient" refers that the sample complexity depends on the input dimension logarithmically. This is because deep CNNs can effectively identify any  $k$  critical coordinates using only  $\mathcal{O}(k^2\log d)$  parameters as demonstrated by the following lemma.
Lemma 4.2 (Adaptive coordinate selection for Linear CNNs). Let  $\mathcal{I} = (i_1, i_2, \ldots, i_k) \subset [d]$ . For the linear CNN model  $h_\theta$  with  $C_l = k$  for  $l \in [L]$  and  $L = \log_2(d)$ . Then, there exist parameters (depending on  $\mathcal{I}$ ) such that  $\mathbf{z}^{(L)}(\mathbf{x}) = (x_{i_1}, x_{i_2}, \ldots, x_{i_k})$ .
Proof. First, consider the case of  $k = 1$  where  $\mathcal{I} = (i)$ . Let  $i - 1 = \sum_{l=0}^{L-1} a_l 2^l$  be the binary representation of  $i$ . Set  $W_{l,l,:} = (1 - a_l, a_l)$  for  $l = [L]$  and all other parameters (including the bias) to zero. Then, it is easy to verify that  $\mathbf{z}^{(L)}(\mathbf{x}) = (x_i)$ ; see Figure 1 for a diagram illustration.
For the case of  $k > 1$ , set the cross-channel weights to zero; for each channel, follow the case of  $k = 1$  to set weights and bias. Under this setup, different channels have no interaction and proceed in
a completely independent way. As a result, each channel selects one critical coordinate, and thus,  $\mathbf{z}^{(L)}(\mathbf{x}) = (x_{i_1}, x_{i_2}, \ldots, x_{i_k})$ .