Title: Theoretical Analysis of the Inductive Biases in Deep Convolutional Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convolutional neural network

Inductive bias

Universality

Sparse function

Equivariance group
Abstract: In this paper, we provide a theoretical analysis of the inductive biases in convolutional neural networks (CNNs). We start by examining the universality of CNNs, i.e., the ability to approximate any continuous functions.  We prove that a depth of $\mathcal{O}(\log d)$ suffices for deep CNNs to achieve this universality, where $d$ in the input dimension.  Additionally, we establish  that learning sparse functions with  CNNs requires only $\widetilde{\mathcal{O}}(\log^2d)$ samples, indicating that deep CNNs can efficiently capture {\em long-range} sparse correlations. These results are made possible through a novel combination of the multichanneling and downsampling when increasing the network depth. 

We also delve into the distinct roles  of weight sharing and locality in CNNs. To this end, we compare the performance of CNNs, locally-connected networks (LCNs), and fully-connected networks (FCNs) on a simple regression task, where LCNs can be viewed as CNNs without weight sharing. On the one hand,  we  prove that LCNs require ${\Omega}(d)$ samples while CNNs need only $\widetilde{\mathcal{O}}(\log^2d)$ samples,  highlighting the  critical role of weight sharing. On the other hand, we prove that FCNs require $\Omega(d^2)$ samples, whereas LCNs need only $\widetilde{\mathcal{O}}(d)$ samples,  underscoring the importance  of locality. These provable separations quantify the difference between the two biases, and the major observation behind our proof is that weight sharing and locality break different symmetries in the learning process.

Theoretical Analysis of Inductive Biases in Deep Convolutional Networks

Abstract

In this paper, we provide a theoretical analysis of the inductive biases in convolutional neural networks (CNNs). We start by examining the universality of CNNs, i.e., the ability to approximate any continuous functions. We prove that a depth of O (log d ) suffices for deep CNNs to achieve this universality, where d in the input dimension. Additionally, we establish that learning sparse functions with CNNs requires only ˜ O (log 2 d ) samples, indicating that deep CNNs can efficiently capture long-range sparse correlations. These results are made possible through a novel combination of the multichanneling and downsampling when increasing the network depth. We also delve into the distinct roles of weight sharing and locality in CNNs. To this end, we compare the performance of CNNs, locally-connected networks (LCNs), and fully-connected networks (FCNs) on a simple regression task, where LCNs can be viewed as CNNs without weight sharing. On the one hand, we prove that LCNs require Ω( d ) samples while CNNs need only ˜ O (log 2 d ) samples, highlighting the critical role of weight sharing. On the other hand, we prove that FCNs require Ω( d 2 ) samples, whereas LCNs need only ˜ O ( d ) samples, underscoring the importance of locality. These provable separations quantify the difference between the two biases, and the major observation behind our proof is that weight sharing and locality break different symmetries in the learning process.

1 Introduction

Convolutional neural networks (CNNs) (Fukushima, 1988; LeCun et al., 1998) are a fundamental model in deep learning, known for their exceptional performance in many tasks. In particular, CNNs consistently outperform the fully-connected neural network (FCN) counterparts in vision-related tasks (Krizhevsky et al., 2012; He et al., 2016; Huang et al., 2017). Uncovering the underlying mechanism behind the success of CNNs is thus of paramount importance in deep learning.

Zhou (2020a;b); Feng et al. (2022); He et al. (2022) studied the approximation capabilities of CNNs for target functions in spaces such as continuous functions and Sobolev spaces. Although these results are important, they cannot explain why CNNs perform better than FCNs. The primary reason for this limitation is that these works did not take into account the specific structures of CNNs, including multichanneling, downsampling, weight sharing, and locality . The locality refers to the use of small filters, e.g., filter sizes can be as small as 3 in popular VGG nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016). Comprehending the inductive biases of these architecture choices is critical to understand the exceptional performance of CNNs.

Li et al. (2020) designed a simple classification task to demonstrate the superiority of CNNs over FCNs. The authors prove that for this task, FCNs need Ω( d 2 ) samples while CNNs need only O (1) samples, thereby providing theoretical support for the superiority of using convolutions. However, this study neither examined the individual impact of weight sharing and locality nor considered the

Lei Wu

inductive biases of multichanneling and downsampling. Additionally, the analysis was limited to shallow CNNs and did not examine the interaction between these structures and network depth.

1.1 Our Results

In this work, we conduct a systematic analysis of the inductive biases associated with the specific structures of CNNs. Our main contributions are summarized as follows.

Universality. We establish the universality of deep CNNs with a depth of O (log d ) . This is in contrast to existing works (Zhou, 2020a;b; He et al., 2022), where the universality requires a depth of at least Ω( d ) . The key to our improvement is an effective leveraging of the inductive biases of multichanneling and downsampling:

Downsampling amplifies the size of the receptive field exponentially, thus explaining the need for logarithmic depth. Furthermore, we prove that if downsampling is not used, CNNs require a depth of at least Ω( d ) , demonstrating the cruciality of downsampling.

Multichanneling serves as a mechanism for storing extracted information. By increasing the number of channels whenever the spatial dimension is reduced by downsampling, we ensure that no information is lost. This combination of multichanneling and downsampling is widely employed in practical CNNs, ranging from classical LeNet (LeCun et al., 1998) to modern VGG nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016).

It is worth mentioning that while studies like Poggio et al. (2017) and Cagnetta et al. (2022) have examined similar CNN architectures with O (log d ) depth, they did not explicitly establish universality as our work does. Specifically, Cagnetta et al. (2022) focused on the kernel associated with deep CNNs having infinitely many channels. In retrospect, one could potentially show that the deep-CNN kernel is universal by verifying that kernel has no zero eigenvalues.

Learning sparse functions. A function f : R d ↦→ R is said to be sparse if it only depends on a few coordinates of the input, e.g., f ( x ) := g ( x 1 , x d ) for some g : R 2 ↦→ R . We prove that learning sparse functions using CNNs requires only ˜ O (log 2 d ) samples, which is nearly optimal as the information-theoretic lower bound of learning such functions is Ω(log d ) (Han and Yuan, 2020).

This result is surprising because it has been widely believed that CNNs struggle to capture long-range correlations. However, our findings suggest that CNNs can efficiently learn long-range sparse ones, which is a valuable attribute for many applications.

In addition, it is important to note that the near-optimal sample complexity of learning sparse functions using CNNs is achieved with only O (log d ) depth and O ( k 2 log d ) total parameters, where k is the number of critical coordinates in sparse functions. A lower bound is established to demonstrate the optimality of the depth requirement. The ability of CNNs to select any k coordinate with only O ( k 2 log d ) total parameters is remarkable, especially considering that even in the linear case, LASSO (Tibshirani, 1996) requires Ω( d ) parameters. It is the synergy of increased depth, weight sharing, and multichanneling that gives CNNs this exceptional capability.

Disentangling the weight sharing and locality. We next study the inductive biases of locality and weight sharing by comparing the performance of CNNs, LCNs, and FCNs on a synthetic regression task. This allows us to separate the effects of weight sharing and locality.

CNNs vs. LCNs. We prove that CNNs requires only ˜ O (log 2 d ) samples to learn, while LCNs trained by SGD or Adam with standard initialization need Ω( d ) samples. This provides a separation between CNNs and LCNs and demonstrates the crucial role of weight sharing.

LCNs vs. FCNs. We prove that LCNs requires only ˜ O ( d ) samples to learn, while FCNs trained by SGD with Gaussian initialization need Ω( d 2 ) samples. This provably separates LCNs from FCNs and demonstrates the benefit of locality.

The difference in sample complexity can be attributed to the different symmetries encoded in the architecture biases. For instance, stochastic gradient descent (SGD) exhibits different symmetries for these models: a lack of equivariance for CNNs, a local permutation equivariance for LCNs, and a global orthogonal equivariance for FCNs. The size of the equivariance groups determines the

minimax sample complexity and distinguishes between different architectures. Similar ideas have been previously explored in Li et al. (2020); Xiao and Pennington (2022) to understand CNNs; but these works did not differentiate the roles of weight sharing and locality. For a detailed comparison with these works, see the related work section in Appendix A.

Technical contribution. The lower bounds for LCNs and FCNs are established using Fano's method from minimax theory (Wainwright, 2019, Section 15). However, different from the traditional statistical setup where the estimator is deterministic, the estimators produced by stochastic optimizers are random. To address this issue, we develop a variant of Fano's method for random estimators, which might be of independent interest. Further details can be found in Appendix B.3.

Related work. We refer to Appendix A for a detailed comparison with other related works.

1.2 Notations

We use poly( z 1 , . . . , z p ) to denote a quantity that depends on z 1 , . . . , z p polynomially. We use X ≲ Y to denote X ⩽ CY for some absolute constant C and X ≳ Y is defined analogously. We also use the standard big-O notations: Θ( · ) , O ( · ) and Ω( · ) . In addition, we use ˜ O and ˜ Ω to hide higher-order terms, e.g., O ((log d )(log log d ) 2 ) = ˜ O (log d ) and O ( d log d ) = ˜ O ( d ) . In addition, we use G ort ( d ) to denote the orthogonal group in dimension d :

Let P (Ω) be the set of probability distributions over Ω and M (Ω) be the set of random variables taking values in Ω . Given two functions f, g over X and { x i } n i =1 , let ˆ ρ n ( f, g ) = √ 1 n ∑ n i =1 | f ( x i ) -g ( x i ) | 2 . Let S d -1 = { x ∈ R d : ∥ x ∥ 2 = 1 } , r S d -1 = { x : x/r ∈ S d -1 } . Let a ∧ b = min( a, b ) , [ k ] = { 1 , 2 , . . . , k } for k ∈ N , and [ a, b ] = { a, a +1 , . . . , b } for b, a ∈ N and b > a . For a vector v , denote by ∥ v ∥ p := ( ∑ i | v i | p ) 1 /p the ℓ p norm. For a matrix A , let ∥ A ∥ and ∥ A ∥ F be the spectral norm and Frobenius norm, respectively. Moreover, denote by A i, : and A : ,j the i -th row and j -th column, respectively, and similar notations are also defined for tensors. Given I = ( i 1 , i 2 , . . . , i k ) , let x I = ( x i 1 , . . . , x i k ) . We use σ to denote both the activation function and standard deviation of label noise. To avoid ambiguity, we shall use σ ( · ) and σ to distinguish them. When applying σ ( · ) to a vector/matrix/tensor, it should be understood in an element-wise manner.

2 Preliminaries

We consider the standard setup of supervised learning. Let X ⊂ R 4 d and Y ⊂ R be the input and output domain, respectively. We are given the training set S n = { ( x i , y i ) } n i =1 with y i = h ∗ ( x i ) + ξ i , x i iid ∼ P , and ξ i iid ∼ N (0 , σ 2 ) . Here P and h ∗ denote the input distribution and target function, respectively. We assume log 2 d ∈ N + for simplicity. Let h : X × Θ ↦→ R be our parametric model with Θ = R p , where p denotes the number of parameters; we often write h θ = h ( · ; θ ) for short. Our task is to recover h ∗ from S n by using h θ .

Given a threshold A > 0 , we also consider the truncated model π A ◦ h θ , where the truncation operator π A is defined by π A ◦ h ( x ) = max(min( h ( x ) , A ) , -A ) . In addition, we consider both the square loss ℓ ( y, y ′ ) = ( y -y ′ ) 2 / 2 and its truncated version ℓ B ( y, y ′ ) = 1 2 ( y -y ′ ) 2 ∧ 1 2 B 2 . This loss truncation is applied to handle the noise unboundedness; see Appendix B.2 for more details.

2.1 Network Architectures

A L -layer neural network is given by h θ ( x ) = M o ◦ σ L ◦ T L ◦ · · · ◦ σ 1 ◦ T 1 ( x ) , where σ l and T l denote the activation function and linear transform with bias at the l -th layer, respectively. Let z ( l ) denote the hidden state of l -th layer: z ( l ) ( x ) = σ l ◦ T l ◦ · · · ◦ σ 1 ◦ T 1 ( x ) for l ∈ [ L ] and z (0) ( x ) = x . When it is clear from context, we will write z ( l ) = z ( l ) ( x ) and T l z = T l ( z ) for short. In different architectures, {T l } L l =1 are parameterized in different ways. M o denotes the output layer, which performs a linear combination of the output features: M o ◦ z ( L ) ( x ) = W o vec( z ( L ) ( x )) , where W o is the weight used to parameterize M o .

FCNs. T l : R C l ↦→ R C l -1 is a fully-connected transform parameterized by T l ( z ) = W ( l ) z + b ( l ) with W ( l ) ∈ R C l × C l -1 and b ( l ) ∈ R C l . Here C l denotes the width of l -th layer.

CNNs. The l -th hidden state is a feature matrix: z ( l ) ( x ) ∈ R D l × C l with D l and C l denoting the spatial dimension and number of channels, respectively. T l : R D l -1 × C l -1 ↦→ R D l × C l is parameterized by a kernel W ( l ) ∈ R C l × C l -1 × s and bias b ( l ) ∈ R C l as follows

where ∗ s : R sk × R s ↦→ R k denotes the convolution with stride given by

where I j := [( j -1) s + 1 , js ] denotes the j -th patch, and v and w denote the signal and filter, respectively. As a comparison, we also consider the convolution without stride ∗ : R k × R s ↦→ R k -s +1 given by ( v ∗ w ) i = ∑ s j =1 v i + j -1 w j . Note that the stride plays a role of downsampling and in practice, it is also common to use other downsampling schemes such as max pooling and average pooling. All results in this paper hold regardless of which one is used and thus, we will focus on the stride case without loss of generality.

LCNs. A LCN has the same architecture as its CNN counterpart but lacks weight sharing. Consequently, LCNs have more parameters. Specifically, the linear transform T l : R D l -1 × C l -1 ↦→ R D l × C l is parameterized by W ( l ) ∈ R C l × C l -1 × D l -1 and b ( l ) ∈ R D l × C l as follows

where the local linear operater ⋆ s : R ks × R ks ↦→ R k is defined by v ⋆ s w = ( v ⊤ I 1 w I 1 , v ⊤ I 2 w I 2 , . . . , v ⊤ I k w I k ) , where I j = [( j -1) s +1 , js ] denotes the indices of j -th patch.

Throughout this paper, we always assume the filter size s = 2 for technical simplicity and thus D l = 4 d/ 2 l = D l -1 / 2 for both CNNs and LCNs.

Regularizer. To regularize CNNs and LCNs, we consider following ℓ 2 -type norm:

where α l = √ D l for CNNs and α l = 1 for LCNs. The factor α l is introduced such that ∥ θ ∥ P can control the Lipschitz norm of h θ , thereby yielding effective capacity controls for CNNs and LCNs. See Appendix C for more details.

3 Universal Approximation

The following theorem shows that deep ReLU CNNs are universal as long as they are logarithmically deep with respect to the input dimension and the proof is deferred to Appendix D.1.

Theorem 3.1 (Universality) . Consider CNNs with all activation functions to be ReLU . Suppose L = log 2 (4 d ) and C l = 2 l +1 for l ∈ [ L -1] to be fixed and allow the number of channels of the last layer C L to increase. Then, the CNNs are universal: for any ϵ > 0 , any compact set Ω ⊂ R 4 d , and any h ∗ ∈ C (Ω) , there exists a CNN h θ such that sup x ∈ Ω | h θ ( x ) -h ∗ ( x ) | ⩽ ϵ.

Proof idea. Write h θ = M o ◦ σ ◦T L ◦ z ( L -1) . First, we show there exists parameters (independent of h ∗ ) such that

This implies that after L -1 layers, the spatial dimension is reduced to 2 but the spatial information is stored to different channels in the form of { σ ( x i ) , σ ( -x i ) } . Comparing z ( L -1) ( x ) with the input x , there is no information loss since x i = σ ( x i ) -σ ( -x i ) for any i ∈ [4 d ] . Then, the universality can be established by simply showing that M o ◦ σ ◦ T L can simulate any two-layer ReLU networks.

The synergy between multichanneling and downsampling. The key to achieving universality with a depth of O (log d ) lies in a unique synergy between multichanneling and downsampling. Downsampling can expand the receptive field at an exponential rate, enabling CNNs to capture long-range correlations with only O (log d ) depth. Meanwhile, multichanneling prevents information loss whenever downsampling operations reduce the spatial dimensions. It is worth noting that this specific collaboration between multichanneling and downsampling has been adopted in most practical CNNs, such as VGG Nets (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016). Our universality analysis provides theoretical support for this widespread architectural choice.

The following proposition further shows that the depth requirement in Theorem 3.1 is optimal, whose proof can be found in Appendix D.2.

Proposition 3.2. Let X = [0 , 1] 4 d and consider the target function h ∗ ( x ) = x 1 x 2 d +1 . If L ⩽ log 2 (4 d ) -1 , then for any C l ∈ N for l ∈ [ L ] and any activation functions { σ l } L l =1 , we have inf θ sup x ∈X | h θ ( x ) -h ∗ ( x ) | ⩾ 1 8 .

The intuition behind this is straightforward: If the depth of the CNN is less than log 2 (4 d ) , the size of the receptive field will not exceed 2 d . Consequently, functions that encode longer-range correlations cannot be accurately approximated.

The cruciality of downsampling. The following proposition shows that without downsampling, a minimum depth of Ω( d ) is necessary for achieving universality. This highlights the importance of downsampling as it enables universality with logarithmic depth (Theorem 3.1).

Proposition 3.3. We temporarily use h θ to denote the CNN without downsampling. Let X = [0 , 1] 4 d and h ∗ ( x ) = x 1 x 4 d . If L ⩽ 4 d -2 , then for any C l ∈ N for l ∈ [ L ] and any activation functions { σ l } L l =1 , we have inf θ sup x ∈X | h θ ( x ) -h ∗ ( x ) | ⩾ 1 8 .

The proof is presented in Appendix D.3. The reason behind is simple: vanilla convolution (without stride) can only capture local correlations of length 2 (since our filter size is s = 2 ). Stacking L layers of vanilla convolutions without downsampling will only allow the network to capture correlations of length L +1 .

4 Efficient Learning of Sparse Functions

Definition 4.1 (Sparse function) . A function f : R d ↦→ R is said to be sparse if f only depends on a few coordinates. Given k ∈ N with k ≪ d , f is said to be k -sparse if there exist an index set I = { i 1 , . . . , i k } ⊂ [ d ] and g : R k ↦→ R such that f ( x ) = g ( x I ) , where x I = ( x i 1 , . . . , x i k ) ∈ R k .

The class of sparse functions includes functions of both functions with short-range correlations, such as f ( x ) = g ( x 1 , x 2 ) , and those with long-range correlations, like f ( x ) = g ( x 1 , x d ) . It is widely held that it is difficult for CNNs to capture long-range correlations due to locality bias. Consequently, it might seem that CNNs are not well-suited to learning sparse functions like x ↦→ g ( x 1 , x d ) . However, for CNNs with downsampling, increasing depth can expand the receptive field exponentially, providing the opportunity to learn long-range correlations. Indeed this has been proven in the universality analysis.

In this section, we further show that deep CNNs are not only capable of, but also efficient at learning long-range sparse functions. The term 'efficient' refers that the sample complexity depends on the input dimension logarithmically. This is because deep CNNs can effectively identify any k critical coordinates using only O ( k 2 log d ) parameters as demonstrated by the following lemma.

Lemma 4.2 (Adaptive coordinate selection for Linear CNNs) . Let I = ( i 1 , i 2 , . . . , i k ) ⊂ [ d ] . For the linear CNN model h θ with C l = k for l ∈ [ L ] and L = log 2 ( d ) . Then, there exist parameters (depending on I ) such that z ( L ) ( x ) = ( x i 1 , x i 2 , . . . , x i k ) .

Proof. First, consider the case of k = 1 where I = ( i ) . Let i -1 = ∑ L -1 l =0 a l 2 l be the binary representation of i . Set W l,l, : = (1 -a l , a l ) for l = [ L ] and all other parameters (including the bias) to zero. Then, it is easy to verify that z ( L ) ( x ) = ( x i ) ; see Figure 1 for a diagram illustration.

For the case of k > 1 , set the cross-channel weights to zero; for each channel, follow the case of k = 1 to set weights and bias. Under this setup, different channels have no interaction and proceed in

a completely independent way. As a result, each channel selects one critical coordinate, and thus, z ( L ) ( x ) = ( x i 1 , x i 2 , . . . , x i k ) .

x

4

x

4

x

2

x

1

0

x

2

1

0

x

4

x

3

0

x

4

1

1

1

x

8

x

6

x

5

0

x

6

1

0

x

8

x

7

0

x

8

1

1

0

Remark 4.3. Lemma 4.2 indicates that deep CNNs are able to effectively identify any k critical coordinates with O (log d ) depth and O ( k 2 log d ) parameters, which is significantly smaller than d . The key to this achievement is the adaptivity of neural networks, in combination with weight sharing, multichanneling, downsampling, and depth, as demonstrated in the proof. Specifically, downsampling and increased depth allow for capturing long-range sparse correlations with only O (log d ) depth, multichanneling facilitates the storage of information regarding different critical coordinates, and weight sharing ensures the number of parameters of each layer to be independent of d .

We now proceed to consider the learning of nonlinear sparse functions. We first need the following feature selection result for ReLU CNNs. The proof is similar to the linear case (Lemma 4.2) and deferred to Appendix E.1.

Lemma 4.4. Given k, m ∈ N , consider a ReLU CNN with depth L = log 2 (4 d ) and the channel numbers C l = 2 k for all l ∈ [ L -1] and C L = m . Then for any I = ( i 1 , . . . , i k ) ⊂ [4 d ] , u 1 , . . . , u m ∈ R k , and c 1 , . . . , c m ∈ R , there exists θ ∈ Θ such that the L -th layer outputs:

Furthermore, this CNN has O ( k 2 log d + km ) parameters.

According to this lemma, deep CNNs are capable of generating adaptive features of the form: x ↦→ σ ( u ⊤ x I + c ) with only O (log d ) depth and O ( k 2 log d ) parameters, where the features depend only on the k critical coordinates. Note that linear combinations of this type of features give twolayer networks. Therefore, functions of the form f ( x ) := g ( x I ) with g : R k ↦→ R can be well approximated by deep CNNs, as long as g can be wll approximated by two-layer ReLU networks. To measure the learnability of g , we adopt the Barron regularity as proposed in E et al. (2022).

Definition 4.5 (Barron space) . Consider functions admiting the following integral representation g ρ ( x ) = ∫ Ω aσ ( u ⊤ x + c ) ρ (d a, d u , d c ) for all x ∈ X , where σ = ReLU , Ω = R 1 × R k × R 1 and ρ ∈ P (Ω) . For a function g : X ↦→ R , denote by A g = { ρ : g ρ ( x ) = g ( x ) for all x ∈ X} . Then, we define

More explicitly, the Fourier-analytic characterization (Barron, 1993; Klusowski and Barron, 2016) showed that ∥ g ∥ B ≲ ∫ (1+ ∥ ξ ∥ 1 ) 2 | ˆ g ( ξ ) | d ξ , where ˆ g denotes the Fourier transform of g . The analysis with Barron regularity is often much simpler (E et al., 2019; 2022) and can yield a better dependence on k than using traditional smoothness measures. Specifically, the Barron regularity allows obtaining generalization bounds with dimension-independent rates like O ( n -1 / 2 ) . In contrast, for traditional smoothness measures such as assuming g ∈ C s ([0 , 1] k ) , the resulting error rate would scale like O ( n -s/k ) , which depends on k exponentially.

Recall ℓ B ( y, y ′ ) = 1 2 ( y -y ′ ) 2 ∧ 1 2 B 2 . Consider the regularized estimator given by ˆ θ n = argmin θ ( 1 n ∑ n i =1 ℓ B ( π A ◦ h θ ( x i ) , y i ) + λ ∥ θ ∥ P ) , where A,B,λ are hyperparameters to be tuned.

Theorem 4.6. Let X = [0 , 1] 4 d and h ∗ ( x ) := g ( x I ) for some I ⊂ [4 d ] be the target function. Let k = |I| and suppose g ∈ B , sup z | g ( z ) | ⩽ 1 , and d ⩾ 3 . For any δ ∈ (0 , 1 / 2) , ϵ ∈ (0 , 1 / 2) , there is a choice of A,B , and λ such that w.p. at least 1 -2 δ over the sampling of training set we have ∥ ∥ ∥ π A ◦ h ˆ θ n -h ∗ ∥ ∥ ∥ 2 L 2 ( P ) ⩽ ϵ, whenever

The proof is presented in Appendix E.2. This theorem shows that learning sparse functions with deep CNNs requires only ˜ O (log 2 d ) samples, which is nearly optimal from an information-theoretic perspective. This is because Han and Yuan (2020) proved that learning sparse functions requires at least Ω(log d ) samples. It is also important to mention that Cagnetta et al. (2022) showed that using deep convolutional kernels to learn long-range sparse functions suffers the curse of dimensionality. The comparison with our results also highlights the significance of adaptivity in neural networks.

Experimental validation. In this experiment, both short-range and long-range sparse target functions are considered. We set the input dimension d = 4096 , the sample size n = 400 and the noise level σ to be zero. For the CNN architecture, the filter size is s = 4 , resulting in a depth L = log 4 ( d ) = 6 ; the number of channels is set to C = 4 across all layers. The Adam optimizer is employed to our models , and importantly, no regularization is applied. As a comparison, we also examine two-layer fully-connected networks (FCNs) with a width 10 , as well as the ordinary least linear regression (OLS). The results are shown in Figure 2. One can see clearly that even without any explicit sparsity regularization, CNN can still learn sparse interactions efficiently in both short-range and long-range scenarios. In contrast, FCN and OLS overfit the data and fail to generalize to test data.

*

0

500

1000

1500

2000

2500

3000

# of iters.

10

5

10

4

10

3

10

2

10

1

10 0

Loss

f

*

(

x

) =

x

1+

x

2

CNN-train

CNN-test

FCN-train

FCN-test

OLS-train

OLS-test

0

500

1000

1500

2000

2500

3000

3500

# of iters.

10

5

10

4

10

3

10

2

10

1

10 0

Loss

f

(

x

) =

x

1+

xd

CNN-train

CNN-test

FCN-train

FCN-test

OLS-train

OLS-test

5 Disentangle the Inductive Biases of Weight Sharing and Locality

To facilitate our statement, in this section, we denote by h cnn θ , h lcn θ , h fcn θ the CNN, LCN, and FCN model, respectively. We shall consider the following task.

Separation Task. Suppose the input distribution P = N (0 , I 4 d ) and the target function ¯ h ∗ ( x ) = π A 0 ◦ h ∗ ( x ) where A 0 is a universal absolute constant to be specified later and

The truncation is employed to ensure boundedness of the output. However, we believe that a more refined analysis could eliminate the need for this constraint. This target function possesses the structures of both 'weight sharing' and 'locality' (the global sum can be computed hierarchically through local additions). We then consider the following comparisons.

CNNs vs. LCNs. This comparison allows us to isolate the bias of weight sharing as both models have the same structures, but LCNs do not have weight sharing.

LCNs vs. FCNs. This comparison allows us to isolate the bias of locality since the only difference between FCNs and LCNs is the presence or absence of locality.

By these comparisons, we can evaluate the effectiveness of each bias in contributing to the improved performance of CNNs over FCNs, and gain insights into how these biases interact with other structures such as depth, multichanneling, and downsampling.

To establish provable separations, we need both upper and lower bounds of the sample complexity of learning ¯ h ∗ . For the upper bounds, we consider the regularized estimator given by

where the model h θ and hyperparameters A,B, and λ will be specified for each comparison. We will use the covering number-based technique (see Appendix B.2) to upper bound the generalization error. See also Appendix C for how to bound the covering numbers of deep CNNs and deep LCNs.

Establishing lower bounds is often much more challenging. We will adopt a similar approach to Ng (2004); Li et al. (2020); Abbe and Boix-Adserà (2022), which involves understanding the learning hardness through the equivariance group of the learning algorithm.

5.1 Learning Algorithm, Group Equivariance, and Lower Bounds

A learning algorithm A : ( X × Y ) n ↦→M (Θ) generates a random variable taking values in Θ by using the training set S n . This definition includes the situation where our models are returned by stochastic optimizers such as SGD and Adam. We will denote X d = Y as Law( X ) = Law( Y ) .

G -equivariant/-invariant. Let G be a group acting on X . A learning algorithm A is said to be G -equivariant 1 if ∀ { x i , y i } n i =1 ∈ ( X × Y ) n and ∀ τ ∈ G, x ∈ X :

A distribution µ is said to be G -invariant if for any τ ∈ G , τ ( X ) ∼ µ if X ∼ µ .

Sample Complexity. Given a target function class F and a learning algorithm A , for a target accuracy ϵ > 0 , the sample complexity C ( A , F , ϵ ) is defined as the smallest integer n such that

where the expectation is taken over both the sampling of S n and the randomness in A . In addition, we define the minimax sample complexity by ¯ C ( F , ϵ ) := inf A C ( A , F , ϵ ) , where the infimum is taken over all learning algorithms, i.e., all the mappings: ( X × Y ) n ↦→M (Θ) .

For a function class F , denote by F ◦ G := { f ◦ τ : f ∈ F , τ ∈ G } the G -enlarged class.

Lemma 5.1. (A restatement of (Li et al., 2020, Lemma D.1)) Let A G be the set of all G -equivariant algorithms. For any function class F and ϵ > 0 , it holds that inf A ∈A G C ( A , F , ϵ ) ⩾ ¯ C ( F ◦ G,ϵ ) .

This lemma shows that the sample complexity of learning F with a G -equivalent algorithm can be lower bounded by the minimax complexity of learning the enlarged class : F ◦ G . The latter can be handled using standard approaches from minimax theory (see (Wainwright, 2019, Section 15)). Specifically, we will adopt Fano's method to lower-bound the minimax sample complexity, which reduces the problem to find a proper packing of the enlarged class F ◦ G (see Appendix B.3). In particular, when F = { f ∗ } , the packing number of F ◦ G can be completely determined by the packing number of the symmetry group G if f ∗ is chosen to satisfy

where ∥ · ∥ G is a proper norm defined on G . Moreover, estimating the packing number of a symmetry group is often much easier.

Note that the specific target function (5) in our separation task is designed to satisfies (9) for the symmetry groups associated with SGD algorithms and thus, the sample complexity can be completely governed by the size of symmetry groups. Specifically, for SGD algorithms, we have

The symmetry group of FCNs is the orthogonal group G ort (4 d ) , whose degree of freedom is Θ( d 2 ) . This explains the lower bound Ω( d 2 ) for FCNs.

The symmetry group of LCNs is the local permutation group G loc , whose degree of freedom is Θ( d ) . This explains the lower bound Ω( d ) for LCNs.

See the sections below for more details.

5.2 CNNs vs. LCNs

In this section, we will derive an upper bound of sample complexity for learning ¯ h ∗ with CNNs and a lower bound for learning ¯ h ∗ with LCNs. The two bounds together provide a quantification of the effect of weight sharing.

Consider the deep CNN h cnn θ with L = log 2 (4 d ) , C 1 = C L = 4 , C l = 2 for l ∈ [2 , L -1] , and σ 1 ( x ) = σ L ( x ) = ReLU 2 ( x ) , σ l ( x ) = ReLU( x ) for l ∈ [2 , L -1] .

Theorem 5.2 (Upper bound of CNNs) . Suppose d ⩾ 3 and A 0 ⩾ 0 and consider the estimator ˆ θ n given by (6) . For any ϵ ∈ (0 , 1 / 2) and δ ∈ (0 , 1 / 2) , there is a choice of A,B , and λ such that

The proof is deferred to Appendix G.2. It is shown that CNNs need only ˜ O (log 2 d ) samples to learn ¯ h ∗ . The reason behind this upper bound is that ¯ h ∗ can be accurately represented by deep CNNs with only O (log d ) parameters and well-controlled norms due to the well matching between the weight sharing and locality in CNNs and the unique characteristics of ¯ h ∗ .

Next we turn to establish a lower bound for LCNs. Let ℓ ( · , · ) be a general differentiable loss function ( )

Denote by A lcn T the algorithm that returns solutions by optimizing ˆ L lcn ( · ) for T steps using SGD or Adam. Under mild conditions, it can be shown that A lcn T is equivariant under the local permutation group:

Obviously, G loc is a group under matrix product. To ensure the G loc -equivariance of A lcn T , we need the following assumption, which is satisfied by all popular initialization schemes used in practice.

Assumption 1. At initialization, the fist layer weight satisfies: W (1) j, 1 , 2 k -1 and W (1) j, 1 , 2 k have the same marginal distribution for any j ∈ [ C 1 ] , k ∈ [2 d ] .

Lemma 5.3. Suppose that the input distribution P is G loc -invariant and Assumption 1 is satisfied. Then, for any T ∈ N , A lcn T is G loc -equivariant for both SGD and Adam.

The proof is deferred to Appendix F.2. Xiao and Pennington (2022) showed that under Gaussian initialization, the equivariance group of A lcn T is O (2) ⊗ I 2 d , which is slightly larger than G loc obtained in the current work. It should be stressed that G loc is obtained under a much milder assumption of initialization which holds for the popular uniform initialization scheme. Moreover, we show below that the G loc equviariance is sufficient for separating LCNs from CNNs.

Theorem 5.4 (Lower bound of LCNs) . Under Assumption 1, there exists absolute constants C, c > 0 such that ∀ T ∈ N , ϵ 0 ∈ (0 , c ] , d, A 0 ⩾ C :

The proof is deferred to Appendix G.1. By comparing Theorem 5.2 and 5.4, we see that the weight sharing in CNNs yields an ˜ Θ( d ) improvement on the sample complexity for learning ¯ h ∗ . This highlights the importance of exploiting the 'weight sharing' in ¯ h ∗ for efficient learning.

5.3 LCNs vs. FCNs

We will derive an upper bound of sample complexity for learning ¯ h ∗ with LCNs and a lower bound for learning ¯ h ∗ with FCNs. The two bounds together provide a quantification of the effect of locality.

We consider the deep LCN h lcn θ with L = log 2 (4 d ) , C 1 = C L = 4 , C l = 2 for l ∈ [2 , L -1] , and σ 1 ( x ) = σ L ( x ) = ReLU 2 ( x ) , σ l ( x ) = ReLU( x ) for l ∈ [2 , L -1] .

Theorem 5.5 (Upper bound of LCNs) . Suppose d ⩾ 3 and A 0 ⩾ 0 and consider the estimator ˆ θ n given by (6) . Then, for any ϵ ∈ (0 , 1 / 2) and δ ∈ (0 , 1 / 2) , there is a choice of A,B , and λ such that

The proof is deferred to Appendix H.2. It is shown that LCNs needs ˜ O ( d ) samples to learn ¯ h ∗ . The reason behind this upper bound is that ¯ h ∗ can be well approximated by LCNs with O ( d ) parameters and well-controlled norm. However, as opposed to CNNs, LCNs lack weight sharing and can only imitate the local features of ¯ h ∗ with the 'weight sharing' structure completely ignored.

Next we turn to establish a lower bound for FCNs. Let ℓ ( · , · ) be a general differentiable loss function. Denote by A fcn T the algorithm: Run SGD for T steps by optimizing ˆ L fcn ( θ ) := 1 n ∑ n i =1 ℓ ( h fcn θ ( x i ) , y i ) + r ( ∥ θ ∥ 2 ) , where r : [0 , + ∞ ) → [0 , + ∞ ) is a general penalty function. We refer to Appendix F.1 for a rigorous definition.

Lemma 5.6. Suppose that the input distribution P is G ort (4 d ) -invariant and entries of W (1) are i.i.d. initialized from N (0 , β 2 ) for some β > 0 . Then, for any T ∈ N , A fcn T is G ort (4 d ) -equivariant.

This lemma was proved in (Li et al., 2020, Corollary C.2) and we state it here for completeness. It implies that training FCNs with SGD induces a larger equivariance group than that of LCNs since G loc ⊂ G ort (4 d ) . It is important to note that this lemma only holds for SGD, as Adam is only permutation invariant. In contrast, the result in Lemma 5.3 applies to both SGD and Adam.

Theorem 5.7 (Lower bound of FCNs) . Suppose that the input distribution P is G ort (4 d ) -invariant and entries of W (1) are i.i.d. initialized from N (0 , β 2 ) for some β > 0 . Then there exists absolute constants C, c > 0 such that ∀ T ∈ N , ϵ 0 ∈ (0 , c ] , d, A 0 ⩾ C :

The proof is deferred to Appendix H.1. By comparing Theorem 5.5 and 5.7, we see that the locality itself yields another ˜ Θ( d ) improvement on the sample complexity for learning ¯ h ∗ . This highlights the importance of exploiting the locality in ¯ h ∗ for efficient learning.

6 Conclusion

In this paper, we delve into the theoretical analysis of the inductive biases associated with the multichanneling, downsampling, weight sharing, and locality in deep CNN architectures with a focus on understanding the interplay between network depth and these biases. Our results highlight the critical role of multichanneling and downsampling in enabling deep CNNs to effectively capture long-range correlations. We also analyze the effects of weight sharing and locality on breaking the learning algorithm's symmetry, through which we make a clear distinction between the two biases: the global orthogonal equivariance vs. the local permutation equivariance. By leveraging these symmetries, we establish provable separations for FCNs vs. LCNs and LCNs vs. CNNs, providing a strong theoretical support for the unique nature of weight sharing and locality.

References

Emmanuel Abbe and Enric Boix-Adserà. On the non-universality of deep learning: quantifying the cost of symmetry. In Advances in Neural Information Processing Systems , 2022. 8, 15, 36

Francis Bach. Learning Theory from First Principles . The MIT Press, 2023. URL https://www. di.ens.fr/~fbach/ltfp_book.pdf . 22

Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993. 6

Alberto Bietti. Approximation and learning with deep convolutional models: a kernel perspective. In International Conference on Learning Representations , 2021. 15

Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under geometric stability. Advances in Neural Information Processing Systems , 34:18673-18684, 2021. 15

Francesco Cagnetta, Alessandro Favero, and Matthieu Wyart. How wide convolutional neural networks learn hierarchical tasks. arXiv preprint arXiv:2208.01003 , 2022. 2, 7, 15

Arturo Deza, Qianli Liao, Andrzej Banburski, and Tomaso Poggio. Hierarchically compositional tasks and deep convolutional networks. arXiv preprint arXiv:2006.13915 , 2020. 15

Simon S Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Russ R Salakhutdinov, and Aarti Singh. How many samples are needed to estimate a convolutional neural network? In Advances in Neural Information Processing Systems , 2018. 15

Weinan E and Bing Yu. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1-12, 2018. 28

Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences , 17:1407-1425, 2019. 6

Weinan E, Chao Ma, and Lei Wu. The Barron space and the flow-induced function spaces for neural network models. Constructive Approximation , 55(1):369-406, 2022. 6, 34

Han Feng, Jianfei Li, and Ding-Xuan Zhou. Approximation analysis of CNNs from feature extraction view. arXiv preprint arXiv:2210.09041 , 2022. 1

Kunihiko Fukushima. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks , 1(2):119-130, 1988. 1

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems , 2018. 15

Cuize Han and Ming Yuan. Information based complexity for high dimensional sparse functions. Journal of Complexity , 57:101443, 2020. 2, 7

Juncai He, Lin Li, and Jinchao Xu. Approximation properties of deep ReLU CNNs. Research in the Mathematical Sciences , 9(3):1-24, 2022. 1, 2, 15

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 1, 2, 5

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017. 1

Meena Jagadeesan, Ilya Razenshteyn, and Suriya Gunasekar. Inductive bias of multi-channel linear convolutional networks with bounded weight norm. In Conference on Learning Theory , 2022. 15

Haotian Jiang, Zhong Li, and Qianxiao Li. Approximation theory of convolutional architectures for time series modelling. In International Conference on Machine Learning , 2021. 15

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 37

Jason M Klusowski and Andrew R Barron. Risk bounds for high-dimensional ridge function combinations including neural networks. arXiv preprint arXiv:1607.01434 , 2016. 6

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , 2012. 1

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 1, 2

Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861-867, 1993. 28

Shanshan Li, BoTang and Haijun Yu. Better approximations of high dimensional smooth functions by deep neural networks with rectified power units. Communications in Computational Physics , 27(2):379-411, 2019. ISSN 1991-7120. 28

Zhiyuan Li, Yi Zhang, and Sanjeev Arora. Why are convolutional nets more sample-efficient than fully-connected nets? In International Conference on Learning Representations , 2020. 1, 3, 8, 10, 15, 35, 36, 38, 44

Shan Lin and Jingwei Zhang. Generalization bounds for convolutional neural networks. arXiv preprint arXiv:1910.01487 , 2019. 15

Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations , 2019. 14, 15, 16, 27, 28

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Conference on Learning Theory , 2021. 15

Theodor Misiakiewicz and Song Mei. Learning with convolution and pooling operations in kernel methods. In Advances in Neural Information Processing Systems , 2022. 15

Andrew Y Ng. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. In International conference on Machine learning , 2004. 8, 15

Sho Okumoto and Taiji Suzuki. Learnability of convolutional neural networks for infinite dimensional input via mixed and anisotropic smoothness. In International Conference on Learning Representations , 2022. 15

Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao. Why and when can deep - but not shallow - networks avoid the curse of dimensionality: a review, 2017. 2

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks. arXiv preprint arXiv:2201.11729 , 2022. 15

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 1, 2, 5

David R So, Wojciech Ma´ nke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling. arXiv preprint arXiv:2109.08668 , 2021. 28

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996. 2

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. 15, 16, 19, 40, 43, 45

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. 3, 8, 16, 21

Lechao Xiao and Jeffrey Pennington. Synergy and symmetry in deep learning: Interactions between the data, model, and inference algorithm. In International Conference on Machine Learning , 2022. 3, 9, 15, 38

Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations , 2016. 15

Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks , 124:319-327, 2020a. 1, 2

Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis , 48(2):787-794, 2020b. 1, 2, 15

Appendix

A Related Works

Approximation and estimation. Zhou (2020b) established the first universality theorem for CNNs, which, however, requires the depth to increase as approximation accuracy improved. Later, He et al. (2022) showed a depth of O ( d ) is sufficient when multichanneling is utilized. In this paper, we further reduce the required depth to O (log d ) by combining multichanneling with downsampling. We also prove that without downsampling, CNNs require a minimum depth of Ω( d ) to achieve universality, highlighting the necessity of downsampling for deep CNNs.

Deza et al. (2020) designed some synthetic tasks to show the benefit of depth in CNNs. Okumoto and Suzuki (2022) analyzed dilated CNNs (Yu and Koltun, 2016) by considering target functions with mixed and anisotropic smoothness. They showed that dilated CNNs can capture long-range sparse signals if the dilation rate is on the order of Ω( d ) . In contrast, we show that for a similar task, vanilla CNNs can capture the long-range sparse correlations with only O (log d ) depth and ˜ O (log 2 d ) samples if downsampling is appropriately combined with multichanneling.

Long and Sedghi (2019); Lin and Zhang (2019) established the upper bounds of the covering number and Rademacher complexity of deep CNNs, where the effect of weight sharing was incorporated. We design concrete tasks to instantiate these bounds to demonstrate the superiority of CNNs over other architectures such as FCNs. Beyond that, we also improve the covering number bound of Long and Sedghi (2019) in various aspects, including improved dependence on parameter norms (from exponential to polynomial). See Appendix C.3 for a detailed discussion.

Understanding the inductive bias. Mei et al. (2021); Bietti et al. (2021) studied the superiority of convolutional kernels by exploiting the translation invariance of convolution. Later, under similar setups, Misiakiewicz and Mei (2022) studied the benefit of using local kernels in convolutions and the effect of downsampling; Bietti (2021); Cagnetta et al. (2022) studied the inductive bias of kernel composition (mimicking the effect of depth in neural networks). However, these studies are limited to kernel methods and it is generally unclear how the results are relevant for understanding CNNs. In addition, Gunasekar et al. (2018); Jagadeesan et al. (2022); Razin et al. (2022); Jiang et al. (2021); Du et al. (2018) studied the inductive biases in linear CNNs. By contrast, we examine vanilla nonlinear CNNs and show that the nonlinearity of activation functions and the network adaptivity are critical for reaping the benefits of the CNN-specific structures.

We highlight the work Li et al. (2020), where a provable separation between CNNs and FCNs was established by exploiting the equivariance groups of the learning algorithm such as SGD and Adam. However, Li et al. (2020) neither disentangled the effects of locality and weight sharing nor studied the effects of increasing network depth, multichanneling, and downsampling. Xiao and Pennington (2022) took a similar idea to explain the inductive bias of CNNs and other architectures but the authors focused on empirical investigations without providing theoretical guarantees. It is worth noting that the idea of studying a learning algorithm through its equivariance group was first introduced in Ng (2004) for linear regression and was also applied to study other hardness problems in deep learning (Abbe and Boix-Adserà, 2022).

B Technical Background

B.1 Toolbox for Concentrations

Theorem B.1. (Hoeffding's inequality, (Vershynin, 2018, Theorem 2.2.6)) Let X 1 , X 2 , . . . , X n be i.i.d. random variables with mean µ and satisfy X i ∈ [ a, b ] . Then,

Definition B.2 (Sub-exponential Random Variable) . For a random variable X , define

to be its sub-exponential norm. A random variable X that satisfies ∥ X ∥ ψ 1 < + ∞ is called a sub-exponential random variable.

Lemma B.3. (Bernstein's inequality, (Vershynin, 2018, Corollary 2.8.3)) Let X 1 , . . . , X N be independent, mean zero, sub-exponential random variables. Then, for every t ⩾ 0 , we have

where K = max i ∥ X i ∥ ψ 1 and c 2 > 0 is an absolute constant.

We will use covering number, packing number, and Rademacher complexity to measure the complexity of a function class.

Definition B.4 (Covering number) . Let ( T, ρ ) be a metric space. Consider a subset K ⊂ T and let ε > 0 . A subset K ε ⊆ K is called an ε -covering of K if every point in K is within distance ε of some point of K ε , i.e. ∀ x ∈ K, ∃ x 0 ∈ K ε : ρ ( x, x 0 ) ⩽ ε . The smallest possible cardinality of an ε -covering of K is called the covering number of K and is denoted by N ( K,ρ,ε ) .

Definition B.5 (Packing number) . Let ( T, ρ ) be a metric space and K ⊂ T . Given ε > 0 , K ε ⊂ K is said to be an ε -packing of K if ρ ( x, y ) > ε for all distinct points x, y ∈ K ε . The largest possible cardinality of an ε -packing of K is called the packing number of K and is denoted by P ( K,ρ,ε ) .

Definition B.6 (Rademacher complexity) . The empirical Rademacher complexity of a function class F on finite samples is defined as

where ξ 1 , ξ 2 , . . . , ξ n are i.i.d. Rademacher random variables: P ( ξ i = 1) = P ( ξ i = -1) = 1 2 . Let Rad n ( F ) = E [ ̂ Rad( F )] be the population Rademacher complexity.

The following lemma provides an upper bound for the covering number of a general ball.

Lemma B.7. (Long and Sedghi, 2019, Lemma A.8.) Let p be a positive integer and ∥·∥ be a norm in R p . Denote by B r = { θ ∈ R p : ∥ θ ∥ ⩽ r } . Then, N ( B r , ∥ · ∥ , t ) ⩽ ( 3 r t ) p provided t ⩽ r .

Lemma B.8 (Covering number of Lipschitz class) . Let g : X × Θ ↦→ R with Θ = R p be a general parametric model and ∥ · ∥ be a norm over Θ . Define Θ r = { θ : ∥ θ ∥ ⩽ r } and G r = { g θ : ∥ θ ∥ ⩽ r } . Suppose for any θ, θ ′ ∈ Θ r , we have | g θ ( z ) -g θ ′ ( z ) | ⩽ B ( z ) ∥ θ -θ ′ ∥ . Given z 1 , z 2 , . . . , z n ∈ X , let ˆ B n = √ 1 n ∑ n i =1 B 2 ( z i ) and ˆ ρ n ( f, g ) = √ 1 n ∑ n i =1 ( f ( z i ) -g ( z i )) 2 for any f, g ∈ F . Then for t ⩽ r ˆ B n , we have

Proof. Note that

This implies that for any t > 0 , an t/ ˆ B n -cover of Θ r w.r.t. ∥ · ∥ induces a t -cover of G r w.r.t. ˆ ρ n . Therefore,

where the last step follows from Lemma B.7. Thus, we complete the proof.

Then we recall the uniform law of large number via Rademacher complexity, which can be found in (Wainwright, 2019, Theorem 4.10).

Lemma B.9. Assume that f ranges in [0 , R ] for all f ∈ F . For any n ⩾ 1 , for any δ ∈ (0 , 1) , w.p. at least 1 -δ over the choice of the i.i.d. training set S = { X 1 , . . . , X n } , we have

Next, the Dudley's integral entropy bound given below shows that the Rademacher complexity can be further upper bounded by using the covering number.

Lemma B.10. Given n samples X 1 , . . . , X n , let ˆ ρ n ( f, g ) := √ 1 n ∑ n i =1 ( f ( X i ) -g ( X i )) 2 for f, g ∈ F and D = sup f,g ∈F ˆ ρ n ( f, g ) be the diameter of F with respect to ˆ ρ n . Then,

Corollary B.11. Given R ⩾ 1 , let F be a set of functions from X to [0 , R ] and P be an arbitrary probability distribution over X . Given n ∈ N , let X 1 , X 2 , . . . , X n be i.i.d. samples drawn from P . Assume that there exists a ˆ K n , which may depend on the training data X 1 , X 2 , . . . , X n , such that N ( F , ˆ ρ n , t ) ⩽ ( 3 ˆ K n t ) p for all 0 < t ⩽ ˆ K n . Then, for any δ ∈ (0 , 1) , w.p. at least 1 -δ we have

Proof. When 2 R ⩽ ˆ K n ,

When 2 R ⩾ ˆ K n ,

where the last step follows from the Jensen's inequality and the fact that both √ z and log( z ) are concave with respect to z . By plugging (20) into (15), we complete the proof.

B.2 A Generalization Bound for Learning with Unbounded Noise

We first recall our setup: y i = h ∗ ( x i ) + ξ i with x i iid ∼ P and ξ i iid ∼ N (0 , σ 2 ) . We assume sup x ∈X | h ∗ ( x ) | ⩽ A and denote by h θ our parametric model.

Truncation. Due to sup x ∈X | h ∗ ( x ) | ⩽ A , we can consider the truncated model π A ◦ h θ , which does not lose the expressivity but can make the mathematical analysis much simpler. In addition, due to the noise, the labels are unbounded but the generalization bound in Corollary B.11 requires boundedness. To handle this challenge, we define a truncated loss ℓ B ( y, y ′ ) := 1 2 ( y -y ′ ) 2 ∧ 1 2 B 2 . The value of B will be specified later. For the truncated loss, we have

To facilitate our statements, we define the following truncated risks

Assumption 2 (Model capacity) . Let ∥ · ∥ be a norm over the parameter space Θ used to control the model capacity. Define H J = { x ↦→ h θ ( x ) : ∥ θ ∥ ⩽ J } . Denote by p H the number of parameters. Let ∥ · ∥ U be a norm satisfying that there exists a constant α H such that

In particular,

We make the following assumptions.

Covering number. Suppose that there exists a ˆ γ n : R ⩾ 0 → R ⩾ 0 such that

Note that ˆ γ n ( · ) may depend on the training data x 1 , x 2 , . . . , x n and we let γ ( J ) = E [ˆ γ n ( J )]

Approximation error. We assume that there is θ ∗ ∈ Θ such that

The assumption (25) essentially means that h ∗ can be well approximated by our model h θ with the parameter norm ∥ θ ∗ ∥ U well controlled.

Consider the regularized estimator given by

where A,B, and λ are hyperparameters to determined later.

Remark B.12. Note that penalizing the upper bound ∥ θ ∥ U instead of ∥ θ ∥ is very common in practice. First, if the constant α H is not too big, then this choice does not hurt the performance too much. Second, the upper bound ∥ θ ∥ U is often more interpretable and easier to compute than ∥ θ ∥ .

Lemma B.13 (Properties of ˆ θ n ) . For any δ ∈ (0 , 1) , let U λ = 1 2 λ ( ϵ ∗ + σ 2 + B 2 √ 2 log (2 /δ ) n ) +

M ∗ . With probability at least 1 -δ , we have

Proof. Noting that θ ∗ is independent of the training data, we have for any δ ∈ (0 , 1) , it holds w.p. at least 1 -δ that

where ( a ) and ( b ) follow from the Hoeffding's inequality (Theorem B.1) and the approximation assumption (25), respectively.

Since ˆ θ n is one solution of the minimization problem, we have

Then, the conclusions follow trivially from the above inequality.

Proposition B.14 (Excess risk) . Denote by ˆ θ n the estimator defined in (26) . Suppose Assumption 2 holds and B > 2 A . For any δ ∈ (0 , 1 / 2) , we have w.p. at least 1 -2 δ that

Remark B.15. If γ ( J ) ∼ J β H , then by taking B = C (2 A + σ √ log n ) with C ⩾ 1 and λ ∼ 1 / √ n , we have

Thus, we show how to set the hyperparameters B,λ to yield a standard ˜ O (1 / n ) rate. Note that on the right hand side of (28) , α H , β H and p H depend on the model h θ and we will specify them for CNNs and LCNs separately in this paper. However, it should be stressed that Proposition B.14 is generally applicable, which might be of independent interest.

Proof. We will prove this theorem by utilizing the following decomposition:

Step 1. We first need the following tail bound of N (0 , σ 2 ) (Vershynin, 2018, Proposition 2.1.2): For X ∼ N (0 , σ 2 ) , it holds that

For any θ , denote Z ( x ) = π A ◦ h θ ( x ) -h ∗ ( x ) + ξ where ξ ∼ N (0 , σ 2 ) . Then, we have

where ( a ) and ( b ) follow from (29). By taking θ = ˆ θ n , we have

we have N ( F J , ˆ ρ n , t ) ⩽ N ( H J , ˆ ρ n , t/B ) ⩽ (3 B ˆ γ n ( J ) /t ) p H for t ⩽ B ˆ γ n ( J ) , where the last inequality comes from Assumption 2. For any J > 0 , by Corollary B.11 and noting γ ( J ) = E [ˆ γ n ( J )] , It is straightforward that the following holds w.p. 1 -δ

By Lemma B.13, it holds w.p. 1 -δ that ∥ ˆ θ n ∥ U ⩽ U λ , which leads to ∥ ˆ θ n ∥ ⩽ U λ /α H . Plugging it into the above bound gives: w.p. 1 -2 δ it holds that

Combining (30), (31) and (32) gives

Then we complete the proof by noting L A,B ( ˆ θ n ) = 1 2 ∥ ∥ ∥ π A ◦ h ˆ θ n -h ∗ ∥ ∥ ∥ 2 L 2 ( P ) + 1 2 σ 2 .

B.3 Fano's Method for Random Estimators

In this paper, the lower bounds for LCNs and FCNs are established using Fano's method from minimax theory (Wainwright, 2019, Section 15). However, different from the traditional statistical setup where the estimator is deterministic, the estimators produced by stochastic optimizers are random. To address this issue, we develop a variant of Fano's method for random estimators, which might be of independent interest.

We first recall the definition of Kullback-Leibler (KL) divergence.

Definition B.16 (KL Divergence) . Given two distributions Q and P , the KL divergence between them is given by

If the distributions have densities with respect to some underlying measure ν : d Q = q d ν, d P = p d ν , then then the KL divergence can be written in the form

In particular, we will use the fact: for P = N ( µ 1 , σ 2 ) and Q = N ( µ 2 , σ 2 ) , we have

and the following lemma.

Lemma B.17. Denote by Q n , P n the n -fold product distribution of Q and P , respectively. When Q is absolutely continuous with respect to P , D KL ( Q n ∥ P n ) = nD KL ( Q ∥ P )

Minimax Risk. Let { P θ : θ ∈ Θ } be a set of probability distributions indexed by θ ∈ Θ and supported on Ω . Let Z θ be a sample drawn from P θ . A random estimator A : Ω ↦→M (Θ) returns a random variable taking values in Θ , which is a random estimate of θ by using the sample Z θ . Note that in this definition, the estimator is a random variable and it can model the one produced by stochastic optimizers such as SGD and Adam. In addition, when it is deterministic, this definition recovers the one commonly used in the statistical literature.

Let d : Θ × Θ ↦→ R ⩾ 0 be a metric over Θ . The minimax risk of learning distributions in { P θ : θ ∈ Θ } is defined by

where E θ, A means the expectation with respect to Z θ and A , and the outer infimum is taken over A , i.e., the set of all the mappings from Ω to M (Θ) .

We next present a modification of Fano's method for random estimators. The proof process resembles that of the original Fano's method, but with some necessary modifications. This extension, although not difficult, can be very useful as random estimators have become ubiquitous nowdays.

̸

Theorem B.18 (Fano's method for random estimators) . Suppose that for M ∈ N and A > 0 , there exists a packing θ 1 , . . . , θ M ∈ Θ such that d ( θ i , θ j ) 2 ⩾ 4 A for any i = j , then

for some A > 0 . This will be useful for techniques based on information theory.

The principle is simple: pack the index set Θ with balls of some radius 4 A , that is find θ 1 , . . . , θ M ∈ Θ such that

̸

and transform the estimation problem into a hypothesis test, that is, an algorithm going from the data Z θ to one out of M potential outcomes.

To this end, we take the supremum over a smaller set:

Any algorithm A gives a test

where ties are broken arbitrarily (e.g., by selecting the minimal index).

̸

If, for some j ∈ [ M ] , g ( A ( Z θ j )) = j , there exists k = j , such that E A [ d ( θ k , A ( Z θ j ) ) 2 ] ⩽ E A [ d ( θ j , A ( Z θ j ) ) 2 ] . Moreover, using the triangle inequality for d ( · , · ) , we get:

̸

then,

which implies E A [ d ( θ j , A ( Z θ j ) ) 2 ] ⩾ 1 4 d ( θ j , θ k ) 2 ⩾ A . Thus, we have

leading to

̸

which can be further lower bounded using the following lemma (Bach, 2023, Corollary 12.1).

Lemma B.19 (Fano's inequality for multiple hypothesis testing) . Given M distributions P 1 , P 2 , . . . , P M over Ω , let Z θ j ∼ P θ j for j = 1 , 2 , . . . , M . Then

̸

where the infimum is taken over all testing methods, i.e., all maps from Ω to [ M ] .

Combining Lemma B.19 and equation (43), we complete the proof.

In this paper, we will mostly use a variant of the above Fano's inequality for regression problems. Recall that in regression, the data are generated by y = h ( x ) + ξ with x ∼ P and ξ ∼ N (0 , σ 2 ) . Let Q h be the joint distribution of ( x , y ) and S n = { ( x i , y i ) } n i =1 be n i.i.d. samples drawn from Q h . Denote by Q n,h the n -fold product distribution of Q h . Let F be the class of target functions. In such a case, the set of probability distributions is indexed by h ∈ F :

̸

̸

̸

Proposition B.20 (Fano's method for regression) . Let F be a target function class. If there exists a packing h 1 , . . . , h M ∈ F such that for any i = j , ∥ h i -h j ∥ 2 L 2 ( P ) ⩾ 4 A , we have

where the expectation is taken over both the sampling of S n and randomness of A .

Proof. Define d ( h, h ′ ) 2 = ∥ h -h ′ ∥ 2 L 2 ( P ) . WLOG, assume that P has a density function p and denote by ϕ h,x the density function of N ( h ( x ) , σ 2 ) . Then, the density function of Q h is p ( x ) ϕ h,x ( y ) . Then, we have

where the first equality follows from Lemma B.17 and the third equality is due to equation (34). We remark that this equality is also right when P does not admit a density. In this case, we can construct and verify the Radon-Nikodym derivative, and then compute the KL divergence by the definition. Lastly, by plugging the above estimate into (36) in Theorem B.18, we complete the proof.

Remark B.21. When we use Fano's method (Proposition B.20) to lower-bound the minimax risk, the most important step is to construct a 'proper' packing. Take a look at the lower bound in (45) . For fixed M ∈ N and A > 0 , a proper packing should ensure ∑ M i,j =1 ∥ h i -h j ∥ 2 L 2 ( P ) to be as small as possible; otherwise the packing may yield a suboptimal lower bound. In fact, in most applications, we will simply attempt to bound the worst-case resolution: sup i,j ∥ h i -h j ∥ 2 L 2 ( P ) .

C Capacity Control for CNNs and LCNs

We first define a norm on the parameter space for controlling the model capacity effectively. Let θ = { θ 1 , θ 2 , . . . , θ L , W o } denote all the parameters of our network, where θ l ∈ Θ l and W o ∈ Θ o denote the parameters used to parameterize T l and M o , respectively.

Denote by V l the space of l -th layer hidden state. Let ˜ V l ∈ R p l be the vectorized version of V l armed with standard ℓ 2 norm and ˜ T l : ˜ V l -1 ↦→ ˜ V l be the corresponding representation of T l . Since ˜ T l is linear, there must exist unique A : Θ l ↦→ R p l × p l -1 and B : Θ l ↦→ R p l such that ˜ T l vec( z ( l ) ) = A ( θ l ) vec( z ( l ) ) + B ( θ l ) . Then we define

to control the 'magnitude' of ˜ T l , which ensures that for any ˜ z ∈ ˜ V l -1 we have ∥ ˜ T l ˜ z ∥ 2 ⩽ ∥ ˜ T l ∥ ( ∥ ˜ z ∥ 2 + 1) . Furthermore, we consider a new norm over the whole parameter space Θ by

It is equivalent to say ∥ θ ∥ = ∥M o ∥ + ∑ L l =1 ∥ ˜ T l ∥ , which is the sum of operator norms of all linear transforms.

We remark that we will show the norm (47) provides tight control of covering number of CNNs and LCNs, but is less informative and interpretable. Thus, we introduce the ∥ θ ∥ P norm in (3) for CNNs and LCNs, which is an upper bound of ∥ θ ∥ as demonstrated below. As opposed to ∥ θ ∥ , ∥ θ ∥ P is much more explicit, interpretable, and easier to compute. In the following, for both CNNs and LCNs, we will first derive the matrix form of ˜ T l , thereby obtaining A ( · ) and B ( · ) . Abusing notation, we will use T l to denote ˜ T l for simplicity.

C.1 CNNs

Recall that for l ∈ [ L ] , T l : R D l -1 × C l -1 ↦→ R D l × C l is parameterized by a kernel W ( l ) ∈ R C l × C l -1 × s and bias b ( l ) ∈ R C l as follows

We further point out that the number of parameters in our CNN is

A patch-based reformulation of CNNs. For technical simplicity, it is easier to take a patch-based viewpoint of T l : T l operates on each patch with an identical local linear transform. Note that D l = D l -1 /s and we can divide z ( l -1) into D l patchs of the shape s × C l -1 :

where I j := [( j -1) s + 1 , js ] denotes the j -th patch. Then T l essentially makes the following transform

where A ( l ) : R s × C l -1 → R 1 × C l is the local patch transform given by

In a matrix form, A ( l ) is given by where

The matrix form of T l . Let

Abusing the notation, we still use T l : R D l -1 C l -1 → R D l C l to denote the matrix form of the one defined in (48). Then, we have

where

where the block matrix has D l blocks and ˜ W ( l ) is given by (50).

The parameter norm. By (51), it is easy to see that

Then, we have

Lemma C.1. For any θ, θ ′ satisfy that ∥ θ ∥ ⩽ J and ∥ θ ′ ∥ ⩽ J and any x ∈ X , assume σ l satisfies σ l (0) = 0 and locally Lipschitz condition ∥ σ l ( y ) -σ l ( z ) ∥ 2 ⩽ Q l ( x ) ∥ y -z ∥ 2 for any y , z ∈ {T l ◦ · · · ◦ σ 1 ◦ T 1 ( x ) : ∥ θ ∥ ⩽ J } . We further denote ¯ Q σ ( x ) = ∏ L l =1 ( Q l ( x ) + 1) . Then, we have

Proof. First assume that θ, θ ′ only differ in the l -th layer, l = 1 , . . . , L .

Second, assume θ, θ ′ only differ in the final layer. We have

Lastly, we consider general θ, θ ′ with ∥ θ ∥ ⩽ J and ∥ θ ′ ∥ ⩽ J . Weclaim that one can find θ 1 , . . . , θ L +1 such that

∥ θ l ∥ ⩽ J for all l ∈ [ L +1] ;

θ ′ = θ L +1 , θ and θ 1 only differ in one layer;

for any l , θ l and θ l +1 only differ in one layer.

It is easy to check that this can be done by first replacing the maximum norm layer in θ by the minimum norm layer in θ ′ , and so on.

Then, by applying the telescoping sum, we complete the proof.

Theorem C.2 (Covering number of CNN) . Let h θ denote the CNN described in Section 2.1 and define H CNN J = { h θ : ∥ θ ∥ ⩽ J } . Given x 1 , x 2 , . . . , x n ∈ X , denote ˆ M n = √ 1 n ∑ n i =1 ( ¯ Q σ ( x i ) ) 2 ( ∥ x i ∥ 2 +1) 2 . Then, we have

given 0 < t ⩽ ˆ M n J (1 + J ) L .

Proof. The conclusion follows trivially from Lemma C.1 and Lemma B.8.

C.2 LCNs

Recall that for LCNs, the linear transform T l : R D l -1 × C l -1 ↦→ R D l × C l is parameterized by W ( l ) ∈ R C l × C l -1 × D l -1 and b ( l ) ∈ R D l × C l as follows

We further point out that the number of parameters in our LCN model is

A patch-based reformulation of LCNs. We reformulate deep LCNs in the same way as CNNs. Let ˜ z ( l ) ∈ R D l C l be the vectorized features; then the matrix form of T l : R D l -1 C l -1 → R D l C l is given by

where K ( l ) = diag { ˜ W ( l ) 1 , · · · , ˜ W ( l ) D l } and ˜ W ( l ) i ∈ R C l × ( sC l -1 ) , where

where I j = [( j -1) s +1 , js ] .

Parameter norm. By (59), it is obvious that

Thus

In the following, we provide an upper bound of the covering number of deep LCNs, whose proof is nearly the same as that of deep CNNs.

Lemma C.3. For any θ, θ ′ satisfy that ∥ θ ∥ ⩽ J and ∥ θ ′ ∥ ⩽ J and any x ∈ X , assume σ l satisfies σ l (0) = 0 and locally Lipschitz condition ∥ σ l ( y ) -σ l ( z ) ∥ 2 ⩽ Q l ( x ) ∥ y -z ∥ 2 for any y , z ∈ {T l ◦ · · · ◦ σ 1 ◦ T 1 ( x ) : ∥ θ ∥ ⩽ J } . We further denote ¯ Q σ ( x ) = ∏ L l =1 ( Q l ( x ) + 1) . Then, we have

The proof is exactly the same as that of Lemma C.1, i.e., the case of CNNs.

Proof. First assume that θ, θ ′ only differ in the l -th layer, l = 1 , . . . , L .

Second, assume θ, θ ′ only differ in the final layer. We have

Lastly, we consider general θ, θ ′ with ∥ θ ∥ ⩽ J and ∥ θ ′ ∥ ⩽ J . Weclaim that one can find θ 1 , . . . , θ L +1 such that

∥ θ l ∥ ⩽ J for all l ∈ [ L +1] ;

θ ′ = θ L +1 , θ and θ 1 only differ in one layer;

for any l , θ l and θ l +1 only differ in one layer.

It is easy to check that this can be done by first replacing the maximum norm layer in θ by the minimum norm layer in θ ′ , and so on.

Then, by applying the telescoping sum, we complete the proof.

Theorem C.4 (Covering number of LCNs) . Let h θ denote the LCN model described in Section 2.1 and define H LCN J = { h θ : ∥ θ ∥ ⩽ J } . Given x 1 , x 2 , . . . , x n ∈ X , denote ˆ M n = √ 1 n ∑ n i =1 ( ¯ Q σ ( x i ) ) 2 ( ∥ x i ∥ 2 +1) 2 . Then, we have

given 0 < t ⩽ ˆ M n J (1 + J ) L .

Proof. The conclusion follows trivially from Lemma C.3 and Lemma B.8.

Remark C.5. Note that the number of parameters of a LCN is much larger that of its CNN counterpart due to the lack of weight sharing. As a result, the covering number of LCNs is accordingly much larger than that of CNNs.

C.3 Comparison with Long and Sedghi (2019).

We acknowledge that our proofs follow a similar approach developed in Long and Sedghi (2019) but we improve Long and Sedghi (2019) in various aspects.

We provide an upper bound of the covering number of LCNs while Long and Sedghi (2019) only considered CNNs. Moreover, our bounds apply to activation functions that are not Lipschitz continuous such as the squared ReLU: σ ( z ) = max(0 , z ) 2 . Theis activation function has become popular recently in solving scientific computing problems (Li and Yu, 2019; E and Yu, 2018) and training large NLP pre-trained models (So et al., 2021).

The bound in Long and Sedghi (2019) depends on the parameter norm J exponentially. In contrast, our bound depends on J only polynomially. This improvement is critical for obtaining a sharp generalization bound for the case where the parameter norm is relatively large. For instance, by applying Long and Sedghi (2019) to the setting of learning the separation task in Theorem 5.5, the corresponding sample complexity bound of LCNs will become ˜ O ( d 2 ) . In contrast, our bound is ˜ O ( d ) . This improvement is very critical since the lower bound of learning with FCNs is Ω( d 2 ) and therefore, without this improvement, we are unable to establish the provable separation between LCNs and FCNs.

D Universal Approximation: Proofs in Section 3

D.1 Proof of Theorem 3.1

We first recall a well-known universality result of two-layer ReLU networks.

Lemma D.1. (Leshno et al., 1993) Let Ω be any compact set in R d and σ be the ReLU function. For any h ∈ C (Ω) and any ϵ > 0 , there exists a a two-layer neural network f m ( x ; θ ′ ) = ∑ m j =1 a j σ ( u ⊤ j x + c j ) such that

Lemma D.2 (Feature extraction) . With the choice of architecture in Theorem 3.1, the exist parameters such that z ( L -1) ( x ) ∈ R 2 × (4 d ) satisfies

Remark D.3. This lemma shows that z ( L -1) ( x ) strictly stores spatial information into different channels. Note that 4 d is the number of channels and 2 is the spatial dimension. The stored information information is in the form of { σ ( x i ) , σ ( -x i ) } 4 d i =1 . Noticing t = σ ( t ) -σ ( -t ) , we can conclude that there is no information loss since for any h : R 4 d ↦→ R , h ( x 1 , x 2 , . . . , x 4 d ) can be represented using the stored features as follows

It is worth noting that obviously, the weights and bias in T 1 , . . . , T L -1 do not depend on the target function h ∗ .

Proof. We provide a constructive proof to this lemma. First, we set all bias to be zero, i.e., b ( l ) = 0 for all l ∈ [ L -1] . Next we state how to set the weights for different layers.

When l = 0 , z (0) = x = ( x 1 , x 2 , . . . , x 4 d ) .

When l = 1 , set W (1) ∈ R 1 × 4 × 2 as follows W (1) 1 , 1 , : = (1 , 0) ⊤ , W (1) 1 , 2 , : = ( -1 , 0) ⊤ , W (1) 1 , 3 , : = (0 , 1) ⊤ , W (1) 1 , 4 , : = (0 , -1) ⊤ . Under this construction, it is easy to verify that

Similarly, for l = 2 , we can set W (2) ∈ R 4 × 8 × 2 such that

By induction, when l = L -1 , we have

Specifically, the above result is achieved by setting the weights of l = 2 , . . . , L -1 as follows 2 . For 1 ⩽ j ⩽ C l -1 and j is odd, we set

Recall that W ( l ) i,j, : represent the filter used to extract the information from the i -th input channel to the j -th output channel. Let j ′ = ( j +1) / 2 if j is odd. To prove the above result, we only show that for any l ∈ [ L -1] ,

We verify this below by induction.

First, the case of l = 1 holds due to (68). Assume (68) holds for 1 , . . . , l -1 , let us compute ( T l ( z ( l -1) )) : ,j . Without loss of generality, we only consider the case where j is odd.

When j ⩽ C l -1 ,

Hence,

When j ⩾ C l -1 we similarly have

So

Thus, the case of l +1 also holds.

Proof of Theorem 3.1. By setting the weights and biases of the first L -1 layers according to Lemma D.2, we have h θ ( x ) = M o ◦ σ ◦ T L ( z ( L -1) ( x )) , where

By Lemma D.1, for any h ∗ ∈ C (Ω) , there exists a a two-layer neural network f m ( x ; θ ′ ) = ∑ m j =1 a j σ ( u ⊤ j x + c j ) such that sup x ∈ Ω | f m ( x ; θ ′ ) -h ∗ ( x ) | ⩽ ϵ. Then our proof is completed by showing that we can construct appropriate T L and M o such that the deep CNN h θ can simulate the two-layer neural network f m ( · ; θ ′ ) .

set C L = m , b ( L ) j = c j . For odd i , let i ′ = ( i +1) / 2 and set W ( L ) i,j, : = (( u j ) i ′ , ( u j ) i ′ +2 d and for even i we set W ( L ) j,i, : = -W ( L ) j,i -1 , : . Under this construction,

Set W o = ( a 1 , . . . , a m ) . Then, the CNN h θ represents exactly the same function as f m ( · ; θ ′ ) :

Thus, we complete the proof.

D.2 Proof of Proposition 3.2

Given two functions f, g over X , let ρ ∞ ( f, g ) := sup x ∈X | f ( x ) -g ( x ) | .

We prove this theorem by contradiction. First, any CNN h θ with the depth L ⩽ log 2 ( d ) + 1 can be represent as

for some g 1 and g 2 . This is because that the size of receptive field is no greater than 2 d . Obviously, h θ cannot represent long-range functions like h ∗ ( x ) := x 1 x 2 d +1 . Specifically, we have

If there exist g 1 and g 2 such that

then taking special x 's gives

From the first three inequalities, we have

which is contradictory to the fourth inequality in (72). Thus (71) cannot hold and we complete the proof.

D.3 Proof of Proposition 3.3

We first recall that ρ ∞ ( f, g ) := sup x ∈X | f ( x ) -g ( x ) | .

We prove this theorem by contradiction. First, without downsampling, any CNN with depth L ⩽ 4 d -2 can be represented as

for some g 1 , g 2 . In this form, x 4 d and x 1 do not have direct correlation and intuitively, it should be impossible to represent functions like h ∗ ( x ) := x 1 x 4 d by using this CNN. This intuition can be made rigorously as follows. Note that

If there is g 1 , g 2 such that

then

From the first three inequalities, we have

which is contradictory to the fourth inequalities in (76). Therefore, (75) cannot hold and we complete the proof.

E Learning Sparse Functions: Proofs of Section 4

E.1 Proof of Lemma 4.4

Here we prove a complete version of Lemma 4.4.

Lemma E.1 (Adaptive coordinate selection) . Given k, m ∈ N , consider a ReLU CNN model with depth L = log 2 (4 d ) and the channel numbers satisfying C l = 2 k for all l = 1 , . . . , L -1 and C L = m . Then for any I = ( i 1 , . . . , i k ) ⊂ [ d ] with 1 ⩽ i 1 < · · · < i k ⩽ d , u 1 , . . . , u m ∈ R k , and c 1 , . . . , c m ∈ R , there exists θ ∈ Θ such that the CNN h θ outputs:

Furthermore, for this CNN, the number of parameters is O ( k 2 log d + km ) and the parameter norm satisfies

In this case, the proof is the same as that of linear CNNs (Lemma 4.2) since we only need to double the channels: every two channels form a group, storing { σ ( x i ) , σ ( -x i ) } k i =1 ; different groups of channels proceed still independently like the case of linear CNNs. Specifically, for l = 1 , 2 , . . . , L -1 , we follow a similar idea to set weights and biases such that

where for any j ∈ [ k ] , either t j = i j or s j = i j . Then, we set the weights and bias of L -th layer according to { u i } i m =1 and { c i } i m =1 such that the outputs are { σ ( u ⊤ i x I + c i ) } i m =1 . We refer to the following proof for details.

Proof. For any j ∈ [ k ] , we write the i j 's binary represention as follows

where a j,l ∈ { 0 , 1 } . Here, ' -1 ' is used to ensure the i j ∈ [0 , 2 L -1] . Next, we set the weights and bias according to the binary representation (77).

Set weights and bias adaptively :

For l = 1 , W (1) ∈ R 1 × 2 k × 2 . For j ∈ [ k ] , set

For l = 2 , . . . , L -1 , W ( l ) ∈ R 2 k × 2 k × 2 . For j ∈ [ k ] , set

For any l ∈ [ L -1] , set all entries in W ( l ) unmentioned above and b ( l ) to zeros.

Note that a j,l ∈ { 0 , 1 } for all j ∈ [ k ] and l ∈ [ L ] . Thus, under the above construction, all nonzero filters are taken from { (1 , 0) , ( -1 , 0) , (0 , 1) , (0 , -1) } .

The forward selection process . Let us compute T l z ( l -1) for l = 1 , . . . , L -1 . Let s j,l = 1 + ∑ l -1 p =0 a j,p 2 p . We claim that for l ∈ [ L -1] ,

We will verify this by induction.

For l = 1

So when j is even ( T 1 z (0) ) : ,j = -( T 1 z (0) ) : ,j -1 and when j is odd we have

where j ′ = ( j +1) / 2 as above. l = 1 has been verified.

Then, assume that the case of 1 , · · · , l -1 hold, we compute ( T l z ( l -1) ) : ,j . Without loss of generality, we only consider the case where j is odd. Observing that

which implies

Thus, the claim is verified and after L -1 layer,

For the L -th layer, for j = 1 , 2 , . . . , k set

Thus we complete the proof of the first part.

Bounding the parameter norm . Under the above construction, we have:

When l = 1 , 2 , . . . , L -1 , there exists an absolute constant C > 0 such that ∥ W ( l ) ∥ F ⩽ C √ k and b ( l ) = 0 . Thus, by equation (52) we have

Furthermore, the parameter norm in L -th layer satisfies

Combining them, we complete the proof.

We compute

E.2 Proof of Theorem 4.6

We will need following approximation result of two-layer ReLU networks .

Lemma E.2. (A restatement of (E et al., 2022, Theorem 4)) For any f ∈ B , any probability distribution P on X = [0 , 1] d and integer m ⩾ 1 , there exists a two-layer neural network f m ( x ; θ ) = 1 m ∑ m k =1 a k σ ( u ⊤ k x + c k ) where θ denotes the parameters { ( a k , u k , c k ) , k ∈ [ m ] } in the neural network, such that

Proposition E.3. Let X = [0 , 1] 4 d . Suppose h ∗ ( x ) = g ∗ ( x I ) with |I| = k . Consider deep ReLU CNNs with L = log 2 (4 d ) and C l = 2 k for l ∈ [ L -1] and C L = m . Then, for any g ∗ ∈ B , there exist θ ∗ such that

with the number of parameters being O ( k 2 log d + km ) and

Proof. Using lemma E.2, there exists a two layer ReLU network f m ( x I ) = 1 m ∑ m k =1 a k σ ( u ⊤ k x I + c k ) such that

By Lemma 4.4, we can choose the weights and bias such that the feature map of L -th layer is

By setting the parameters of output layer as W o = 1 m ( a 1 , . . . , a m ) , it is easy to check that h θ ∗ ( x ) = M o ◦ z ( L ) ( x ) = f m ( x I ) and the parameter number N cnn = O ( k 2 log d + km ) .

Next, we turn to bound the parameter norm. Under the above parameter setting, we have

Applying Lemma 4.4 gives

Thus, we complete the proof.

Furthermore, we have

Proof of Theorem 4.6. Let H CNN J = { h θ : ∥ θ ∥ ⩽ J } . By proposition B.14 and the covering number satisfies

where ˆ γ n ( J ) = ˆ M n J (1+ J ) L and N cnn = O ( k 2 log d + km ) . Here we use the fact that ¯ Q σ ( x ) ⩽ 2 L for all x and L = log 2 d +2 for deep ReLU CNNs. In addition, note that

where the second inequality follows from the Jensen's inequality since √ · is concave.

Recalling we have α H = 1 from equation (53) and applying Proposition B.14, we have

where

Taking A = 1 , B = 2 + σ √ log n, λ = ϵ/n 2 , m = 6 ∥ g ∗ ∥ 2 B ϵ and applying Lemma B.13, we have

where

Simplify the right side of equation (87) becomes

This implies that, for a target accuracy ϵ and failure probability δ ,

is enough.

F Symmetries of Learning Algorithm and Proofs of Lemma 5.3 and 5.6

F.1 Learning Algorithm and Group Equivariance

(Li et al., 2020, Appendix C) established a framework to verify the group equivariance of an iterative algorithm, which however considers neither stochastic nor adaptive algorithms such as Adam. In this

section, we provide an extension of (Li et al., 2020, Appendix C) to include these situations. It should be stressed that the extension is mostly straightforward compared with (Li et al., 2020, Appendix C) and we provide the extension here only for clarity and completeness.

In the following, denote A d = B as Law( A ) = Law( B ) . Let G X be a group acting on X . Then, for any τ ∈ G X and S n = { x i , y i } n i =1 , let τ ( S n ) = { ( τ ( x i ) , y i ) } n i =1 . We also occasionally write the group action τθ = τ ( θ ) for simplicity when it is clear from the context.

Recall that h θ is our parametric model and θ ∈ Θ = R p . We make the following assumption.

Assumption 3. Given a group G X acting on X . We assume that there exist a group G Θ acting on Θ and a group isomorphism Q : G X → G Θ such that for all τ ∈ G X ,

The above assumption is satisfied by both FCNs and LCNs and we will verify it later. Here, we take linear regression as an example to gain some intuition. In such a case, h θ ( x ) = ⟨ w , x ⟩ . Given the group G X = O ( d ) acting on the input domain, we can set Q = id and G Θ = O ( d ) . Then, for any U ∈ O ( d ) , we have h Q ( U )( θ ) ◦ U ( x ) = ⟨ U w , U x ⟩ = ⟨ w , x ⟩ = h θ ( x ) .

Suppose Assumption 3 holds. Given a learning algorithm A : ( X × Y ) n ↦→ M (Θ) , we say A is G X -equivariant if ∀ S n ∈ ( X × Y ) n and τ ∈ G X :

We can also informally define the equivariance in the model space as follows

where h A ( S n ) and h A ( τ ( S n )) should be understood as random variables taking values in the space of all the algorithms A . The definition (89) is intuitive and has been adopted in Abbe and Boix-Adserà (2022); Li et al. (2020) but it should be stressed that (89) is not rigorous as it is generally unclear how to deal with the measurability issue in A , the space of algorithm. We thus will adopt (88) for its rigorous nature.

Iterative algorithm. In the following, we will focus on the learning algorithm given by

where F t : (Θ) t × ( X × Y ) n × Ω ↦→ Θ is a deterministic update map and { ξ t } T t =1 are i.i.d. freshly generated random variables taking values in Ω , which encode the algorithm randomness and ξ t is independent of S n and { θ 0 , θ 1 , . . . , θ t } .

Take SGD as a concrete example. Let Ω = { 0 , 1 } n denote the set of minibatch selection masks and the corresponding algorithm map is given by

Note that one should not confuse ξ t with the SGD noise, where latter is state-dependent.

Assumption 4. · P is G

init Θ -invariant.

∀ τ ∈ G X , θ ∈ Θ , t ∈ N , S n ∈ ( X × Y ) n , and ξ ∈ Ω :

The second assumption can be thought as that the update map F t ( · ) is equivariant under the joint group action ( τ, Q ( τ )) . Still taking the linear regression as an example, let X = ( x 1 , . . . , x n ) ∈ R d × n be n inputs and y ∈ R n be the labels. Then, the GD update for minimizing ˆ L ( w ) := 1 2 ∥ ∥ X ⊤ w -y ∥ ∥ 2 2 would be

Let G X = G Θ = G ort ( d ) , Q = id . We thus have for any U ∈ O ( d ) , Q ( U ) = U and

which verifies the equivariance (91).

Proposition F.1. Let A T denote the iterative algorithm (90) . Under Assumption 3 and 4, we have for any T ∈ N , A T is G X -equivariant.

Proof. Fix S n ∈ ( X × Y ) n and τ ∈ G X . Let { θ t } ⊤ t =0 and { ˜ θ t } ⊤ t =0 be the trajectories independently generated by the iterative algorithm by using the data S n and τ ( S n ) , respectively. By the definition (88), it suffices to prove that for any t ∈ N ,

Next we prove it by induction.

When T = 0 , θ 0 , ˜ θ 0 ∼ P init . For any τ ∈ G X , we have Q ( τ ) ∈ G Θ . The assumption that P init is G Θ -invariant implies ˜ θ 0 d = Q ( τ ) θ 0 .

Assume (92) holds for T = 0 , 1 , . . . , t . Let { ξ t } and { ˜ ξ t } be the 'noise' used to generate { θ t } and { ˜ θ t } , respectively. Then, we have

where the second step follows from Assumption (4); the third step follows from the assumption that (92) holds for T = t . Thus, we prove that (92) holds for T = t +1 .

By induction, we complete the proof.

F.2 Verifying the Group Equivariance of SGD and Adam

Let ℓ ( · , · ) be a general loss function. Consider a regularized empirical risk

where p ⩾ 1 and and r : [0 , + ∞ ) → [0 , + ∞ ) .

At the t -th step, let S t = { i 1 , . . . , i k } be k i.i.d. indices uniformly drawn from [ n ] . Define the minibatch risk by

In each step, stochastic gradient descent (SGD) updates as follows

Adam optimizer (Kingma and Ba, 2014) updates as follows

where α, β ∈ (0 , 1) , ϵ > 0 , and the square and division should be understood in an element-wise manner. We just consider initialization v 0 = m 0 = 0 for simplicity. For both SGD and Adam, η t is the learning rate at the t -th step.

FCNs: Proof of Lemma 5.6 For h fcn θ , write θ = ( w 1 , . . . , w m , ¯ θ ) ∈ R p , where m is the width of the first layer, w j ∈ R d for j = 1 , . . . , m denotes the weights of the first layer, and ¯ θ denotes other parameters. Then, we can write h fcn θ ( x ) = g ¯ θ ( w ⊤ 1 x , . . . , w ⊤ m x ) for some g : R m × R p -md ↦→ R .

Consider G X = G ort ( d ) and

Let G Θ := { Q ( U ) : U ∈ G X } . Then, it is not hard to verify that G Θ is a group under matrix product and Q : G X ↦→ G Θ is a group isomorphism.

First, it is not hard to verify that the Gaussian initialization P init is invariant under G Θ . To verify (91) for SGD, we can simply consider the case where n = 1 and S n = { ( x , y ) } without loss of generality. First, we can show that there exist functions { s i ( · ) } i m =1 and O ( · ) such that

where W = ( w 1 , w 2 , . . . , w m ) ∈ R d × m . It is not hard to verify that (91) holds for the above update map. Then, Lemma 5.6 follows trivially from Proposition F.1.

LCNs: Proof of Lemma 5.3 Here we only prove Lemma 5.3 for the Adam optimizer since the SGD case is similar to the proof above.

WLOG, we let C 1 = 1 in the LCN model. Denote by ¯ θ = W (1) 1 , 1 , : the the filter weights of the first layer and ¯ θ c all the other parameters. Then, θ = { ¯ θ, ¯ θ c } and W (1) 1 , 1 , : ∈ R D 0 . Let G X = G loc and define

Let G Θ = { Q ( U ) : U ∈ G X } . It is not hard to verify that G Θ is a group under the matrix multiplication and Q is the group isomorphism.

First, it is obvious that under Assumption 1, P init is G loc -invariant. To verify (91) for Adam, we can simply consider the case where r ( · ) = 0 , n = 1 and S n = { ( x , y ) } without loss of generality. In this case, for the first step, we have

for some function { s i ( · ) } D 1 i =1 and O ( · ) . Here we recall that in our paper s = 2 and the local linear operater ⋆ s : R ks × R ks ↦→ R k is defined by v ⋆ s w = ( v ⊤ I 1 w I 1 , v ⊤ I 2 w I 2 , . . . , v ⊤ I k w I k ) , where I j = [( j -1) s +1 , js ] denotes the indices of j -th patch.

Noting ( U ¯ θ 0 ) ⋆ s ( U x ) = ¯ θ 0 ⋆ s x for any U ∈ G loc , it is therefore not hard to verify that (91) holds for the update map (96). Thus we show the G loc -equivariance of Adam for training LCNs with T = 1 . The case of T > 1 can be shown in the same way.

Remark F.2. Note that the case of FCNs was established in (Li et al., 2020, Corollary C.2) but the proof there is not fully rigorous. We here provide a completely rigorous proof. The case of LCNs under Gaussian initialization was studied in Xiao and Pennington (2022), where the equivariance group is G ort (2) ⊗ I 2 d . We instead show that under much milder condition on initialization, the equivariance group becomes the local permutation group (10) . It should be stressed that the major contribution of this section is providing a rigorous/unified framework to verify these group equivariance instead of yielding new insights.

G CNNs vs. LCNs: Proofs of Section 5.2

G.1 Proof of Theorem 5.4

By Lemma 5.1, we only need to lower bound the minimax error of learning the enlarged class { ¯ h ∗ } ◦ G loc . To this end, we need to find a proper packing of this enlarged class and then apply the Fano's method, i.e., Proposition B.20. To this end, we will prove

Lemma G.1. There exist absolute positive constants C, c 1 , c 2 > 0 , c 3 > 1 such that if d ⩾ C and A 0 ⩾ C , there exists h 1 , h 2 , . . . , h M ∈ { ¯ h ∗ } ◦ G loc satisfying that

̸

Assuming Lemma G.1 is right for now and applying the above packing to Fano's inequality (Proposition B.20), we have

Taking n such that the RHS of the above inequality satisfies

gives n = Ω( σ 2 d ) . Thus, we can conclude that C ( { ¯ h ∗ } ◦ G loc , ϵ 0 ) = Ω( σ 2 d ) . Thus, we complete the proof of Theorem 5.4.

G.1.1 Proof of Lemma G.1

We now show how to construct a packing of { ¯ h ∗ } ◦ G loc to satisfy the condition in Lemma G.1. The key idea is to reduce the problem to pack the equivariance group G loc . For U = diag ( U 1 , U 2 , . . . , U 2 d ) ∈ G loc , denote by U i ∈ R 2 × 2 the i -th block matrix. Define a metric over G loc : for any U, U ′ ∈ G loc ,

̸

In addition, we consider a subgroup of G loc :

Clearly, a packing of { ¯ h ∗ } ◦ G semiloc is also a packing of { ¯ h ∗ } ◦ G loc . For G semiloc , we have Lemma G.2. There exist absolute constants C, c 1 , c 2 > 0 such that if d, A 0 ⩾ C , for any τ, τ ′ ∈ G semiloc , we have

Proof idea. The proof of this lemma is quite technically lengthy and is thus deferred to Appendix I.1. Recall that ¯ h ∗ = π A 0 ◦ h ∗ . If there is no truncation, i.e., A 0 = ∞ , the proof is straightforward in the sense that a direct calculation gives ∥ h ∗ ◦ τ -h ∗ ◦ τ ′ ∥ 2 L 2 ( P ) = cρ loc ( τ, τ ′ ) /d for some absolute constant c . When A 0 is finite but large enough, by concentration inequalities, we can expect that the truncated target function works in a way similar to the orginal one. Indeed the lengthy part in the proof is to estimate the influence of truncation.

In the following, we show that G semiloc is isometric to the Hamming space:

Definition G.3 (Hamming Space) . The Hamming cube { 0 , 1 } n consists of all binary strings of length n . The Hamming distance d H ( x, y ) between two binary strings is defined as the number of bits where x and y disagree, i.e.

̸

We call ( { 0 , 1 } n , d H ) the Hamming space.

Specifically, it is obvious that

( G loc , ρ loc ) and ( { 0 , 1 } 2 d , d H ) are isometric.

( G semiloc , ρ loc ) and ( { 0 , 1 } d , d H ) are isometric.

For the Hamming space, we have the following bounds for packing and covering numbers.

Lemma G.4. (Vershynin, 2018, Excercise 4.2.16) Let K = { 0 , 1 } n . For every 0 < m ⩽ n , we have

Proof. Without loss of generality we set m as an integer. The first inequality follows trivially from the definition and thus, we only need to prove the second one.

Set B m,x = { y ∈ K : d H ( x, y ) ⩽ m } . For any x ∈ K , any k ⩽ m , there are ( n k ) elements satisfing d H ( x, y ) = k . This is because the ways of choosing k elements from n elements are ( n k ) , and d H ( x, y ) = k if and only if x and y have k different coordinates. By taking a union bound we have | B m,x | ⩽ ∑ m k =0 ( n k ) , where | · | denotes the cardinality. The ball of radius m and center x can only cover at most | B m,x | elements, However there are 2 n elements in K . So we have

By the isometries and Lemma G.4, we trivially have

Lemma G.5. For any 0 < m ⩽ d ,

To further simplify the lower bounds, we need

Lemma G.6. (Vershynin, 2018, Exercise 0.0.5) Given m ∈ [ n ] , ∑ m k =0 ( n k ) ⩽ ( en m ) m .

Proof. To prove this we only need to prove ( m n ) m ∑ m k =0 ( n k ) ⩽ e m , and we observe that

Proof of Lemma G.1. Note that for any τ, τ ′ ∈ G semiloc , ρ loc ( τ, τ ′ ) ⩽ d . Then, applying Lemma G.2, we have for any h = ¯ h ∗ ◦ τ, h ′ = ¯ h ∗ ◦ τ ′ ∈ { ¯ h ∗ } ◦ G semiloc that

In addition, Lemma G.2 implies ∥ h -h ′ ∥ 2 L 2 ( P ) ⩾ c 1 d ρ loc ( τ, τ ′ ) . Thus, for 0 < δ ⩽ 1 2 √ c 1 :

where the last step follows from Lemma G.5. By Lemma G.6 and taking δ ∗ = 1 2 √ c 1 , we have

Let c 3 = 2 (5 e ) 1 / 4 , which is greater than 1 .

Combining equation (104) and (105), we complete the proof.

G.2 Proof of Theorem 5.2

Let σ ( z ) := ReLU( z ) in this subsection for notation simplicity. We first recall the architecture of the CNN model: L = log 2 (4 d ) , channel number C 1 = C L = 4 , C l = 2 for l = 2 , . . . , L -1 , and activation functions σ 1 ( z ) = σ L ( z ) = σ 2 ( z ) , σ l ( z ) = σ ( z ) for l = 2 , . . . , L -1 .

Step I: Representing the target using CNNs .

Lemma G.7. There is a parametrization θ ∗ such that h cnn θ ∗ = h ∗ and ∥ θ ∗ ∥ P ≲ log d .

Proof. We provide a constructive proof of this lemma. We fist set all the bias to zeros. Then we set the parameter of the filters W ( l ) ∈ R C l × C l -1 × 2 as follows.

For the final linear layer, W o = 1 4 d (1 , 1 , -1 , -1) .

First, it is easy to check the output of the first two layer by direct calculation. Concretely, we have

and

Next, we are going to prove the following conclusion for l = 2 , . . . , L -1 by induction

First, by (107), it holds for l = 2 . Next we assume (108) holds for the case of l -1 and verify it holds also for the case of l .

By the definition of our CNN model, we have

which verifies (108) for j = 1 . The case of j = 2 can be verified in the same way.

So by now we have verified that the output of the L -1 layer is

Noting σ ( x ) -σ ( -x ) = x , we can recover ∑ d i =1 ( x 2 2 i -1 -x 2 2 i ) and ∑ d i =1 ( x 2 2 i +2 d -1 -x 2 2 i +2 d ) by using the feature stored in z ( L -1) .

Recall that in our last layer, our activation function is σ L ( x ) = σ 2 ( x ) , and we have σ 2 ( x )+ σ 2 ( -x ) = x 2 . Noting that 4 αβ = ( α + β ) 2 -( α -β ) 2 and viewing ∑ d i =1 ( x 2 2 i -1 -x 2 2 i ) , ∑ d i =1 ( x 2 2 i +2 d -1 -x 2 2 i +2 d ) as α and β , respectively, a simple calculation would lead to h cnn θ ∗ ( x ) = h ∗ ( x ) .

Note that for all l ∈ [ L ] , b ( l ) = 0 , C l ⩽ 4 . Then, it is obvious that there exists an absolute constant C > 0 such that ∥ ∥ W ( l ) ∥ ∥ F ⩽ C for any l ∈ [ L ] and ∥ W o ∥ 2 ⩽ C . Thus, we have ∥ θ ∗ ∥ P = ∥ W o ∥ 2 + ∑ L l =1 ∥ ∥ W ( l ) ∥ ∥ F ≲ L ≲ log d .

Step II: Estimating the sample complexity.

Recall the input distribution P = N (0 , I 4 d ) . Let H CNN J = { h θ : ∥ θ ∥ ⩽ J } , where we recall ∥·∥ is defined in Appendix C. By Proposition B.14, the covering number satisfies

where ˆ γ n ( J ) = ˆ M n J (1 + J ) L , ˆ M n = √ 1 n ∑ n i =1 ( ¯ Q σ ( x i ) ) 2 ( ∥ x i ∥ 2 +1) 2 and N cnn = O (log d ) . We will use the fact that L = log 2 d +2 .

First, we turn to the estimate of ¯ Q σ ( x i ) . To this end, we give a simple lemma to estimate the local Lipschitz constant of our activation functions.

Lemma G.8. If for | x | , | y | ⩽ M , we have | β ( x ) -β ( y ) | ⩽ K | x -y | , then we will have

for any ∥ x ∥ 2 , ∥ y ∥ 2 ⩽ M .

Our σ 1 ( · ) = σ L ( · ) = σ 2 ( · ) activation functions are 2 K Lipschitz continuous when restricted in [ -K,K ] . So for Q 1 ( x i ) , since ∥T 1 ( x i ) ∥ 2 ⩽ ∥T 1 ∥ ( ∥ x i ∥ 2 + 1) ⩽ J ( ∥ x i ∥ 2 + 1) , we have Q 1 ( x i ) ⩽ 2 J ( ∥ x i ∥ 2 +1) . For Q L ( x i ) , we similarly have

so Q L ( x i ) ⩽ 2(1+ J ) L +1 ( ∥ x i ∥ 2 +1) 2 is straightforward. Thus, ¯ Q σ ( x i ) ⩽ 2 L +2 (1+ J ) L +2 ( ∥ x i ∥ 2 + 1) 3 . Therefore, we will have

where the third step follows from the Jensen's inequality and the fact that z ↦→ √ z is concave; the last step uses the higher-order moment bound of sub-exponential random variables (Vershynin, 2018, Proposition 2.7.1).

Recalling we have α H = 1 from equation (53) and applying Proposition B.14, we have w.p. at least 1 -2 δ

where is enough.

H LCNs vs. FCNs: Proofs of Section 5.3

H.1 Proof of Theorem 5.7

Under the assumptions in Theorem 5.7, Lemma 5.6 implies that A fcn T is G ort (4 d ) -equivalent. Then, by Lemma 5.1, we have

We next use the Fano's method (Proposition B.20) to bound the right hand side, by which the key to obtaining a tight bound is to construct a packing of { ¯ h ∗ } ◦ G ort (4 d ) as large as possible. Specifically, we will prove the following lemma.

Lemma H.1. There exist absolute positive constants C, c 5 , c 6 > 0 such that if d, A 0 ⩾ C , there exists h 1 , h 2 , . . . , h M ∈ { ¯ h ∗ } ◦ O (4 d ) satisfying that

̸

Assuming Lemma H.1 is right for now and applying the above packing to Fano's inequality (Proposition B.20), we have

Taking n such that the right hand side to satisfy

gives n = Ω( σ 2 d 2 ) . Thus, we can conclude that ¯ C ( { ¯ h ∗ } ◦ G ort (4 d ) , ϵ 0 ) = Ω( σ 2 d 2 ) . Otherwise, the minimax error becomes larger than ϵ 0 .

Taking A = A 0 , B = 2 A + σ log n, λ = 1 / n and applying Lemma B.13, we have

where

Simplify the inequality, we have

This implies that, for a target accuracy ϵ and failure probability δ ,

H.1.1 Proof of Lemma H.1

We now turn to construct a proper packing for { ¯ h ∗ } ◦ G ort (4 d ) to satisfy the condition in Lemma H.1. To simplify the statement, define q : R 2 d ↦→ R and g U : R 2 d ↦→ R for U ∈ R d × d by

Lemma H.2. (A restatement of (Li et al., 2020, Lemma D.2)) For any U ∈ G ort ( d ) , there exist τ U ∈ G ort (2 d ) such that

This lemma implies that for any U ∈ G ort ( d ) ,

where we slightly abuse the notation: using f ( x ) to denote the function f ( · ) .

Let

Note that equation (113) implies

Next, the following lemma shows that when the target truncation threshold A 0 is large enough, finding a packing of ( U A 0 , ∥ · ∥ L 2 ( P ) ) , thus a packing of ( { ¯ h ∗ } ◦ G ort (4 d ) , ∥·∥ L 2 ( P ) ) , can be reduced to pack the equivariance group ( G ort ( d ) , ∥ · ∥ F / √ d ) .

Lemma H.3. There exist absolute constants C, c 3 , c 4 > 0 such that when d, A 0 ⩾ C , we have for any U, U ′ ∈ G ort ( d ) that

Proof idea. The proof of this lemma is technically lengthy and deferred to Appendix I.2. Note that when A 0 = + ∞ , i.e., there is no truncation in the target function, a simple calculation leads to

When A 0 is finite but large enough, by concentration inequalities, we can show that the influence of truncation is negligible, although the proof is lengthy.

Lemma H.4. (Li et al., 2020, Lemma D.4.) For any positive integer d ⩾ 2 and any positive ϵ ⩽ c 2 / 2 , we have

where c 1 and c 2 are two positive absolute constants.

The above lemma provides a lower bound of the packing number of ( G ort ( d ) , ∥ · ∥ F / √ d ) . Intuitively speaking, this lemma is true due to the fact that G ort ( d ) can be viewed as a d ( d -1) 2 dimensional compact manifold from a Lie group perspective.

Proof of Lemma H.1 . First, for any U, U ′ ∈ G ort ( d ) , we have

̸

By Lemma H.4 and taking ϵ 0 = c 2 / 2 , we there exist U 1 , . . . , U M ∈ G ort ( d ) with M ⩾ 2 d ( d -1) 2 such that ∥ U i -U j ∥ F ⩾ c 2 √ d 2 c 1 for any i = j . Thus, by Lemma H.3, we have for any i = j that

̸

Combining (115) and (116), we can conclude that { π A 0 ◦ f U j } j m =1 are a packing that satisfy the condition in Lemma H.1. Thus, we complete the proof.

H.2 Proof of Theorem 5.5

Let σ ( z ) := ReLU( z ) in this subsection for notation simplicity. We first recall the architecture of the LCN model: L = log 2 (4 d ) , channel number C 1 = C L = 4 , C l = 2 for l = 2 , . . . , L -1 , and activation functions σ 1 ( z ) = σ L ( z ) = σ 2 ( z ) , σ l ( z ) = σ ( z ) for l = 2 , . . . , L -1 .

Step I: Representing the target using LCNs .

Lemma H.5. There is a parametrization θ ∗ such that h lcn θ ∗ = h ∗ and ∥ θ ∗ ∥ P ≲ d .

Proof. With the same depth, activation functions, and channel numbers, a LCN can simulate its CNN counterpart. By the proof of Lemma G.7, the exact representation is obvious. The parameter norm bound is due to

Step II: Estimating the sample complexity.

Recall the input distribution P = N (0 , I 4 d ) . Let H LCN J = { h θ : ∥ θ ∥ ⩽ J } , where we recall ∥·∥ is defined in Appendix C. By Proposition B.14, the covering number satisfies

where ˆ γ n ( J ) = ˆ M n J (1 + J ) L , ˆ M n = √ 1 n ∑ n i =1 ( ¯ Q σ ( x i ) ) 2 ( ∥ x i ∥ 2 +1) 2 and N lcn = O ( d ) . Here we use the fact that L = log 2 d + 2 . Following the same argument as in Appendix G.2 we have ¯ Q σ ( x i ) ⩽ 2 L +2 (1 + J ) L +2 ( ∥ x i ∥ 2 +1) 3 . Therefore, we will have

where ( a ) uses the Jensen's inequality and ( b ) follows from the higher-order moment bound of sub-exponential random variables (Vershynin, 2018, Proposition 2.7.1).

Recalling we have α H = 1 from equation (61) and applying Proposition B.14, we have with probability at least 1 -2 δ

where

Taking A = A 0 , B = 2 A + σ √ log n, λ = 1 / √ n and applying Lemma B.13, we have

where

Simplify the inequality, we have

This implies that, for a target accuracy ϵ and failure probability δ ,

is enough.

I Auxiliary Lemmas for Appendix G and H

Recall our input distribution P = N (0 , I 4 d ) . Recall that target function ¯ h ∗ = π A 0 ◦ h ∗ where

In the following, we show that the truncation operator π A 0 on h ∗ ◦ U for any U ∈ G ort (4 d ) will not matter a lot when A 0 is moderately large.

Lemma I.1. Let X ∼ P . For any U ∈ G ort (4 d ) , there exists one absolute constant c 1 > 0 such that for any δ ∈ (0 , 1) , if A 0 ⩾ max ( log(4 /δ ) c 1 , log 2 (4 /δ ) c 2 1 d ) , it holds that

Proof. Without loss of generality, we can set U = I 4 d since P is invariant under G ort (4 d ) . For simplicity, let Y i = X 2 2 i -1 -X 2 2 i for i ∈ [2 d ] in this proof. It is easy to check that Y i are i.i.d. random variables with ∥ Y i ∥ ψ 1 ≲ 1 . Then, using the Bernstein's inequality (Lemma B.3) gives

where c 1 > 0 is one absolute constant. Setting the failure probability to be smaller than δ/ 2 : 2 e -c 1 min( t 2 , √ dt ) ⩽ δ/ 2 gives t ⩾ max (√ log(4 /δ ) c 1 , log(4 /δ ) c 1 √ d ) =: C δ . This implies that it holds w.p. at least 1 -δ/ 2 that

Similarly, it holds w.p. at least 1 -δ/ 2 that

Combining them leads to

I.1 Proof of Lemma G.2

For any τ = diag ( U 1 , U 2 , . . . , U 2 d ) , τ ′ = diag ( U 1 , U ′ 2 , . . . , U ′ 2 d ) ∈ G semiloc , let

̸

be the set of indices where the local permutations are different for τ and τ ′ . Let s = |I| . By the definition of G semiloc , we must have j ⩽ d for any j ∈ I . Thus, we have

where o j ∈ { +1 , -1 } for j ∈ I . Because of the symmetry of P , we can assume o j = 1 for all j ∈ I without loss of generality.

Upper bound. First, recall that X ∼ N (0 , I 4 d ) , we have

where Y, Z ∼ N (0 , 1) are independent random variables.

Lower Bound. Denote the events

for the lower bound, we have

where the last inequality follows from the Cauchy-Schwarz inequality: ( E Y Z ) 2 ⩽ E Y 2 E Z 2 . For the first term on the RHS of (121), we have from (120) that

For the second term on the RHS of (121), we have

We first compute the first term by expanding it.

where we use the fact: E [ ( X 2 2 j -1 -X 2 2 j ) 2 ( X 2 2 i -1 -X 2 2 i ) 2 ] = O (1) for all i, j . By the same argument, we have

That leads to the following estimate: there exists an absolute constant c 3 > 0 such that

By Lemma I.1, there is an absolute constant C such that when A 0 ⩾ C , we have

Plugging (122), (125), and (126) into (121) gives

Combining the upper bound and the lower bound, we complete the proof.

I.2 Proof of Lemma H.3

For any U ∈ O ( d ) , let h U = π A 0 ◦ f U with

Thus, we have h U ∈ { ¯ h ∗ } ◦ G ort (4 d ) due to Lemma H.2.

Upper Bound. We have

where the last step follows from the independence between X 1:2 d and X 2 d +1:4 d . Note that by equation (120),

And, for the first term, we have

Plugging (129) and (130) into (128) gives

Lower Bound. Define events

Then, we have

where the last step uses the Cauchy-Schwartz inequality: ( E [ Y Z ]) 2 ⩽ E Y 2 E Z 2 . We next bound the two terms of the right hand side separately.

For the second term, we observe that

where the last step follows from (124). Note that conditioned on X d +1:2 d ,

Combining with the fact that EY 4 = 3 σ 2 for Y ∼ N (0 , σ 2 ) , we have

Plugging (133) into (132) gives

for some absolute constant c 4 > 0 . By Lemma I.1, there is an absolute constant C such that when A 0 ⩾ C , we have

Continuing the equation (131), we have

where in the third step we use the result from the above upper bound:

Combining the upper bound and the lower bound, we complete the proof.

(136)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper theoretically analyses the sample complexity of CNNs as compared to fully connected networks. The authors find that while fully connected networks require quadratic samples with input dimension, the local connections of CNNs reduce the sample complexity to linear, while weight sharing further reduces sample complexity to logarithmic.\n\nSTRENGTHS:\n**Originality**\n\nAs the authors point out, separating the impact of local connections vs weight sharing in CNNs is not a new idea; nevertheless, as far as I am aware, no previous works have shown theoretical sample complexity results for CNNs vs LCNs vs FCNs.\n\n**Quality**\n\nTechnically speaking, the paper appears solid. The results and proof look correct as far as I can tell. The appendix is quite comprehensive.\n\n**Clarity**\n\nThe paper is generally well-written. The notation is appropriately introduced and concepts are adequately explained.\n\n**Significance**\n\nI think this paper may be significant to the specific subfield that studies the sample complexity of specific model architectures.\n\nWEAKNESSES:\nOverall, I'm unfortunately not too convinced about the significance of this work given that there is already work analyzing at least empirically the sample complexity benefit of weight sharing vs locality (Xiao & Pennington 2022 as the authors point out, but also Poggio et al. 2016). Certainly, it seems that the theoretical result proved by the authors is new. On the other hand, it would benefit the paper to validate these theoretical results with experimental evidence, or at least point to prior experiments that line up with the authors' theory.\n\n\nMinor comment: Using similar-looking notation for orthogonal groups and big O is not ideal.\n\nQUESTIONS:\nIs this the first work to provide theoretical sample complexity bounds for CNNs vs LCNs vs FCNs?\n\nAre the asymptotic sample complexities found by the authors practically relevant? Can the authors empirically demonstrate these trends?\n\nLIMITATIONS:\nLimitations are adequately addressed; the authors clearly state the assumptions of their theory. No negative potential societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study inductive biases in deep CNNs. They analyse approximation abilities of these networks and compare the learning performances of CNNs for regression with those of locally-connected networks (LCNs) and fully-connected networks. They show that CNNs of depth O(\\log d) is sufficient for achieving universality. They also present some analysis for CNNs in approximating functions with sparse variable structures.\n\nSTRENGTHS:\nThe result that CNNs of depth O(\\log d) with exponentially increasing channels is sufficient for achieving universality is very interesting. It shows the power of CNNs and channels in approximating functions. The example presented in Proposition 3.2 about the lower bound for the universality is nice. The comparison between CNNs and LCNs on learning with bounds O(\\log^2 d) and \\Omega (d) is simulating. Though the methods used by the authors are not new, the obtained results are valuable and insightful and can be used to explain the efficiency of deep learning algorithms induced by structures of deep neural networks. \n\nAnalysing structured deep neural networks in approximating and learning functions with sparse variable structures is an important topic in deep learning theory.\n\nWEAKNESSES:\nThe topic of analysing structured deep neural networks in approximating and learning functions with sparse variable structures has been studied in the community of deep learning theory. the related literature should be mentioned in the paper. The work of Mhaskar and Poggio (Analysis and Applications 2016) with fully connected neural networks is based on a known sparse variable structure while the recent work of Mao, Shi, and Zhou (Analysis and Applications 2023) with CNNs is for unknown sparse variable structures. Another paper of Chui, Lin, Zhang, and Zhou (IEEE NNLS 2020) is about learning of spatially sparse functions.\n\nQUESTIONS:\nThe number of channels for CNNs is often large, but it does not increase exponentially. Can the authors derive a universality result with polynomially increasing channels?\n\nLIMITATIONS:\nThe CNNs with exponentially increasing channels might be restrictive.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a fundamental analysis of the superior performance of CNNs, specifically focusing on their downsampling and multichanneling capabilities. Moreover, it establishes evidence with theoretical guarantees that CNNs exhibit efficient learning of sparse functions compared to FCNs, attributing this efficiency to the weight sharing and locality properties inherent in CNNs.\n\nSTRENGTHS:\nThis paper presents a comprehensive and rigorous theoretical analysis of the intrinsic inductive biases linked to multichanneling, downsampling, weight sharing, and locality within CNNs. It introduces a new theoretical framework that characterizes the crucial role of multichanneling and downsampling in empowering deep CNNs to efficiently capture long-range dependencies within the target function, yielding valuable insights into their operational mechanisms. Furthermore, leveraging these theoretical insights, the paper substantiates the superior performance of CNNs compared to FCNs, providing further support for their efficacy.\n\nWEAKNESSES:\n1. This paper primarily concentrates on exploring the expressive power of Convolutional Neural Networks (CNNs) and argues that a CNN with a depth on the order of $\\log d$ possesses the capacity to learn spatial information within the data. However, it lacks the convergence and generalization analysis, i.e., whether CNNs with a layer depth in the order of $\\log d$ can effectively learn spatial information using SGD-based algorithms remains uncertain.\n\n2. This paper lacks empirical experiments, particularly those using real data, to provide concrete evidence in support of its theoretical findings.\n\n3. Recent observations suggest that Transformers are more efficient in learning spatial information compared to CNNs, and CNNs are not as widely utilized in current practices. In light of these trends, it would be valuable for the paper to offer insights or considerations pertaining to the Transformer model, further exploring its strengths and potential advantages in learning spatial information.\n\nQUESTIONS:\n1. In lines 158-159, the authors assert that \"universality can be established by simply showing that ... can simulate any two-layer ReLU networks.\" It raises the question of whether the authors assume that a two-layer ReLU network can approximate any continuous function.\n\n2. Could you please provide more context or specify the notations in Assumption 1 you are referring to? This will help in understanding Assumption 1 and the difficulties in following the notations.\n\n3.  It would be better to provide a summary of the technical novelty of this paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the inductive bias of deep convolutional neural networks (CNNs) in three distinct ways. With $d$ the input dimensions, it is shown that (i) depth $O(\\log{d})$ is sufficient to achieve universality, i.e. the capacity to approximate any function of $d$ variables, when there is downsampling between layers; (ii) all sparse functions can be learned with $O(\\log^2{d})$ samples, close to the optimal $O(\\log{d})$; (iii) both weight sharing and locality of the connection are crucial for achieving the nearly-optimal sample complexity.\n\nSTRENGTHS:\nOur theoretical understanding of deep CNNs is still very limited and this paper has the potential to provide a solid contribution to the topic. The explanation of how multichannelling and downsampling conspire to achieve universality with $O(\\log{d})$ layers is nice and intuitive. The result of Theorem 4.6 on the sample complexity of learning sparse functions is novel and interesting. The separation results start from a known idea  (using the symmetry of an algorithm to unveil its limitations) but give significantly new insights into the separate roles of weight sharing and locality.\n\nWEAKNESSES:\nOverall, I believe this paper would be a valid contribution to the community but there are some issues that I would like to see addressed before recommending publication. \n\nI like section 3 and I believe it is a valuable addition to the literature, but I would downplay the claim that all previous results required a depth of $\\log{d}$, as currently written in the abstract. The idea that sufficiently deep CNNs achieve the same expressivity as fully-connected networks (hence they are universal) has been there for a while and the difference between $d$ and $\\log{d}$ is solely due to the amount of stride included in the architecture. The case considered in this paper, where the stride equals the filter size, is commonly referred to as *nonoverlapping patches*---the authors should say it explicitly at least in section 2.1---and it has been studied in several other works, including some cited in this paper such as  Cagnetta et al. (2022) and some that are not cited but should be such as Poggio et al. (2016), *'Why and When Can Deep – but Not Shallow – Networks Avoid the Curse of Dimensionality: a Review'*. Both these works imply that deep CNNs with nonoverlapping patches and depth $\\log{d}$, so that the post-activations of the last hidden layer look at the whole input, are universal.\n\nThere is also something about theorem 4.6 that I think requires further explanation. As far as I understand, theorem 4.6 uses lemma 4.4 together with the generalisation properties of one-hidden layer neural networks (theorem 4 from E et al. (2022)). If this is true, then I do not understand why one needs a deep CNN to achieve sample complexity $O(\\log^2{d})$: the result of lemma 4.4, i.e. that there exist a set of parameters such that the $L$-th layer outputs the right-hand side of Eq. (4), is also applicable to the post-activations of a  one-hidden-layer fully-connected network. I should remark that my concern might be due to a misunderstanding of the proof idea, but if that is the case I will be happy to raise mi mark after rebuttal.\n\nFinally, the symmetry-based lowe bounds like those of theorems 5.4 and 5.7 can usually be explained very well at an intuitive level. I think not having such intuitive explanations is a weakness of this manuscript.\n\nQUESTIONS:\n1. Regarding the difference between CNNs and LCNs, Bietti et al. (2021) provides a very simple explanation of why there should be a factor $d$ of difference between the sample complexity of a LCN and that of a CNN. Is their argument valid also in the setting of section 5?\n\n2. Is there a simple explanation based on symmetry as to why the sample complexity of FCNs presents another factor of $d$ with respect to that of LCNs?\n\nThe following are not questions but suggestions:\n\n3. There is a typo in line 12 of the abstract (separete instead of separate).\n\n4. In the same line of the abstract I would use a different word than `introduce', since locally-connected networks have been introduced before.\n\n5. Could you please quickly recall the meaning of the various big-O notations in section 1.2?\n\n6. State explicitly that this work considers CNNs with nonoverlapping patches at all layers. e.g. in section 2.1.\n\n7. The related works section needs further references, for instance, the aforementioned Poggio et al. (2016), *'Why and When Can Deep – but Not Shallow – Networks Avoid the Curse of Dimensionality: a Review'* and other works of the same group. Schmidt-Hierber (2021), *'Nonparametric regression using deep neural networks with ReLU activation function'* is also relevant. Furthermore, Bietti (2021) and Favero et al. (2021) *'Locality defeats the curse of dimensionality in convolutional teacher-student scenarios'* studied the benefits of locality before Misiaciewicz and Mei (2022), and Xiao (2022) *'Eigenspace restructuring: a principle of space and frequency in neural networks'* also studied the inductive bias of deep CNNs in the kernel regime.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}