Title: Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Nonparametric Classification

Low Dimensional Manifolds

Overparameterized ResNets

Function Approximation.
Abstract: Convolutional residual neural networks (ConvResNets), though overparametersized, can achieve remarkable prediction performance in practice, which cannot be well explained by conventional wisdom. To bridge this gap, we study the performance of ConvResNeXts trained with weight decay, which cover ConvResNets as a special case, from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks. Specifically, we consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNeXts can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of dimensionality. Our findings partially justify the advantage of overparameterized ConvResNeXts over conventional machine learning models.

Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks

Abstract

Convolutional residual neural networks (ConvResNets), though overparametersized , can achieve remarkable prediction performance in practice, which cannot be well explained by conventional wisdom. To bridge this gap, we study the performance of ConvResNeXts trained with weight decay, which cover ConvResNets as a special case, from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks. Specifically, we consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNeXts can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of dimensionality. Our findings partially justify the advantage of overparameterized ConvResNeXts over conventional machine learning models.

1 Introduction

Deep learning has achieved significant success in various real-world applications, such as computer vision [14, 23, 26], natural language processing [2, 15, 42], and robotics [16]. One notable example of this is in the field of image classification, where the winner of the 2017 ImageNet challenge achieved a top-5 error rate of just 2.25% [19] using Convolutational Residual Network (ConvResNets) on a training dataset of 1 million labeled high-resolution images in 1000 categories.

Researchers have attributed the remarkable performance of deep learning to its great flexibility in modeling complex functions, which has motivated many works on investigating the representation power of deep neural networks. For instance, early work such as Barron [3], Cybenko [7], Kohler and Krzy˙ zak [22] initialized this line of research for simple feedforward neural networks (FNNs) [19, 20, 38, 37]. More recently, Suzuki [32], Yarotsky [41] gave more precise bounds on the model sizes in terms of the approximation error, and Oono and Suzuki [30] further established a bound for more advanced architectures - ConvResNets. Based on these function approximation theories,

one can further establish generalization bounds of deep neural networks with finite samples. Taking Oono and Suzuki [30] as an example again, they showed that ConvResNets with ˜ O ( n D/ (2 α + D ) ) parameters can achieve a minimax optimal convergence rate ˜ O ( n -2 α/ (2 α + D ) ) while approximating a C α nonparametric regression function with n samples. Unfortunately, these theoretical results cannot well explain the empirical successes of deep learning well, as they require the model size to be no larger than ˜ O ( n ) (the generalization bounds become vacuous otherwise). However, in real applications, practical deep learning models are often overparmameterized, that is the model size can greatly exceeds the sample size.

1.1 Main Results

Overparameterization of neural networks has been considered as one of the most fundamental research problems in deep learning theories, where parameters can significantly exceed training samples. There has been substantial empirical evidence showing that overparameterization can help fit the training data, ease the challenging nonconvex optimization, and gain robustness. However, existing literature on deep learning theories under such an overparameterized regime is very limited. To the best of our knowledge, we are only aware of Zhang and Wang [43], which attempts to analyze overparameterized neural networks trained with weight decay. However, their work still suffers from two major restrictions: (1) They consider parallel FNN, which is rarely used in practice. Whether similar results hold for more practical architectures remains unclear; (2) Their generalization bound from the curse of dimensionality, where the sample size is require to scale exponentially with the input dimension.

To address (1), we propose to develop a new theory for nonparametric classification using overparameterized ConvResNeXts trained with weight decay [40]. The ConvResNeXt generalizes ConvResNets and includes them as a special case [5, 18, 33, 44]. Compared with FNNs, ConvResNeXts exhibit three features: (i) Instead of using dense weight matrices, they use convolutional filters, which can naturally investigate the underlying structures of the input data such as images and acoustic signals; (ii) They are equipped with skip-layer connections, which divides the entire network into blocks. The skip-layer connection can effectively address the vanishing gradient issue and therefore allow the networks to be significantly deeper; (iii) They are equipped with parallel architectures, which enable multiple 'paths' within each block of the network, and allows the network to learn a more diverse set of features. Figure 1b illustrates the structure of ConvResNeXts (detailed introductions of ConvResNeXts is deferred to Section 2.3). This architecture introduces a significantly more complex nested function form, presenting us with the challenge of addressing novel issues in bounding the metric entropy of the function class.

To address (2), our proposed theory considers the optimal classifier is supported on a d -dimensional smooth manifold M isometrically embedded in R D with d ≪ D . The low-dimensional manifold assumption is highly practical, since it aligns with the inherent nature of many real-world datasets. For example, images typically represent projections of 3-dimensional objects subject to various transformations like rotation, translation, and skeletal adjustments. Such a generating mechanism inherently involves a limited set of intrinsic parameters. More broadly, various forms of data, including visual, acoustic, and textual, often exhibit low dimensional structures due to rich local regularities, global symmetries, repetitive patterns, or redundant sampling. It is reasonable to model these data as samples residing in proximity to a low dimensional manifold.

Our theoretical results can be summarized as follows:

· We prove that when ConvResNeXts are overparameterized, i.e., the number of blocks is larger than the order of the sample size n , they can still achieve an asymptotic minimax rate for learning Besov functions when trained with weight decay. That is, given that the target function belongs to the Besov space B α p,q ( M ) 2 , the risk of the estimator given by the ConvResNeXt class converges to the optimal

α/d risk at the rate ˜ O ( n -2 α/d +1 (1 -o (1)) ) with n samples. Notably, the statistical rate of convergence in our theory only depends on the intrinsic dimension d , which circumvents the curse of dimensionality in Zhang and Wang [43].

Moreover, our theory shows that one can scale the number of 'paths' M in each block with the depth N as roughly MN ≳ n 1 2 α/d +1 , which does not affect the convergence rate. This partially justifies the flexibility of the ConvResNeXt architecture when designing the bottlenecks, which simple structures like FNNs cannot achieve. Moreover, we can exchange the number of 'paths' M and depth N as long as their product remains the same. This further provides the architectural insight that we don't necessarily need parallel blocks when we have residual connections. To say it differently, we provide new insight into why 'residual connection' and "parallel blocks' in ResNeXts are useful in both approximation and generalization.

Another technical highlight of our paper is bounding the covering number of weight-decayed ConvResNeXts, which is essential for computing the critical radius of the local Gaussian complexity. Specifically, we adopted a more advanced method that leverages the Dudley's chaining of the metric entropy [4]. This technique provides a tighter bound than choosing a single radius of the covering number as in Zhang and Wang [43].

To the best of our knowledge, our work is the first to develop approximation and statistical theories for ConvResNeXts, as well as overparameterized ConvResNets.

1.2 Related Works

Our work is closely related to Liu et al. [25], which studies nonparametric classification under a similar setup - the optimal classifier belongs to the Besov space supported on a low dimensional manifold. Despite they develop similar theoretical results to ours, their analysis does not allow the model to be overparameterized, and therefore is not applicable to practical neural networks. Moreover, they investigate ConvResNets, which is a special case of ConvResNeXt in our work.

Our work is closely related to the reproducing kernel methods, which are also often used for nonparametric regression. However, existing literature has shown that the reproducing kernel methods lack the adaptivity to handle the heterogeneous smoothness in estimating Besov space functions, and only achieve suboptimal rate of convergence in statistical estimation [9, 32].

Our work is closely related neural tangent kernel theories [21, 1], which study overparameterized neural networks. Specifically, under certain regularity conditions, they establish the equivalence between overparameterized neural networks and reproducing kernel methods, and therefore the generalization bounds of overparameterized networks can be derived based on the associated reproducing kernel Hilbert space. Note that neural tangent kernel theories can be viewed as special cases of the theories for general reproducing kernel methods. Therefore, they also lack the adaptivity to be successful in the Besov space thus do not capture the properties of overparameterized neural networks.

2 Preliminaries

In this section, we introduce some concepts on manifolds. Details can be found in [35] and [24]. Then we provide a detailed definition of the Besov space on smooth manifolds and the ConvResNeXt architecture.

2.1 Smooth manifold

Firstly, we briefly introduce manifolds, the partition of unity and reach. Let M be a d -dimensional Riemannian manifold isometrically embedded in R D with d much smaller than D .

Definition 2.1 (Chart) . A chart on M is a pair ( U, ϕ ) such that U ⊂ M is open and ϕ : U ↦→ R d , where ϕ is a homeomorphism (i.e., bijective, ϕ and ϕ -1 are both continuous).

In a chart ( U, ϕ ) , U is called a coordinate neighborhood, and ϕ is a coordinate system on U . Essentially, a chart is a local coordinate system on M . A collection of charts that covers M is called an atlas of M .

Definition 2.2 ( C k Atlas) . A C k atlas for M is a collection of charts { ( U i , ϕ i ) } i ∈A which satisfies ⋃ i ∈A U i = M , and are pairwise C k compatible:

are both C k for any i, β ∈ A . An atlas is called finite if it contains finitely many charts.

Definition 2.3 (Smooth Manifold) . A smooth manifold is a manifold M together with a C ∞ atlas.

Classical examples of smooth manifolds are the Euclidean space, the torus, and the unit sphere. Furthermore, we define C s functions on a smooth manifold M as follows:

Definition 2.4 ( C s functions on M ) . Let M be a smooth manifold and f : M→ R be a function on M . Afunction f : M→ R is C s if for any chart ( U, ϕ ) on M , the composition f ◦ ϕ -1 : ϕ ( U ) → R is a continuously differentiable up to order s .

We next define the C ∞ partition of unity, which is an important tool for studying functions on manifolds.

Definition 2.5 (Partition of Unity, Definition 13.4 in [35]) . A C ∞ partition of unity on a manifold M is a collection of C ∞ functions { ρ i } i ∈A with ρ i : M→ [0 , 1] such that for any x ∈ M , there is a neighbourhood of x where only a finite number of the functions in { ρ i } i ∈A are nonzero, and ∑ i ∈A ρ i ( x ) = 1 .

An open cover of a manifold M is called locally finite if every x ∈ M has a neighborhood that intersects with a finite number of sets in the cover. The following proposition shows that a C ∞ partition of unity for a smooth manifold always exists.

Proposition 2.6 (Existence of a C ∞ partition of unity, Theorem 13.7 in [35]) . Let { U i } i ∈A be a locally finite cover of a smooth manifold M . Then there is a C ∞ partition of unity { ρ i } ∞ i =1 where every ρ i has a compact support such that supp( ρ i ) ⊂ U i .

Let { ( U i , ϕ i ) } i ∈A be a C ∞ atlas of M . Proposition 2.6 guarantees the existence of a partition of unity { ρ i } i ∈A such that ρ i is supported on U i . To characterize the curvature of a manifold, we adopt the geometric concept: reach.

Definition 2.7 (Reach [12, 29]) . Denote

̸

as the set of points with at least two nearest neighbors on M . The closure of G is called the medial axis of M . Then the reach of M is defined as

Reach has a simple geometrical interpretation: for every point x ∈ M , the osculating circle's radius is at least τ . A large reach for M indicates that the manifold changes slowly.

2.2 Besov functions on a smooth manifold

We next define the Besov function space on the smooth manifold M , which generalizes more elementary function spaces such as the Sobolev and Hölder spaces. Roughly speaking, functions in the Besov space are only required to have weak derivatives with bounded total variation. Notably, this includes functions with spatially heterogeneous smoothness, which requires more locally adaptive methods to achieve optimal estimation errors [10]. Examples of Besov class functions include piecewise linear functions and piecewise quadratic functions that are smoother in some regions and more wiggly in other regions; see e.g., Figure 2 and Figure 4 of Mammen and van de Geer [27].

To define Besov functions rigorously, we first introduce the modulus of smoothness.

Definition 2.8 (Modulus of Smoothness [8, 32]) . Let Ω ⊂ R D . For a function f : R D → R be in L p (Ω) for p > 0 , the r -th modulus of smoothness of f is defined by

Definition 2.9 (Besov Space B α p,q (Ω) ) . For 0 < p, q ≤ ∞ , α > 0 , r = ⌊ α ⌋ +1 , define the seminorm | · | B α p,q as

The norm of the Besov space B s p,q (Ω) is defined as ∥ f ∥ B α p,q (Ω) := ∥ f ∥ L p (Ω) + | f | B α p,q (Ω) . Then the Besov space is defined as B α p,q (Ω) = { f ∈ L p (Ω) |∥ f ∥ B α p,q < ∞} .

Moreover, we show that functions in the Besov space can be decomposed using B-spline basis functions in the following proposition.

Proposition 2.10 (Decomposition of Besov functions) . Any function f in the Besov space B α p,q , α > d/p can be decomposed using B-spline of order m,m > α : for any x ∈ R d , we have

where J ( k ) := { 2 -k s : s ∈ [ -m, 2 k + m ] d ⊂ Z d } , M m,k, s ( x ) := M m (2 k ( x -s )) , and M k ( x ) = ∏ d i =1 M k ( x i ) is the cardinal B-spline basis function which can be expressed as a polynomial:

We next define B α p,q functions on M .

Definition 2.11 ( B α p,q Functions on M [13, 34]) . Let M be a compact smooth manifold of dimension d . Let { ( U i , ϕ i ) } C M i =1 be a finite atlas on M and { ρ i } C M i =1 be a partition of unity on M such that supp( ρ i ) ⊂ U i . A function f : M→ R is in B α p,q ( M ) if

Since ρ i is supported on U i , the function ( fρ i ) ◦ ϕ -1 i is supported on ϕ ( U i ) . Wecan extend ( fρ i ) ◦ ϕ -1 i from ϕ ( U i ) to R d by setting the function to be 0 on R d \ ϕ ( U i ) . The extended function lies in the Besov space B s p,q ( R d ) [34, Chapter 7].

2.3 Architecture of ConvResNeXt

We introduce the architecture of ConvResNeXts. ConvResNeXts have three main features: convolution kernel, residual connections, and parallel architecture.

Consider one-sided stride-one convolution in our network. Let W = {W j,k,l } ∈ R w ′ × K × w be a convolution kernel with output channel size w ′ , kernel size K and input channel size w . For z ∈ R D × w , the convolution of W with z gives y ∈ R D × w ′ such that

where 1 ≤ i ≤ D, 1 ≤ j ≤ w ′ and we set z i + k -1 ,l = 0 for i + k -1 > D , as demonstrated in Figure 1a.

The building blocks of ConvResNeXts are residual blocks. Given an input x , each residual block computes x + F ( x ) , where F is a subnetwork called bottleneck, consisting of convolutional layers. In ConvResNeXts, a parallel architecture is introduced to each building block, which enables multiple 'paths' in each block. In this paper, we study the ConvResNeXts with rectified linear unit (ReLU) activation function, i.e., σ ( z ) = max { z, 0 } . Wenext provide the detailed definition of ConvResNeXts as follows:

a

x

+

+

+

f

(

x

)

id

id

id

f

1

,

1

. . .

f

1

,M

f

N,

1

. . .

f

N,M

. . .

b

W

j-th column

K

W

2

D

W*2

D

Wj,

0

W

Definition 2.12. Let the neural network comprise N residual blocks, each residual block has a parallel architecture with M building blocks, and each building block contains L layers. The number of channels is w , and the convolution kernel size is K . Given an input x ∈ R D , a ConvResNeXt with ReLU activation function can be represented as

where id is the identity operator, P : R D → R D × w 0 is the padding operator satisfying P ( x ) = [ x , 0 . . . 0 ] ∈ R D × w , { W ( n,m ) l } L l =1 is a collection of convolution kernels for n = 1 , . . . , N, m = 1 , . . . , M , W out ∈ R w L denotes the linear operator for the last layer, and ⋆ is the convolution operation defined in (4).

The structure of ConvResNeXts is shown in Figure 1b. When M = 1 , the ConvResNeXt defined in (5) reduces to a ConvResNet. For notational simplicity, we omit biases in the neural network structure by extending the input dimension and padding the input with a scalar 1 (See Proposition F.4 for more details). The channel with 0's is used to accumulate the output.

3 Theory

In this section, we study a binary classification problem on M⊆ [ -1 , 1] D . Specifically, we are given i.i.d. samples { x i , y i } n i =1 ∼ D where x i ∈ M and y i ∈ { 0 , 1 } is the label. The label y ∈ { 0 , 1 } follows the Bernoulli-type distribution

for some f ∗ : M→ R belonging to the Besov space. More specifically, we make the following assumption on f ∗ .

Assumption 3.1. Let 0 < p, q ≤ ∞ , d/p < α < ∞ . Assume f ∗ ∈ B α p,q ( M ) and ∥ f ∗ ∥ B α p,q ( M ) ≤ C F for some constant C F > 0 .

To learn f ∗ , we minimize the empirical logistic risk over the training data:

where F Conv is some neural network class specified later. For notational simplicity, we denote the empirical logistic risk function in (6) as L n ( f ) , and denote the population logistic risk as

We next specify the class of ConvResNeXts for learning f ∗ :

F Conv ( N,M,L,K,w,B res , B out ) = { f | f is in the form of (5) with N residual blocks. Every residual block has M building blocks with each building block containing L layers. Each layer has kernel size bounded by K, number of channels bounded by w.

Note that the hyperparameters of F Conv will be specified in our theoretical analysis later.

As can be seen, F Conv contains the Frobenius norm constraints of the weights. For the sake of computational convenience in practice, such constraints can be replaced with weight decay regularization the residual blocks and the last fully connected layer separately. More specifically, we can use the following alternative formulation:

where λ 1 , λ 2 > 0 are properly chosen regularization parameters.

3.1 Approximation theory

In this section, we provide a universal approximation theory of ConvResNeXts for Besov functions on a smooth manifold:

Theorem 3.2. For any Besov function f 0 on a smooth manifold satisfying p, q ≥ 1 , α -d/p > 1 ,

for any P > 0 and any ConvResNeXt class F Conv ( N,M,L,K,w,B res , B out ) satisfying L = L ′ + L 0 -1 , L ′ ≥ 3 , where L 0 = ⌈ D K -1 ⌉ , and

MN ≥ C M P, w ≥ C 1 ( dm + D ) , B res ≤ C 2 L/K,B out ≤ C 3 C 2 F (( dm + D ) LK ) L ( C M P ) L -2 /p , there exists f ∈ F Conv ( N,M,L,K,w,B res , B out ) such that

where C 1 , C 2 , C 3 are universal constants and C 4 , C 5 , C 6 are constants that only depends on d and m , d is the intrinsic dimension of the manifold and m is an integer satisfying 0 < α < min( m,m -1 + 1 /p ) .

The approximation error of the network is bounded by the sum of two terms. The first term is a polynomial decay term that decreases with the size of the neural network and represents the trailing term of the B-spline approximation. The second term reflects the approximation error of neural networks to piecewise polynomials, decreasing exponentially with the number of layers. The proof is deferred to Section 4.1 and the appendix.

3.2 Estimation theory

Theorem 3.3. Suppose Assumption 3.1 holds. Set L = L ′ + L 0 -1 , L ′ ≥ 3 , where L 0 = ⌈ D K -1 ⌉ , and

Let ˆ f be the global minimizer given in (6) with the function class F = F Conv ( N,M,L,K,w,B res , B out ) . Then we have

where the logarithmic terms are omitted. C 1 is the constant defined in Theorem 3.2, C 7 , C 8 are constants that depend on C F , C M , d, m , K is the size of the convolution kernel.

We would like to make the following remarks about the results:

Strong adaptivity: By setting the width of the neural network to w = 2 C 1 D , the model can adapt to any Besov functions on any smooth manifold, provided that dm ≤ D . This remarkable flexibility can be achieved simply by tuning the regularization parameter. The cost of overestimating the width is a slight increase in the estimation error.

No curse of dimensionality: the above error rate only depends polynomially on the ambient dimension D and exponentially on the hidden dimension d . Since in real data, the hidden dimension d can be much smaller than the ambient dimension D , this result shows that neural networks can explore the low-dimension structure of data to overcome the curse of dimensionality.

Overparameterization is fine: the number of building blocks in a ConvResNeXt does not influence the estimation error as long as it is large enough. In other words, this matches the empirical observations that neural networks generalize well despite overparameterization.

Close to minimax rate: The lower bound of the 1-Lipschitz error for any estimator θ is

where ≳ notation hides a factor of constant. The proof can be found in Appendix E. Comparing to the minimax rate, we can see that as L →∞ , the above error rate converges to the minimax rate up to a constant term. In other words, overparameterized ConvResNeXt can achieve close to the minimax rate in estimating functions in Besov class. In comparison, all kernel ridge regression including any NTKs will have a suboptimal rate lower bounded by 2 α -d 2 α , which is suboptimal.

Deeper is better: with larger L , the error rate decays faster and gets closer to the minimax rate. This indicates that deeper model can achieve better performance than shallower models.

Tradeoff between width and depth: With a fixed budget in the number of parameters, the tradeoff between width and depth is crucial for achieving the best performance, and this often requires repeated, time-consuming experiments. On the other hand, our results suggests that such a tradeoff less important in a ResNeXt. The lower bound of error does not depend on the arrangements of the residual blocks M and N , as long as their product is large enough. This can partly explain the benefit of ResNeXt over other architecture.

By choosing L = O (log( n )) in Theorem 3.3, the second term in the error can be merged with the first term, and close to the minimax rate can be achieved:

Corollary 3.4. Given the conditions in Theorem 3.3, set the depth of each block is L = O (log( n )) and then the estimation error of the empirical risk minimizer ˆ f satisfies

where ˜ O ( · ) omits the logarithmic term.

The proof of Theorem 3.3 is deferred to Section 4.2 and Section D.2. The key technique is computing the critical radius of the local Gaussian complexity by bounding the covering number of weightdecayed ConvResNeXts. This technique provides a tighter bound than choosing a single radius of the covering number as in Suzuki [32], Zhang and Wang [43]. The covering number of an overparameterized ConvResNeXt with norm constraint (Lemma 4.1) is one of our key contributions.

4 Proof overview

4.1 Approximation error

We follow the method in Liu et al. [25] to construct a neural network that achieves the approximation error we claim. It is divided into the following steps:

Step 1: Decompose the target function into the sum of locally supported functions.

In this work, we adopt a similar approach to [25] and partition M using a finite number of open balls on R D . Specifically, we define B ( c i , r ) as the set of unit balls with center c i and radius r such that their union covers the manifold of interest, i.e., M⊆∪ C M i =1 B ( c i , r ) . This allows us to partition

the manifold into subregions U i = B ( c i , r ) ∩ M , and further decompose a smooth function on the manifold into the sum of locally supported smooth functions with linear projections. The existence of function decomposition is guaranteed by the existence of partition of unity stated in Proposition 2.6. See Section C.1 for the detail.

Step 2: Locally approximate the decomposed functions using cardinal B-spline basis functions. In the second step, we decompose the locally supported Besov functions achieved in the first step using B-spline basis functions. The existence of the decomposition was proven by D˜ ung [11], and was applied in a series of works [43, 32, 25]. The difference between our result and previous work is that we define a norm on the coefficients and bound this norm, instead of bounding the maximum value. The detail is deferred to Section C.2.

Step 3: Approximate the polynomial functions using neural networks. In this section, we follow the method in Zhang and Wang [43], Suzuki [32], Liu et al. [25] and show that neural networks can be used to approximate polynomial functions, including B-spline basis functions and the distance function. The key technique is to use a neural network to approximate square function and multiply function [3]. The detail is deferred to the appendix. Specifically, Lemma F.3 proves that a neural network with width w = O ( dm ) and depth L can approximate B-spline basis functions, and the error decreases exponentially with L ; Similarly, Proposition C.3 shows that a neural network with width w = O ( D ) can approximately calculate the distance between two points d 2 ( x ; c ) , with precision decreasing exponentially with the depth.

Step 4: Use a ConvResNeXt to Approximate the target function. Using the results above, the target function can be (approximately) decomposed as

We first demonstrate that a ReLU neural network taking two scalars a, b as the input, denoted as a ˜ × b , can approximate y × 1 ( x ∈ B r,i ) , where ˜ × satisfy that y ˜ × 1 = y for all y , and y ˜ × ˜ x = 0 if any of x or y is 0, and the soft indicator function ˜ 1 ( x ∈ B r,i ) satisfy ˜ 1 ( x ∈ B r,i ) = 1 when x ∈ B r,i , and ˜ 1 ( x ∈ B r,i ) = 0 when x / ∈ B r +∆ ,i . The detail is deferred to Section C.3.

Then, we show that it is possible to construct MN = C M P number of building blocks, such that each building block is a feedforward neural network with width C 1 ( md + D ) and depth L , where m is an integer satisfying 0 < α < min ( m,m -1 + 1 /p ) . The k -th building block (the position of the block does not matter) approximates a i,k j , s j M m,k j , s j ◦ ϕ i × 1 ( x ∈ B ( c i , r )) , where i = ceiling ( k/N ) , j = rem ( k, N ) . Each building block has where a sub-block with width D and depth L -1 approximates the chart selection, a sub-block with width md and depth L -1 approximates the B-spline function, and the last layer approximates the multiply function. The norm of this block is bounded by

Making use of the 1-homogeneous property of the ReLU function, by scaling all the weights in the neural network, these building blocks can be combined into a neural network with residual connections, that approximate the target function and satisfy our constraint on the norm of weights. See Section C.4 for the detail.

By applying Lemma C.6, which shows that any L -layer feedforward neural network can be reformulated as an L + L 0 -1 -layer convolution neural network, the neural network constructed above can be converted into a ConvResNeXt that satisfies the conditions in Theorem 3.2.

4.2 Estimation error

We first prove the covering number of an overparameterized ConvResNeXt with norm-constraint as in Lemma 4.1, then compute the critical radius of this function class using the covering number as in Corollary F.5. The critical radius can be used to bound the estimation error as in Theorem 14.20 in Wainwright [36]. The proof is deferred to Section D.2.

Lemma 4.1. Consider a neural network defined in Definition 2.12. Let the last layer of this neural network is a single linear layer with norm ∥ W out ∥ 2 F ≤ B out . Let the input of this neural network

satisfy ∥ x ∥ 2 ≤ 1 , ∀ x , and is concatenated with 1 before feeding into this neural network so that part of the weight plays the role of the bias. The covering number of this neural network is bounded by

where the logarithmic term is omitted.

The key idea of the proof is to split the building block into two types ('small blocks' and 'large blocks') depending on whether the total norm of the weights in the building block is smaller than ϵ . By properly choosing ϵ , we prove that if all the 'small blocks' in this neural network are removed, the perturbation to the output for any input ∥ x ∥ ≤ 1 is no more than δ/ 2 , so the covering number of the ConvResNeXt is only determined by the number of 'large blocks', which is no more than B res /ϵ .

Proof. Using the inequality of arithmetic and geometric means, from Proposition F.6, Proposition F.8 and Proposition F.9, if any residual block is removed, the perturbation to the output is no more than

where B m is the total norm of parameters in this block. Because of that, the residual blocks can be divided into two kinds depending on the norm of the weights B m < ϵ ('small blocks') and B m ≥ ϵ ('large blocks'). If all the 'small blocks' are removed, the perturbation to the output for any input ∥ x ∥ 2 ≤ 1 is no more than

Choosing ϵ = L ( δL 2 exp(( KB res /L ) L/ 2 ) K L/ 2 B res B 1 / 2 out ) 1 / ( L/ 2 -1) , the perturbation above is no more than δ/ 2 . The covering number can be determined by the number of the 'large blocks' in the neural network, which is no more than B res /ϵ . As for any block, B in L post ≤ B 1 / 2 out exp(( KB res /L ) L/ 2 ) , taking our chosen ϵ finishes the proof, where B in is the upper bound of the input to this block defined in Proposition D.1, and L post is the Lipschitz constant of all the layers following the block.

Remark 4.2 . The proof of Lemma 4.1 shows that under weight decay, the building blocks in a ConvResNeXt are sparse, i.e. only a finite number of blocks contribute non-trivially to the network even though the model can be overparameterized. This explains why a ConvResNeXt can generalize well despite overparameterization, and provide a new perspective in explaining why residual connections improve the performance of deep neural networks.

5 Discussions

This paper focuses on developing insightful generalization bounds for the regularized empirical risk minimizer. We opt not to delve into the end-to-end analysis of optimization algorithms in order to explore the adaptivity of complex architectures such as ConvResNeXts, while works on optimization behaviour of neural networks are limited to simple network structures [28, 39]. Notably, this approach to decouple learning and optimization has been widely adopted [3, 17, 31, 6, 25]. We made the same choice in the interest of getting a more fine-grained learning theory. However, our paper considers weight decay and overparameterization which are tightly connected to real-world training of neural networks, and can be the most promising work to bridge the gap between optimization and statistical guarantees. We defer more details to the appendix, including discussions on the Besov space and numerical experiments for supporting our theories as well as supplementary technical proof.

Acknowledgments

The work was partially supported by NSF Award No 2134214. KZ and YW were with UCSB and MC was with Princeton when the work was completed. YT contributed to the project during his summer 2023 visit to UCSB. We appreciate anonymous reviewers and ACs for their input.

References

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via overparameterization. In International conference on machine learning , pages 242-252. PMLR, 2019.

D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993.

P. L. Bartlett, O. Bousquet, and S. Mendelson. Local rademacher complexities. Annals of Statistics , pages 1497-1537, 2005.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

M. Chen, H. Jiang, W. Liao, and T. Zhao. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA , 11(4):1203-1253, 2022.

G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

R. A. DeVore and G. G. Lorentz. Constructive approximation , volume 303. Springer Science & Business Media, 1993.

D. L. Donoho, R. C. Liu, and B. MacGibbon. Minimax risk over hyperrectangles, and implications. The Annals of Statistics , pages 1416-1437, 1990.

D. L. Donoho, I. M. Johnstone, et al. Minimax estimation via wavelet shrinkage. The annals of Statistics , 26(3):879-921, 1998.

D. D˜ ung. Optimal adaptive sampling recovery. Advances in Computational Mathematics , 34 (1):1-41, 2011.

H. Federer. Curvature measures. Transactions of the American Mathematical Society , 93(3): 418-491, 1959.

D. Geller and I. Z. Pesenson. Band-limited localized parseval frames and besov spaces on compact homogeneous manifolds. Journal of Geometric Analysis , 21(2):334-371, 2011.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014.

A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 6645-6649. IEEE, 2013.

S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3389-3396. IEEE, 2017.

M. Hamers and M. Kohler. Nonasymptotic bounds on the l 2 error of neural network regression estimates. Annals of the Institute of Statistical Mathematics , 58:131-151, 2006.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132-7141, 2018.

T. Hu, Z. Shang, and G. Cheng. Sharp rate of convergence for deep neural network classifiers under the teacher-student setting. arXiv preprint arXiv:2001.06892 , 2020.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

M. Kohler and A. Krzy˙ zak. Adaptive regression estimation with multilayer feedforward neural networks. Nonparametric Statistics , 17(8):891-913, 2005.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012.

J. M. Lee. Riemannian manifolds: an introduction to curvature , volume 176. Springer Science &Business Media, 2006.

H. Liu, M. Chen, T. Zhao, and W. Liao. Besov function approximation and binary classification on low-dimensional manifolds using convolutional residual networks. In International Conference on Machine Learning , pages 6770-6780. PMLR, 2021.

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

E. Mammen and S. van de Geer. Locally adaptive regression splines. The Annals of Statistics , 25(1):387-413, 1997.

E. Nichani, A. Damian, and J. D. Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks. Advances in Neural Information Processing Systems , 36, 2024.

P. Niyogi, S. Smale, and S. Weinberger. Finding the homology of submanifolds with high confidence from random samples. Discrete & Computational Geometry , 39:419-441, 2008.

K. Oono and T. Suzuki. Approximation and non-parametric estimation of resnet-type convolutional neural networks. In International conference on machine learning , pages 4922-4931. PMLR, 2019.

J. Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics , 48:1875-1897, 2020.

T. Suzuki. Adaptivity of deep reLU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations , 2019.

C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

H. Tribel. Theory of function space ii. Monographs in Mathematics , 78, 1992.

L. W. Tu. Manifolds. In An Introduction to Manifolds , pages 47-83. Springer, 2011.

M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.013.

S. Wang and Z. Shang. Minimax optimal high-dimensional classification using deep neural networks. Stat , 11(1):e482, 2022.

S. Wang, G. Cao, Z. Shang, and A. D. N. Initiative. Deep neural network classifier for multidimensional functional data. Scandinavian Journal of Statistics , 50(4):1667-1686, 2023.

Z. Wang, E. Nichani, and J. D. Lee. Learning hierarchical polynomials with three-layer neural networks. In International Conference on Learning Representations , 2024.

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

D. Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks , 94: 103-114, 2017.

T. Young, D. Hazarika, S. Poria, and E. Cambria. Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine , 13:55-75, 2018.

K. Zhang and Y.-X. Wang. Deep learning meets nonparametric regression: Are weight-decayed DNNs locally adaptive? In International Conference on Learning Representations , 2023.

Q. Zhang, Z. Cui, X. Niu, S. Geng, and Y. Qiao. Image segmentation with pyramid dilated convolution based on resnet and u-net. In Neural Information Processing , pages 364-372. Springer, 2017.

A Why Besov Classes?

In this section, we discuss why we choose to consider the Besov class of functions and why this makes our results particularly interesting.

To see this, we need to first define two smaller function classes: the Holder class and the Sobolev class. Instead of giving fully general definitions for these function classes let us illustrate their main differences using univariate functions defined on [0 , 1] . We also introduce the so-called Total Variation class - which is sandwiched in between Besov( p = 1 , q = 1) and Besov( p = 1 , q = ∞ ) .

Holder class functions satisfy | f ( α ) ( x ) | < C for all x .

Sobolev class functions satisfy ∫ [0 , 1] | f ( α ) ( x ) | 2 dx < C

Total Variation class functions satisfy ∫ [0 , 1] | f ( α ) ( x ) | dx < C

The L1-norm used in defining total variation class makes it the most flexible of the three. It allows functions with α th order derivative f ( α ) ( x ) to be very large at some places, e.g., Dirac delta functions, while Holder and Sobolev class functions cannot contain such spikes (no longer integrable in Sobolev norm above).

Generically speaking under the appropriate scaling: Holder ⊂ Sobolev ⊂ Besov . The Besov space contains functions with heterogeneous smoothness while Holder and Sobolev classes contain functions with homogeneous smoothness. Despite the Besov space being larger, it has the same minimax rate of n -(2 α ) / (2 α + d ) as the smaller Holder and Sobolev class.

A new perspective on overparameterized NN. We study the adaptivity of deep networks in overparameterized regimes. The most popular method for understanding overparameterization is through the neural tangent kernel (NTK) regime. However, based on the classical linear smoother lower-bound for estimating functions in Besov classes with p = 1 [9, 10], all kernel ridge regression including any NTKs will have a suboptimal rate lower bounded by n -2 α -d 2 α . To say it differently, there is a formal separation between NTKs and the optimal method. The same separation does not exist in smaller function classes such as Sobolev and Holders because they are more homogeneously smooth.

In summary, in order to study what neural networks can achieve that is not achievable by kernels, e.g., NTK; we had to define and approximate Besov class functions. Our results show that ConvResNeXT not only overcomes the curse of dimensionality of the ambient space, but also has nearly optimal dependence in the intrinsic dimension d -in contrast to the kernel-based approaches.

We believe this offers a new perspective to understand overparameterization and is more fine-grained that of NTK.

B Numerical Simulation

In this section, we validate our theoretical findings with numerical experiments. We focus on nonparametric regression problems for simplicity and consider the following function f 0 : R D → R :

where U ∈ R D × D is a randomly-chosen rotation matrix and ˜ x = Ux ∈ R D satisfies that for t ∈ [0 , 1] , the first three coordinates

and the remaining coordinates of ˜ x are irrelevant features iid sampled from a uniform distribution. Note that the first three coordinates of ˜ x are completely determined by a scalar t , and the corresponding label y is determined by t via a piecewise linear function, i.e., for a bag of t 1 , ..., t n ∈ [0 , 1] , we can generate a labeled dataset by y i = g 0 ( t i ) + N (0 , 1) . An illustration of the function f 0 is given in Figure 2 where colors indicate the value.

Role of irrelevant features and rotation. The purpose of irrelevant features and rotation is to make the problem harder and more interesting.

10.0

7.5

5.0

0.25

0.20

2.5

0.15

0.10

0.05

0.0

0.00

2.5

1.00

0.75

0.50

0.6

0.25

0.00

0.0

0.2

0.4

-5.0

0.25

0.2

0.50

0.75

-0.8

-0.6

-0.4

7.5

where t i , i = 1 , . . . , n are evenly spaced over [0 , 1] . This process generates a 1 -dimensional manifold in R 3 which does not intersect with itself, as shown in Figure 2.

Baseline methods To estimate the underlying function on a manifold, we conducted experiments with ConvResNeXts (this paper), as well as a mix of popular off-the-shelf methods including kernel ridge regression, XGBoost, Decision tree, Lasso regression, and Gaussian Processes.

Hyperparameter choices. In all the experiments the following architecture was used for PNN: w = 6 , L = 10 , M = 4 , batch _ size = 128 , learning _ rate = 1e -3

In all the experiments the following architecture was used for ConvResNeXt: w = 8 , L = 6 , K = 6 , M = 2 , N = 2 . Batch_size and learning_rate were adjusted for each task.

For off-the-shelf methods, their hyperparameters are either tuned automatically or avoided using tools provided from the package, e.g., GP. For GP, a Matern kernel is used, and for ridge regression, the standard Gaussian RBF kernel is used.

Results. Our results are reported in Figure 3, 4, 5 which reports the mean square error (MSE) as a function of the effective degree-of-freedom of each method, ambient dimension D and also the number of data points n respectively.

As we can see in Figure 3, ConvResNeXt is able to achieve the lowest MSE at a relatively smaller degree of freedom. It outperforms the competing methods with notable margins despite using a simpler hypothesis.

Figure 4 illustrates that standard non-parametric methods such as kernel ridge regression and Gaussian processes deteriorate quickly as the ambient dimension gets bigger. On the contrary, ConvResNeXt and PNN obtain results that are almost dimension-independent due to the representation learning that helps identify the low-dimensional manifold.

Finally, the log-log plot in Figure 5 demonstrates that there is a substantially different rate of convergence between our methods and kernel ridge regression and GPs, indicating the same formal separation that we have established in the theoretical part - kernels must be suboptimal for estimating Besov classes while the neural architectures we considered can be locally adaptive and nearly optimal for Besov classes.

100

H

dofvsmse

lasso_regression

kernel_ridge

decision_tree

xgboost

101

convresnext

100

500

1000

1500

2000

dof

lasso_regression decision_tree

kernel_ridge xgboost

gp convresnext

101

2 × 101

Dimensions

C Proof of the approximation theory

C.1 Decompose the target function into the sum of locally supported functions.

Lemma C.1. Approximating Besov function on a smooth manifold using B-spline: Let f ∈ B α p,q ( M ) . There exists a decomposition of f :

and ˜ f i = f · ρ i ∈ B α p,q , ∑ C M i =1 ∥ ˜ f i ∥ B α p,q ≤ C ∥ f ∥ B α p,q ( M ) , ϕ i : M→ R d are linear projections, B ( c i , r ) denotes the unit ball with radius r and center c i .

The lemma is inferred by the existence of the partition of unity, which is given in Proposition 2.6.

C.2 Locally approximate the decomposed functions using cardinal B-spline basis functions.

Proposition C.2. For any function in the Besov space on a compact smooth manifold f ∗ ∈ B s p,q ( M ) , any N ≥ 0 , there exists an approximated to f ∗ using cardinal B-spline basis functions:

100

lasso_regression

kernel_ridge

10-1

decision_tree

xgboost

convresnext

103

104

Number of data

where m is the integer satisfying 0 < α < min ( m,m -1 + 1 /p ) , M m,k, s = M m (2 k ( · -s )) , M m denotes the B-spline basis function defined in (2), the approximation error is bounded by

for some constant C 9 , C 10 that only depends on α .

As will be shown below, the scaled coefficients 2 k j a i,k j , s j corresponds to the total norm of the parameters in the neural network to approximate the B-spline basis function, so this lemma is the key to get the bound of norm of parameters in (12).

Proof. From the definition of B α p,q ( M ) , and applying Proposition 2.6, there exists a decomposition of f ∗ as

where f i := f ∗ · ρ i , ρ i satisfy the condition in Definition 2.5, and f i ◦ ϕ -1 i ∈ B α p,q . Using Proposition F.2, for any i , one can approximate f i ◦ ϕ -1 i with ¯ f i :

such that ∥ f i ◦ ϕ -1 i ∥ ∞ ≤ C 1 M -α/d , and the coefficients satisfy

one can verify that ∥ f -˜ f ∥ ∞ ≤ C 9 C M N -α/d . On the other hand, using triangular inequality (and padding the vectors with 0),

which finishes the proof.

C.3 Neural network for chart selection

In this section, we demonstrate that a feedforward neural network can approximate the chart selection function z × 1 ( x ∈ B ( c i , r )) , and it is error-free as long as z = 0 when r < d ( x , c i ) < R . We start by proving the following supporting lemma:

Proposition C.3. Fix some constant B > 0 . For any x , c ∈ R D satisfying | x i | ≤ B and | c i | ≤ B for i = 1 , . . . , D , there exists an L -layer neural network ˜ d ( x ; c ) with width w = O ( d ) that approximates d 2 ( x ; c ) = ∑ D i =1 ( x i -c i ) 2 such that | ˜ d 2 ( x ; c ) -d 2 ( x ; c ) | ≤ 8 DB 2 exp( -C 11 L ) with an absolute constant C 11 > 0 when d ( x ; c ) < τ , and ˜ d 2 ( x ; c ) ≥ τ 2 when d ( x ; c ) ≥ τ , and the norm of the neural network is bounded by

and the coefficients satisfy

Define

Proof. The proof is given by construction. By Proposition 2 in Yarotsky(2017), the function f ( x ) = x 2 on the segment [0 , 2 B ] can be approximated with any error ϵ > 0 by a ReLU network g having depth and the number of neurons and weight parameters no more than c log(4 B 2 /ϵ ) with an absolute constant c . The width of the network g is an absolute constant. We also consider a single layer ReLU neural network h ( t ) = σ ( t ) -σ ( -t ) , which is equal to the absolute value of the input.

Now we consider a neural network G ( x ; c ) = ∑ D i =1 g ◦ h ( x i -c i ) . Then for any x , c ∈ R D satisfying | x i | ≤ B and | c i | ≤ B for i = 1 , . . . , D , we have

Moreover, define another neural network

which has depth and the number of neurons no more than c ′ log(4 B 2 /ϵ ) with an absolute constant c ′ . The weight parameters of G are upper bounded by max { τ 2 , Dϵ, c log(4 B 2 /ϵ ) } and the width of G is O ( D ) .

If d 2 ( x ; c ) < τ 2 , we have

For the first case when G ( x ; c ) < τ 2 -Dϵ , | F ( x ; c ) -d 2 ( x ; c ) | ≤ 2 Dϵ since d 2 ( x ; c ) can be approximated by G ( x ; c ) up to an error ϵ . For the second case when G ( x ; c ) ≥ τ 2 -Dϵ , we have d 2 ( x ; c ) ≥ G ( x ; c ) -Dϵ ≥ τ 2 -2 Dϵ and . Thereby we also have | F ( x ; c ) -d 2 ( x ; c ) | ≤ 2 Dϵ .

If d 2 ( x ; c ) ≥ τ 2 instead, we will obtain G ( x ; c ) ≥ d 2 ( x ; c ) -Dϵ ≥ τ 2 -Dϵ . This gives that F ( x ; c ) = τ 2 in this case.

Finally, we take ϵ = 4 B 2 exp( -L/c ′ ) . Then F ( x ; c ) is an L -layer neural network with O ( L ) neurons. The weight parameters of G are upper bounded by max { τ 2 , 4 DB 2 exp( -L/c ′ ) , cL/c ′ } and the width of G is O ( D ) . Moreover, F ( x ; c ) satisfies | F ( x ; c ) -d 2 ( x ; c ) | < 8 DB 2 exp( -L/c ′ ) if d 2 ( x ; c ) ≤ τ 2 and F ( x ; c ) = τ 2 if d 2 ( x ; c ) ≥ τ 2 .

Proposition C.4. There exists a single layer ReLU neural network that approximates ˜ × , such that for all 0 ≤ x ≤ C, y ∈ { 0 , 1 } , x ˜ × y = x when y = 1 , and x ˜ × y = 0 when either x = 0 or y = 0 .

Proof. Consider a single layer neural network g ( x , y ) := A 2 σ ( A 1 ( x , y ) ⊤ ) with no bias, where

Then we can rewrite the neural network g as g ( x, y ) = -Cσ ( -x/C + y ) + Cσ ( y ) . If y = 1 , we will have g ( x, y ) = -Cσ ( -x/C + 1) + C = x , since x ≤ C . If y = 0 , we will have g ( x, y ) = -Cσ ( -x/C ) = 0 , since x ≥ 0 . Thereby we can conclude the proof.

By adding a single linear layer

after the one shown in Proposition C.3, where ∆ = 8 DB 2 exp( -CL ) denotes the error in Proposition C.3, one can approximate the indicator function 1 ( x ∈ B ( c i , r )) such that it is error-free when d ( x , c i ) ≤ r or ≥ R . Choosing R ≤ τ/ 2 , r < R -2∆ , and combining with Proposition C.4, the proof is finished. Considering that f i is locally supported on B ( c i , r ) for all i by our method of construction, the chart selection part does not incur any error in the output.

C.4 Constructing the neural network to Approximate the target function

In this section, we focus on the neural network with the same architecture as a ResNeXt in Definition 2.12 but replacing each building block with a feedforward neural network, and prove that it can achieve the same approximation error as in Theorem 3.2. For technical simplicity, we assume that the target function f ∗ ∈ [0 , 1] without loss of generality. Then our analysis automatically holds for any bounded function.

Theorem C.5. For any f ∗ under the same condition as Theorem 3.2, any neural network architecture with residual connections containing N number of residual blocks and each residual block contains M number of feedforward neural networks in parallel, where the depth of each feedforward neural networks is L , width is w :

where P ( x ) = [ x T , 1 , 0] T is the padding operation, satisfying

there exists an instance f of this ResNeXt class, such that

where C 1 , C 2 , C 3 , C 4 , C 5 , C 6 are the same constants as in Theorem 3.2.

Proof. We first construct a parallel neural network to approximate the target function, then scale the weights to meet the norm constraint while keeping the model equivalent to the one constructed in the first step, and finally transform this parallel neural network into the ConvResNeXt as claimed.

Combining Lemma F.3, Proposition C.3 and Proposition C.4, by putting the neural network in Lemma F.3 and Proposition C.3 in parallel and adding the one in Proposition C.4 after them, one can construct a feedforward neural network with bias with depth L , width w = O ( d ) + O ( D ) = O ( d ) , that approximates M m,k j , s j ( x ) × 1 ( x ∈ B ( c i , r )) for any i, j .

To construct the neural network with residual connections that approximates f ∗ , we follow the method in Oono and Suzuki [30], Liu et al. [25]. This network uses separate channels for the inputs and outputs. Let the input to one residual layer be [ x 1 , y 1 ] , the output is [ x 1 , y 1 + f ( x 1 )] . As a result, if one scale the outputs of all the building blocks by any scalar a , then the last channel of the output of the entire network is also scaled by a . This property allows us to scale the weights in each building block while keeping the model equivalent. To compensate for the bias term, Proposition F.4 can be applied. This only increases the total norm of each building block by no larger than a constant term that depends only L , which is no more than a factor of constant.

Let the neural network constructed above has parameter ˜ W ( i,j ) 1 , ˜ b ( i,j ) 1 , . . . , ˜ W ( i,j ) L , b ( i,j ) L in each layer, one can construct a building block without bias as

Remind that the input is padded with the scalar 1 before feeding into the neural network, the above construction provide an equivalent representation to the neural network including the bias, and route the output to the last channel. From Lemma F.3, it can be seen that the total square norm of this block is bounded by (10).

Finally, we scale the weights in the each block, including the '1' terms to meet the norm constraint. Thanks to the 1-homogeneous property of ReLU layer, and considering that the network we construct use separate channels for the inputs and outputs, the model is equivalent after scaling. Actually the property above allows the tradeoff between B res and B out. If all the weights in the residual blocks are scaled by an arbitrary positive constant c , and the weight in the last layer W out is scaled by c -L , the model is still equivalent. We only need to scale the all the weights in this block with | a i,k j , s j | 1 /L , setting the sign of the weight in the last layer as sign( a i,k j , s j ) , and place C M P number of these building blocks in this neural network with residual connections. Since this block always output 0 in the first D +1 channels, the order and the placement of the building blocks does not change the output. The last fully connected layer can be simply set to

Combining Proposition F.2 and Lemma F.1, the norm of this ResNeXt we construct satisfy

By scaling all the weights in the residual blocks by ¯ B -1 / 2 res , and scaling the output layer by ¯ B L/ 2 res , the network that satisfy (12) can be constructed.

Notice that the chart selection part does not introduce error by our way of construction, we only need to sum over the error in Section 4.1 and Section 4.1, and notice that for any x , for any linear projection ϕ i , the number of B-spline basis functions M m,k, s that is nonzero on x is no more than m d log P , the approximation error of the constructed neural network can be proved.

C.5 Constructing a convolution neural network to approximate the target function

In this section, we prove that any feedforward neural network can be realized by a convolution neural network with similar size and norm of parameters. The proof is similar to Theorem 5 in [30].

Lemma C.6. For any feedforward neural network with depth L ′ , width w ′ , input dimension h and output dimension h ′ , for any kernel size K > 1 , there exists a convolution neural network with depth L = L ′ + L 0 -1 , where L 0 = ⌈ h -1 K -1 ⌉ number of channels w = 4 w ′ , and the first dimension of the output equals the output of the feedforward neural network for all inputs, and the norm of the convolution neural network is bounded as

where W ′ 1 ∈ R w ′ × h ′ ; W ′ ℓ ∈ R w ′ × w ′ , ℓ = 2 , . . . , L ′ -1; W ′ L ′ ∈ R h ′ × w ′ are the weights in the feedforward neural network, and W 1 ∈ R K × w × h , W ℓ ∈ R K × w × w , ℓ = 2 , . . . , L -1; W L ∈ R K × h × w are the weights in the convolution neural network.

Proof. We follow the same method as Oono and Suzuki [30] to construct the CNN that is equivalent to the feedforward neural network. By combining Oono and Suzuki [30] lemma 1 and lemma 2, for

any linear transformation, one can construct a convolution neural network with at most L 0 = ⌈ h -1 K -1 ⌉ convolution layers and 4 channels, where h is the dimension of input, which equals D +1 in our case, such that the first dimension in the output equals the linear transformation, and the norm of all the weights is no more than

where W ℓ is the weight of the linear transformation. Putting w number of such convolution neural networks in parallel, a convolution neural network with L 0 layers and 4 w channels can be constructed to implement the first layer in the feedforward neural network.

To implement the remaining layers, one choose the convolution kernel W ℓ + L 0 -1 [: , i, j ] = [0 , . . . , W ′ [ i, j ] , . . . , 0] , ∀ 1 ≤ i, j ≤ w , and pad the remaining parts with 0, such that this convolution layer is equivalent to the linear layer applied on the dimension of channels. Noticing that this conversion does not change the norm of the parameters in each layer. Adding both sides of (14) by the norm of the 2 -L ′ -th layer in both models finishes the proof.

D Proof of the estimation theory

D.1 Covering number of a neural network block

Proposition D.1. If the input to a ReLU neural network is bounded by ∥ x ∥ 2 ≤ B in , the covering number of the ReLU neural network defined in Proposition F.6 is bounded by

Proof. Similar to Proposition F.6, we only consider the case ∥ W ℓ ∥ F ≤ √ B/L . For any 1 ≤ ℓ ≤ L , for any W 1 , . . . W ℓ -1 , W ℓ , W ′ ℓ , W ℓ +1 , . . . W L that satisfy the above constraint and ∥ W ℓ -W ′ ℓ ∥ F ≤ ϵ , define g ( . . . ; W 1 , . . . W L ) as the neural network with parameters W 1 , . . . W L , we can see

Choosing ϵ = δ L ( B/L ) ( L -1) / 2 , the above inequality is no larger than δ/L . Taking the sum over ℓ , we can see that for any W 1 , W ′ 1 , . . . , W L , W ′ L such that ∥ W ℓ -W ′ ℓ ∥ F ≤ ϵ ,

Finally, observe that the covering number of W ℓ is bounded by

Substituting B and ϵ and taking the product over ℓ finishes the proof.

Proposition D.2. If the input to a ReLU convolution neural network is bounded by ∥ x ∥ 2 ≤ B in , the covering number of the ReLU neural network defined in (5) is bounded by

Proof. Similar to Proposition D.1, for any 1 ≤ ℓ ≤ L , for any W 1 , . . . W ℓ -1 , W ℓ , W ′ ℓ , W ℓ +1 , . . . W L that satisfy the above constraint and ∥ W ℓ -W ′ ℓ ∥ F ≤ ϵ , define g ( . . . ; W 1 , . . . W L ) as the neural network with parameters W 1 , . . . W L , we can see

where the first inequality comes from Proposition F.10. Choosing ϵ = δ K L/ 2 B in L ( B/L ) ( L -1) / 2 , the above inequality is no larger than δ/L . Taking this into (15) finishes the proof.

D.2 Proof of Theorem 3.3

Define ˜ f = arg min f E D [ L ( f )] . From Theorem 14.20 in Wainwright [36], for any function class ∂ F that is star-shaped around ˜ f , the empirical risk minimizer ˆ f = arg min f ∈F L n ( f ) satisfy

with probability at least 1 -c 1 exp( -c 2 nδ 2 n ) for any δ n that satisfy (20), where c 1 , c 2 are universal constants.

The function of neural networks is not star-shaped, but can be covered by a star-shaped function class. Specifically, let { f -˜ f : f ∈ F Conv } ⊂ { f 1 -f 2 : f 1 , f 2 ∈ F Conv } := ∂ F .

Any function in ∂ F can be represented using a ResNeXt: one can put two neural networks of the same structure in parallel, adjusting the sign of parameters in one of the neural networks and summing up the result, which increases M,B res and B out by a factor of 2. This only increases the log covering number in (11) by a factor of constant (remind that B res = O (1) by assumption).

Taking the log covering number of the ResNeXt (11), the sufficient condition for the critical radius as in (20) is

where ≲ hides the logarithmic term.

Because L is 1-Lipschitz, we have

Choosing

and taking Theorem 3.2 and (17) into (16) finishes the proof.

E Lower bound of error

In this section, we study the minimax lower bound of any estimator for Besov functions on a d -dimensional manifold. It suffices to consider the manifold M as a d -dimensional hypersurface. Without the loss of generalization, assume that ∂ L ( y ) ∂y ≥ 0 . 5 for -ϵ ≤ y ≤ ϵ . Define the function space

where M ( m ) denotes the Cardinal B-spline basis function that is supported on (0 , 1) d , j = [ j 1 , . . . , j d ] . The support of each B-spline basis function splits the space into s d number of blocks, where the target function in each block has two choices (positive or negative), so the total number of different functions in this function class is |F| = 2 s d . Using D˜ ung [11, Theorm 2.2], we can see that for any f ∈ F ,

For a fixed f ∗ ∈ F , let D = { ( x i , y i ) } n i =1 be a set of noisy observations with y i = f ∗ ( x i )+ ϵ i , ϵ i ∼ SubGaussian (0 , σ 2 I ) . Further assume that x i are evenly distributed in (0 , 1) d such that in all

regions as defined in (18), the number of samples is n j := O ( n/s d ) . Using Le Cam's inequality, we get that in any region, any estimator θ satisfy

as long as ( ϵ σs α ) 2 ≲ s d n , where ∥ · ∥ j := 1 n i ∑ s ( x -j ) ∈ [0 , 1] d | f ( x ) | denotes the norm defined in the block indexed by i , C m is a constant that depends only on m . Choosing s = O ( n 1 2 α + d ) , we get

Observing 1 n ∑ n i =1 L ( ˆ ( f ( x i ))) ≥ 0 . 5 ∑ n i =1 | f ( x i ) -f ∗ ( x i ) | ≂ 1 s d ∑ j ∈ [ s ] d ∥ ˆ f -f ∗ ∥ j finishes the proof.

F Supporting theorem

Lemma F.1. [Lemma 14 in Zhang and Wang [43]] For any a ∈ R ¯ M , 0 < p ′ < p , it holds that:

Proposition F.2 (Proposition 7 in Zhang and Wang [43]) . Let α -d/p > 1 , r > 0 . For any function in Besov space f ∗ ∈ B α p,q and any positive integer ¯ M , there is an ¯ M -sparse approximation using B-spline basis of order m satisfying 0 < α < min( m,m -1 + 1 /p ) : ˇ f ¯ M = ∑ ¯ M i =1 a k i , s i M m,k i , s i for any positive integer ¯ M such that the approximation error is bounded as ∥ ˇ f ¯ M -f ∗ ∥ r ≲ ¯ M -α/d ∥ f ∗ ∥ B α p,q , and the coefficients satisfy

Lemma F.3 (Lemma 11 in [43]) . Let M m,k, s be the B-spline of order m with scale 2 -k in each dimension and position s ∈ R d : M m,k, s ( x ) := M m (2 k ( x -s )) , M m is defined in (2). There exists a neural network with d -dimensional input and one output, with width w d,m = O ( dm ) and depth L ≲ log( C 13 /ϵ ) for some constant C 13 that depends only on m and d , approximates the B spline basis function M m,k, s ( x ) := M m (2 k ( x -s )) . This neural network, denoted as ˜ M m,k, s ( x ) , x ∈ R d , satisfy

˜ M m,k, s ( x ) = 0 , otherwise.

The total square norm of the weights is bounded by 2 2 k/L C 14 dmL for some universal constant C 14 .

Proposition F.4. For any feedforward neural network f with width w and depth L with bias, there exists a feedforward neural network f ′ with width w ′ = w +1 and depth L ′ = L , such that for any x , f ( x ) = f ′ ([ x T , 1] T )

Proof. Proof by construction: let the weights in the ℓ -th layer in f be W ℓ , and the bias be b ℓ , and choose the weight in the corresponding layer in f ′ be

The constructed neural network gives the same output as the original one.

Corollary F.5 (Corollary 13.7 and Corollary 14.3 in Wainwright [36]) . Let

denotes the local Gaussian complexity and local Rademacher complexity respectively, where w i ∼ N (0 , 1) are the i.i.d. Gaussian random variables, and ϵ i ∼ uniform {-1 , 1 } are the Rademacher random variables. Suppose that the function class F is star-shaped, for any σ > 0 , any δ ∈ (0 , σ ] such that satisfies

satisfies

Furthermore, if F is uniformly bounded by b , i.e. ∀ f ∈ F , x | f ( x ) | ≤ b any δ > 0 such that

Proposition F.6. An L -layer ReLU neural network with no bias and bounded norm

is Lipschitz continuous with Lipschitz constant ( B/L ) L/ 2

Proof. Notice that ReLU function is 1-homogeneous, similar to Proposition 4 in [43], for any neural network there exists an equivalent model satisfying ∥ W ℓ ∥ F = ∥ W ℓ ′ ∥ F for any ℓ, ℓ ′ , and its total norm of parameters is no larger than the original model. Because of that, it suffices to consider the neural network satisfying ∥ W ℓ ∥ F ≤ √ B/L for all ℓ . The Lipschitz constant of such linear layer is ∥ W ℓ |∥ 2 ≤ ∥ W ℓ |∥ F ≤ √ B/L , and the Lipschitz constant of ReLU layer is 1 . Taking the product over all layers finishes the proof.

Proposition F.7. An L -layer ReLU convolution neural network with convolution kernel size K , no bias and bounded norm

is Lipschitz continuous with Lipschitz constant ( KB/L ) L/ 2

This proposition can be proved by taking Proposition F.10 into the proof of Proposition F.6.

Proposition F.8. Let f = f post ◦ (1 + f NN + f other ) ◦ f pre be a ResNeXt, where 1 + f NN + f other denotes a residual block, f pre and f post denotes the part of the neural network before and after this residual block, respectively. f NN denotes one of the building block in this residual block and f other denotes the other residual blocks. Assume f pre , f NN , f post are Lipschitz continuous with Lipschitz constant L pre , L NN , L post respectively. Let the input be x , if the residual block is removed, the perturbation to the output is no more than L pre L NN L post ∥ x ∥

Proof.

Proposition F.9. The neural network defined in Lemma 4.1 with arbitrary number of blocks has Lipschitz constant exp(( KB res /L ) L/ 2 ) , where K = 1 when the feedforward neural network is the building blocks and K is the size of the convolution kernel when the convolution neural network is the building blocks.

Proof. Note that the m -th block in the neural network defined in Lemma 4.1 can be represented as y = f m ( x ; ω m ) + x , where f m is an L -layer feedforward neural network with no bias. By Proposition F.6 and Proposition F.7, such block is Lipschitz continuous with Lipschitz constant 1+( KB m /L ) L/ 2 , where the weight parameters of the m -th block satisfy that ∑ L ℓ =1 ∥ W ( m ) ℓ ∥ 2 F ≤ B m and ∑ M m =1 B m ≤ B res .

Since the neural network defined in Lemma 4.1 is a composition of M blocks, it is Lipschitz with Lipschitz constant L res . We have

where we use the inequality 1 + z ≤ exp( x ) for any x ∈ R . Furthermore, notice that ∑ M m =1 ( KB m /L ) L/ 2 is convex with respect to ( B 1 , B 2 , . . . , B M ) when L > 2 . Since ∑ M m =1 B m ≤ B res and B m ≥ 0 , then we have ∑ M m =1 ( KB m /L ) L/ 2 ≤ ( KB res /L ) L/ 2 by convexity. Therefore, we obtain that L res ≤ exp(( KB res /L ) L/ 2 ) .

Proposition F.10. For any x ∈ R d , w ∈ R K , K ≤ d , ∥ Conv( x , w ) ∥ 2 ≤ √ K ∥ x ∥ 2 ∥ w ∥ 2 .

Proof. For simplicity, denote x i = 0 for i ≤ 0 or i > d .

where the second line comes from Cauchy-Schwarz inequality, the third line comes by expanding ∥ x [ i -K -1 2 : i + K -1 2 ] ∥ 2 2 by definition and observing that each element in x appears at most K times.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main theories are presented in Section 3. Section 4 presents the proof sketch of the main results and highlights our technical contributions.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Section 5.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are presented in Section 3. We outline the proof sketch in Section 4 and defer proof details to Appendix C, D, E and F.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment details are given in Appendix B.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our experiments use simulated data and do not require any datasets. The experiment implementation is simple and clearly described in Appendix B, so we feel there is no need to publish data and code.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide our experimental results and details in Appendix B.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are included in our experiments in Appendix B. See Figure 3 for illustration.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our experiments do not require large compute resources and can be reproduced on laptops.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the performance of ConvResNets trained with weight decay from the perspective of nonparametric classification. Specifically, the authors consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNets can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of\ndimensionality.\n\nSTRENGTHS:\nThis is an excellent, novel and strong work. The paper provides theoretical guarantees for the ConvResNets in the setting of nonparametric classification. The minimax optimal rate under ConvResNets has been an open problem which was successfully resolved by the authors in the conditional class probability.\n\nWEAKNESSES:\nLiterature review is not sufficient. Minimax optimal nonparametric classification using DNN has been recently studied by other authors, though the current paper consider another novel neural network, the ConvResNets, which is different from the traditional DNN.\n\nQUESTIONS:\nNonparametric classification can be done in different frameworks such as (1) smooth decision boundary, (2) smooth conditional class probability or (3) smooth margin. The minimax optimal rates in the above settings are different. For instance, some authors find that DNN classification in (1) and (2) yield different minimax rates.  In particular, the minimax optimal rate under (1) is an open problem (see Kim's work) which was recently addressed by Hu et al. The minimax rate under (2) has been recently established for ultrahigh-dimensional case for DNN classification by Wang's work. The minimax rates under (1) and (2) may have different expressions.\n\nI guess that this paper considers the setting of smooth conditional class probability, and the manifold dimension D is fixed. It would be great to comment on its extension to ultrahigh-dimensional case as in Wang's work.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the performance of an overparametrized convolutional residual network architecture on a nonparametric classification problem trained with weight decay. This model, known as ConvResNeXt, involves N residual blocks, where each residual block has a parallel architecture of M building blocks, and each building block with L layers. As such it generalizes alternative ConvResNets in the literature and includes them as a special case, e.g., ConvResNet [He et al 2016] when M=1 and aggregated Resnet [Xie et at 2017] when N=1. Specifically, the analysis results consider infinitely many building blocks and shows that weight decay training implicitly enforces sparsity of these blocks considering that the optimal classifier is supported on a low dimensional smooth manifold. As such, they show that this model can efficiently learn the assumed function class without suffering from the  curse of dimensionality.\n\nSTRENGTHS:\n1)\tThe paper shows that for learning Besov functions trained with weight decay out of n samples, the rate of convergence depends only on the intrinsic low dimension enabling the curse of dimensionality.\n2)\tThey also provide a tighter bound for the weight-decayed ConvResNeXt in computing the critical radius of local Gaussian complexity.\n\nWEAKNESSES:\n1) The actual benefit of having parallel paths in the proposed ConvResNeXt model was not addressed fully. Indeed, the results stating that only the product of number of parallel paths M and depth N determines the convergence rate, and one may not need parallel blocks when residual connections are present contradict with some of the earlier findings in the literature. For example, in [Xie et at 2017]. In the mentioned prior work, several experimental results focus on the posive effect of having higher cardinality (more multiple parallel paths) than increased depth and width. \n2) If there is no additional benefit of having additional parallel paths, and an equivalent performance can be achieved by increasing the depth only, this will limit the capability of the proposed ConvResNeXt over the ConvResNets unline the claimed benefits of the former.\n\nQUESTIONS:\n1)\tThis new architecture introduce significantly more complex nested function form, but similar performance in both approximation and estimation errors can be achieved by rather simpler one. The provided bounds on the two errors do not suggest any practical benefit of ConvResNeXts over ConvResNets. If so, what would be the motivation to generalize for the former?\n2)\tIn Sec 3.2, it was stated that the number of building blocks M does not influence the estimation error as long as it is large enough. This must be contrasted with the earlier findings about the cardinality of the parallel paths, and any difference must be clearly stated.\n3)\tIn Sec 3.2, the tradeoff between width and depth was mentioned to be less important in ResNext. In particular, it was stated that the lower bound on the error does not depend on the arrangements of the blocks M and N as long as their product remains same and large enough. One should further address the reason behind this and compare it with the literature again. Moreover, if that’s so, does this suggest that an equivalent model that can achieve equivalent performance without parallel paths?\n\nLIMITATIONS:\nNo, authors did not adequately address the limitations of their work, and no potential negative societal impact of their work is identified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds the deep learning theory for studying convolutional residual neural networks with data lying on an embedded lower-dimensional manifold. Theoretical results on both approximation error and estimation error are provided, and interesting implications from the theory are discussed.\n\nSTRENGTHS:\n1. The paper is well-written and I have enjoyed reading it.\n2. The theory is well-developed and clear.\n3. Good contributions are made to the deep learning theory.\n\nWEAKNESSES:\n1. This paper avoids the curse of dimensionality through assuming that the data lies in a low-dimensional manifold embedded in the ambient space. While real world data typically exhibit manifold structure, they commonly don't lie precisely on a manifold, but instead are surrounding a manifold (i.e. + noise). The theory of this paper highly depends on the exact manifold assumption since it needs the associated partition of unity to split the problem into local charts.\n2. Among the remarks/implications made on page 7, it is said that overparameterization is fine. However, to my understanding, the problem of overparameterization is not about an overparameterized model not being expressive enough (which this paper addresses), but really about the difficulty in finding a good enough local minimizer for it during optimization. To this regard, Theorem 3.3 based on the global minimizer doesn't really answer it.\n\nQUESTIONS:\n1. The remarks on tradeoff between width and depth are very interesting. Have you tried this in numerical experiments to see if this theoretical driven intuition actually appears in experiments?\n2. Other than the analysis being more general, what are the theoretical benefits of adding the identity map (i.e. the residual network) and multiple CNN blocks together compared to just using a CNN block?\n\nLIMITATIONS:\nSee weaknesses and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper provide theoretical analysis for the good prediction performance of convolutional residual neural networks, even overparameterized.

STRENGTHS:
1. The theory they developed with overparamterized ConvResNeXts trained with weight decay is novel. 
2. their theory does not suffer from the curse of dimensionality.

WEAKNESSES:
In line 71, 72, the overparameterization is defined as " the number of blocks is larger than the order of the sample size n". Where is this definition comes from?

QUESTIONS:
1. Can you generalize from  the empirical logistic risk to other risks? How will the choice of loss change the bound?
2. Can you explain in details about the extra effort (underlying theoretical techinique) you made in this paper to differentiate from  Liu et al. 2021 (Besov function approximation and binary classification on low-dimensional manifolds using convolutional residual networks)? How do you generalize it to over-parameterized setting and ConvResNeXt?

LIMITATIONS:
The analysis is limited to convolutional networks. It's interensting to see if it can be further generalized to other architectures like transformers.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14