Title: Norm-based Generalization Bounds for Sparse Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalization bounds

convolution

rademacher

generalization

sparsity
Abstract: In this paper, we derive norm-based generalization bounds for sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones because they consider the sparse structure of the neural network architecture and the norms of the convolutional filters, rather than the norms of the (Toeplitz) matrices associated with the convolutional layers. Theoretically, we demonstrate that these bounds are significantly tighter than standard norm-based generalization bounds. Empirically, they offer relatively tight estimations of generalization for various simple classification problems. Collectively, these findings suggest that the sparsity of the underlying target function and the model's architecture plays a crucial role in the success of deep learning.

Norm-based Generalization Bounds for Sparse Neural Networks

Abstract

In this paper, we derive norm-based generalization bounds for sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones because they consider the sparse structure of the neural network architecture and the norms of the convolutional filters, rather than the norms of the (Toeplitz) matrices associated with the convolutional layers. Theoretically, we demonstrate that these bounds are significantly tighter than standard norm-based generalization bounds. Empirically, they offer relatively tight estimations of generalization for various simple classification problems. Collectively, these findings suggest that the sparsity of the underlying target function and the model's architecture plays a crucial role in the success of deep learning.

1 Introduction

Over the last decade, deep learning with large neural networks has significantly advanced the solution of a myriad of tasks. These include image classification [1, 2, 3], language processing [4, 5, 6], interactions with open-ended environments [7, 8], and code synthesis [9]. Contrary to traditional theories such as [10], recent findings [11, 12] indicate that deep neural networks can generalize effectively even when their size vastly exceeds the number of training samples.

To address this question, recent work has proposed different generalization guarantees for deep neural networks based on various norms of their weight matrices [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Many efforts have been made to improve the tightness of these bounds to realistic scales. Some studies have focused on developing norm-based generalization bounds for complex network architectures, such as residual networks [24]. Other studies investigated ways to reduce the dependence of the bounds on the product of spectral norms [21, 25], or to use compression bounds based on PAC-Bayes theory [26, 27], or on the optimization procedure used to train the networks [19, 28, 29]. However, most of this research is centered around fully-connected networks, which generally underperform compared to other architectures like convolutional networks [30], residual network [1] and transformers [4, 31]. Thus, the ability of these bounds to explain the success of contemporary architectures is rather limited.

To fully understand the success of deep learning, it is necessary to analyze a wider scope of architectures beyond fully-connected networks. An interesting recent direction [32, 33] introduces generalization bounds for neural networks with shared parameters, such as convolutional neural networks. For example, [32] showed that by taking into account the structure of the convolutional

layers, we can derive generalization bounds with a norm component smaller than the norm of the associated linear transformation. However, many questions remain unanswered, including (a) Why certain architectures, such as convolutional networks [30] and MLP-mixers [34], perform better than fully-connected neural networks? (b) Is weight sharing necessary for the success of convolutional neural networks? (c) Can we establish norm-based generalization bounds for convolutional neural networks that are reasonably tight in practical settings? In this paper, we contribute to an understanding of all three questions.

1.1 Related Work

Approximation guarantees for multilayer sparse networks. While fully-connected networks, including shallow networks, are universal approximators [35, 36] of continuous functions, they are largely limited in theory and in practice. Classic results [37, 38, 39, 40, 41] show that, in the worstcase, the number of parameters required to approximate a continuously differentiable target functions (with bounded derivatives) grows exponentially with the input dimension, a property known as the 'curse of dimensionality'.

A recent line of work [42, 43, 44] shows that the curse of dimensionality can be avoided by deep, sparse networks, when the target function is itself compositionally sparse. Furthermore, it has been conjectured that efficiently computable functions, that is functions that are computable by a Turing machine in polynomial time, are compositionally sparse. This suggests, in turns, that, for practical functions, deep and sparse networks can avoid the curse of dimensionality. These results, however, lack any implication about generalization; in particular, they do not show that overparametrized sparse networks have good generalization.

Norm-based generalization bounds. A recent thread in the literature [13, 14, 15, 16, 17, 18, 19, 20, 21, 23] has introduced norm-based generalization bounds for neural networks. In particular, let S = { ( x i , y i ) } i m =1 be a training dataset of m independently drawn samples from a probability measure P defined on the sample space X × Y , where X ⊂ R d and Y = {± 1 } . A fully-connected network is defined as f w ( x ) = W L σ ( W L -1 σ ( . . . σ ( W 2 σ ( W 1 x )) . . . )) , where W l ∈ R d l +1 × d l and σ ( x ) is the element-wise ReLU activation function max(0 , x ) . A common approach for estimating the gap between the train and test errors of a neural network is to use the Rademacher complexity of the network. For example, in [13], an upper bound on the Rademacher complexity is introduced based on the norms of the weight matrices of the network of order O ( 2 L √ m ∏ L l =1 ∥ W l ∥ F ) . Later, [14] showed that the exponential dependence on the depth can be avoided by using the contraction lemma and obtained a bound that scales with O ( √ L ) .

While these results provide solid upper bounds on the test error of deep neural networks, they only take into account very limited information about the architectural choices of the network. In particular, when applied to convolutional networks, the matrices W l represent the linear operation performed by a convolutional layer whose filters are w l . However, since W l applies w l to several patches ( d l patches), we have ∥ W l ∥ F = √ d l ∥ w l ∥ F . As a result, the bound scales with

O ( √ ∏ L -1 l =1 d l ) , that grows exponentially with L . This means that the bound is not suitable for convolutional networks with many layers as it would be very loose in practice. In this work, we establish generalization bounds that are customized for convolutional networks and scale with ∏ L l =1 ∥ w l ∥ F instead of ∏ L l =1 ∥ W l ∥ F .

In [45] they conducted a large-scale experiment evaluating multiple norm-based generalization bounds, including those of [17, 14]. They argued that these bounds are extremely loose and negatively correlated with the test error. However, in all of these experiments, they trained the neural networks with the cross-entropy loss which implicitly maximizes the network's weight norms once the network perfectly fits the training data. This can explain the observed negative correlation between the bounds and the error. In this work, we empirically show that our bounds provide reasonably tight estimations of the generalization gap for convolutional networks trained with weight normalization and weight decay using the MSE loss.

Generalization bounds for convolutional networks. Several recent papers have introduced generalization bounds for convolutional networks that take into account their unique structure. In [23], they introduced a generalization bound for neural networks with weight sharing. However, this bound only holds under the assumption that the weight matrices are orthonormal, which is not real-

istic in practice. In [33], they introduced generalization bounds for convolutional networks based on parameter counting. However, this bound scales roughly as the square root of the ratio between the number of parameters and the number of samples, which is vacuous when the network is overparameterized. In [32], they extended the generalization bounds of [17] for convolutional networks where the linear transformations W l at each layer are replaced with the trainable parameters. While this paper provides generalization bounds in which each convolutional filter contributes only once to the bound, it does not hold when different filters are used for different patches, even if their norms are the same. In short, their analysis treats different patches as 'datapoints' in an augmented problem where only one linear function is applied at each layer. If several choices of linear functions (different weights for different patches) are allowed, the capacity of the function class would increase. Although all of these papers offer generalization guarantees for convolutional networks, they base their findings either on the number of trainable parameters or on weight sharing. Notably, none of these studies directly address the question of whether weight sharing is essential for the effective generalization of convolutional networks. Furthermore, none provide empirical evidence to confirm that their bounds are reasonably tight in practical settings. In our previous work [46], we derived generalization bounds using a technique similar to the one employed here. The results in this paper extend those preliminary results to a more general and more detailed formulation.

1.2 Contributions

In this work, we study the generalization guarantees of a broad class of sparse deep neural networks [42], such as convolutional neural networks. Informally, a sparse neural network is a graph of neurons represented as a Directed Acyclic Graph (DAG), where each neuron is a function of a small set of other neurons. We show how a simple modification to the classic norm-based generalization bound of [14] yields significantly tighter bounds for sparse neural networks (e.g., convolutional networks). Unlike previous bounds [33, 32, 47], our analysis demonstrates how to obtain generalization guarantees for sparse networks, without incorporating weight sharing, while having a weak dependence on the actual size of the network. These results suggest that it is possible to obtain good generalization performance with sparse neural networks without relying on weight sharing. Finally, we conduct multiple experiments to evaluate our bounds for overparameterized convolutional neural networks trained on simple classification problems. These experiments show that in these settings, our bound is significantly tighter than many bounds in the literature [14, 33, 32, 47]. As a result, this research provides a better understanding of the pivotal influence of the structure of the network's architecture [30, 34, 2] on its test performance.

2 Problem Setup

We consider the problem of training a model for classification. Formally, the task is defined by a distribution P over samples ( x, y ) ∈ X × Y , where X ⊂ R c 0 × d 0 is the instance space (e.g., images), and Y ⊂ R C is a label space containing the C -dimensional one-hot encodings of the integers 1 , . . . , C . When thinking about the samples as images, we view c 0 as the number of input channels and d 0 as the image size. We consider a hypothesis class F ⊂ { f ′ : X → R C } (e.g., a neural network architecture), where each function f w ∈ F is specified by a vector of parameters w ∈ R N (i.e., trainable parameters). A function f w ∈ F assigns a prediction to an input point x ∈ X , and its performance on the distribution P is measured by the expected error , err P ( f w ) := E ( x,y ) ∼ P [ I [max j = y ( f w ( x i ) j ) ≥ f w ( x i ) y ]] , where I : { True , False } → { 0 , 1 } be the indicator function (i.e., I [ True ] = 1 and vice versa). Since we do not have direct access to the full population distribution P , the goal is to learn a predictor, f w , from some training dataset S = { ( x i , y i ) } i m =1 of independent and identically distributed (i.i.d.) samples drawn from P along with regularization to control f w 's complexity.

̸

2.1 Rademacher Complexities

We examine the generalization abilities of overparameterized neural networks by investigating their Rademacher complexity. This quantity can be used to upper bound the worst-case generalization gap (i.e., the distance between train and test errors) of functions from a certain class. It is defined as the expected performance of the class when averaged over all possible labelings of the data, where the

labels are chosen independently and uniformly at random from the set {± 1 } . In other words, it is the average performance of the function class on random data. For more information, see [48, 49, 50].

C

Definition 2.1 (Rademacher Complexity) . Let F be a set of real-valued functions f w : X → R defined over a set X . Given a fixed sample X ∈ X m , the empirical Rademacher complexity of F is defined as follows: R X ( F ) := 1 m E ξ : ξ ir ∼ U [ {± 1 } ] [ sup f w ∈F ∣ ∣ ∣ ∑ m i =1 ∑ C r =1 ξ ir f w ( x i ) r ∣ ∣ ∣ ] .

̸

In contrast to the Vapnik-Chervonenkis (VC) dimension, the Rademacher complexity has the added advantage that it can be upper bounded based on a finite sample. The Rademacher complexity can be used to upper bound the gap between test and train errors of a certain class of functions [48]. In the following lemma we bound the gap between the test error and the empirical margin error err γ S ( f w ) = 1 m ∑ m i =1 I [max j = y ( f w ( x i ) j ) + γ ≥ f w ( x i ) y ] .

Lemma 2.2. Let P be a distribution over R c 0 d 0 × [ C ] and F ⊂ { f ′ : X → R C } . Let S = { ( x i , y i ) } i m =1 be a dataset of i.i.d. samples selected from P and X = { x i } i m =1 . Then, with probability at least 1 -δ over the selection of S , for any f w ∈ F , we have

The above bound is decomposed into two parts; one is the Rademacher complexity and the second scales as O (1 / √ m ) which is small when m is large. In section 3 we derive norm-based bounds on the Rademacher complexity of sparse networks. This lemma and the rest of the mathematical statements are proven in the appendix.

2.2 Architectures

A neural network architecture can be formally defined using a Directed Acyclic Graph (DAG) G = ( V, E ) . The class of neural networks associated with this architecture is denoted as F G . The set of neurons in the network is given by V = ⋃ L l =0 { z l 1 , . . . , z l d l } , which is organized into L layers. An edge ( z l i , z l -1 j ) ∈ E indicates a connection between a neuron in layer l -1 and a neuron in layer l . The full set of neurons at the layer l th is denoted by v l := ( z l j ) d l j =1 .

Aneural network f w : R c 0 × d 0 → R C takes 'flattened' images x as input, where c 0 is the number of input channels and d 0 is the image dimension represented as a vector. Each neuron z l i : R c 0 × d 0 → R c l computes a vector of size c l (the number of channels in layer l ). To avoid confusion, in our definition, we think of each neuron as a vector of dimension c l . This is analogous to a pixel holding three coordinates of RGB. The set of predecessor neurons of z l i , denoted by pred ( l, i ) , is the set of all j ∈ [ d l -1 ] such that ( z l i , z l -1 j ) ∈ E , and v l i := ( z l j ) j ∈ pred ( l,i ) denotes the set of predecessor neurons of z l i . The network is recursively defined as follows:

where w L ri ∈ R c L -1 , z l i ( x ) := σ ( w l i v l -1 i ( x )) , w l i ∈ R c l × ( c l -1 ·| pred ( l -1 ,i ) | ) is a weight matrix, x = ( z 0 j ( x )) d 0 j =1 , each z 0 j ( x ) is a vector of dimension c 0 representing the j th 'pixel' of x and σ is the ReLU activation function. For simplicity, we denote w L := W L = ( w ri ) r,i .

The degree of sparsity of a neural network can be measured using the degree of the graph, which is defined as the maximum number of predecessors for each neuron. Specifically, the degree of a neural network architecture G is given by: deg ( G ) := max l ∈ [ L ] deg ( G ) l , where deg ( G ) l := max j ∈ [ d l ] | pred ( l, j ) | is the maximal degree of the l th layer.

Convolutional neural networks. A special type of compositionally sparse neural networks is convolutional neural networks. In such networks, each neuron acts upon a set of nearby neurons from the previous layer, using a kernel shared across the neurons of the same layer.

To formally analyze convolutional networks, we consider a broader set of neural network architectures that includes sparse networks with shared weights. Specifically, for an architecture G with | pred ( l, j ) | = k l for all j ∈ [ d l ] , we define the set of neural networks F sh G to consist of all neural

networks f w ∈ F sh G that satisfy the weight sharing property w l := w l j 1 = w l j 2 for all j 1 , j 2 ∈ [ d l ] and l ∈ [ L ] . Convolutional neural networks are essentially sparse neural networks with shared weights and locality (each neuron is a function of a set of nearby neurons of its preceding layer).

Norms of neural networks. As mentioned earlier, previous papers (e.g., [14]) proposed different generalization bounds based on different types of norms for measuring the complexity of fullyconnected networks. One approach that was suggested by [14] is to use the product of the norms of the weight matrices given by ˜ ρ ( w ) := ∏ L l =1 ∥ W l ∥ F .

In this work, we derive generalization bounds based on the product of the maximal norms of the kernel matrices across layers, defined as: ρ ( w ) := ∥ w L ∥ F · ∏ L -1 l =1 max j ∈ [ d l ] ∥ w l j ∥ F , where ∥ · ∥ F and is the Frobenius norm. For a convolutional neural network, we have a simplified form of ρ ( w ) = ∏ L l =1 ∥ w l ∥ F , due to the weight sharing property. This quantity is significantly smaller than the quantity ˜ ρ ( w ) = ∥ w L ∥ F · ∏ L -1 l =1 √ ∑ d l j =1 ∥ w l j ∥ 2 F used by [14]. For instance, when weight sharing is applied, we can see that ˜ ρ ( w ) = ρ ( w ) · √ ∏ L -1 l =1 d l which is significantly larger than ρ ( w ) .

Classes of interest. In the next section, we study the Rademacher complexity of classes of compositionally sparse neural networks that are bounded in norm. We focus on two classes: F G,ρ := { f w ∈ F G | ρ ( w ) ≤ ρ } and F sh G,ρ := { f w ∈ F sh G | ρ ( w ) ≤ ρ } , where G is a compositionally sparse neural network architecture and ρ is a bound on the norm of the network parameters.

3 Theoretical Results

In this section, we introduce our main theoretical results. The following theorem provides a bound on the Rademacher complexity of the class F G,ρ of networks of architecture G of norm ≤ ρ .

Proposition 3.1. Let G be a neural network architecture of depth L and let ρ > 0 . Let X = { x i } i m =1 be a set of samples. Then,

where the maximum is taken over j 0 , j 1 , . . . , j L , such that, j l -1 ∈ pred ( l, j l ) for all l ∈ [ L ] .

The proof for this theorem builds upon the proof of Theorem 1 in [14]. A sketch of the proof is presented in Section 3.1. As we show next, by combining Lemma 2.2 and Proposition 3.1 we can obtain an upper bound on the test error of compositionally sparse neural networks.

Theorem 3.2. Let P be a distribution over R c 0 d 0 ×{± 1 } . Let S = { ( x i , y i ) } i m =1 be a dataset of i.i.d. samples selected from P . Then, with probability at least 1 -δ over the selection of S , for any f w ∈ F G ,

where the maximum is taken over j 0 , . . . , j L , such that, j l -1 ∈ pred ( l, j l ) for all l ∈ [ L ] .

The theorem above provides a generalization bound for neural networks of a given architecture G . To understand this bound, we first analyze the term ∆ := max j 0 ,...,j L ∏ L -1 l =1 | pred ( l, j l ) | · ∑ m i =1 ∥ z 0 j 0 ( x i ) ∥ 2 2 . We consider a setting where d 0 = 2 L , c l = 1 and each neuron takes two neurons as input, k l := | pred ( l, j ) | = 2 for all l ∈ [ L ] and j ∈ [ d l ] . In particular, ∏ L -1 l =1 k l = 2 L -1 and z 0 j ( x i ) is the j th pixel of x i . Therefore, we have ∆ = d 0 2 · max j 0 ∑ m i =1 ∥ z 0 j 0 ( x i ) ∥ 2 2 . We

note that in the worst-case, when all of the norm ∥ x i ∥ is concentrated in the first pixel z 0 1 ( x i ) , we have ∆ = d 0 2 · ∑ m i =1 ∥ x i ∥ 2 . But in practice, the norms of the pixels z 0 j ( x i ) are typically more evenly distributed. For instance, we can assume that the pixels are β -balanced. Meaning, ∀ i ∈ [ m ] : max j ∈ [ d 0 ] ∥ z 0 j ( x i ) ∥ 2 ≤ β Avg j ∈ [ d 0 ] [ ∥ z 0 j ( x i ) ∥ 2 ] = β d 0 ∥ x i ∥ 2 (for some constant β > 0 ). Note that the rate β is a property exclusively dependent on the data as it is independent of the architecture and the training process. In particular, we obtain that ∆ ≤ β 2 ∑ m i =1 ∥ x i ∥ 2 . In addition, we note that the second term in the bound is typically smaller than the first term as it scales with √ log( ρ ( w )) instead of ρ ( w ) and has no dependence on the size of the network. Therefore, in this case, our bound can be simplified to O ( ρ ( w ) √ m √ Lβ Avg m i =1 [ ∥ x i ∥ 2 ]) .

Bounds for convolutional networks. As previously stated in section 2, convolutional neural networks utilize weight sharing across neurons in each layer, with each neuron in the l th layer having k l input neurons (each of dimension c l -1 ). The norm of the network is calculated as ρ ( w ) = ∏ L l =1 ∥ w l ∥ F , and the degree at each layer is simply the kernel size deg( G ) l = k l . This results in a simplified version of the theorem.

Corollary 3.3 (Rademacher complexity of convolutional networks) . Let G be a neural network architecture of depth L and let ρ > 0 . Let X = { x i } i m =1 be a set of samples. Then,

where k l denotes the kernel size in the l 'th layer.

Comparison with the bound of [14]. The result in Corollary 3.3 is a refined version of the analysis in [14] for the specific case of convolutional networks. Theorem 1 in [14] can of course be applied to convolutional networks by treating their convolutional layers as fully-connected layers. However, this approach yields a substantially worse bound compared to the one proposed in Corollary 3.3.

Consider a convolutional neural network G . The l th convolutional layer takes the concatenation of ( σ ( z l 1 ) , . . . , σ ( z l d l )) as input and returns ( z l +1 1 , . . . , z l +1 d l +1 ) as its output. Each z l +1 j is computed as follows z l +1 j = w l +1 σ ( v l j ( x )) . Therefore, the matrix W l +1 associated with the convolutional layer contains d l +1 copies of w l +1 and its Frobenius norm is therefore √ d l +1 · ∥ w l +1 ∥ F . In particular, by applying Theorem 1 in [14], we obtain a bound that scales as O ( ρ m √ L ∏ L -1 l =1 d l · ∑ m i =1 ∥ x i ∥ 2 ) . On the other hand, we have ∏ L -1 l =1 k l ≤ ∏ L -1 l =1 d l and ∥ z 0 j ( x i ) ∥ ≤ ∥ x i ∥ . Therefore, our bound is always smaller than O ( ρ m √ L ∏ L -1 l =1 d l · ∑ m i =1 ∥ x i ∥ 2 ) , which is the bound we obtained with [14]. In particular, if each convolutional layer has k l = 2 with no overlaps and d 0 = 2 L , then, d l = 2 L -l and the bound of [14] would scale as O ( ρ √ m √ L 2 0 . 5 L ( L -1) · Avg i m =1 [ ∥ x i ∥ 2 ] ) . On the other hand, in our bound we have the term ∏ L -1 l =1 k l = 2 L -1 , and therefore, our bound scales as O ( ρ √ m √ L 2 L Avg i m =1 [ ∥ x i ∥ 2 ] ) which is significantly smaller. Finally, as we discussed earlier, if the norms of the pixels of each sample x are β -balanced (for some constant β > 0 ), our bound scales as O ( ρ √ m √ L Avg i m =1 [ ∥ x i ∥ 2 ] ) which is smaller by a factor of 2 0 . 25 L ( L -1) than the bound of [14].

Comparison with the bound of [33]. A recent paper [33] introduced generalization bounds for convolutional networks based on parameter counting. This bound roughly scales like O (√ N ( ∑ L l =1 ∥ w l ∥ 2 +log(1 /γ ))+log(1 /δ ) m ) , where γ is a margin (typically smaller than 1 ), and N is the number of trainable parameters (taking weight sharing into account by counting each parameter of convolutional filters only once). While these bounds provide improved generalization guarantees when reusing parameters, it scales as Ω( √ N/m ) which is very large in practice. For example, the standard ResNet-50 architecture has approximately N = 23 M trainable parameters while the MNIST dataset has only m = 50000 training samples.

Comparison with the bounds of [32] and [47]. Recent papers [32] introduced generalization bound for convolutional networks based on covering numbers and weight sharing. For example,

the bounds in Theorem 17 of [32] roughly scale as O   L ∏ l =1 ∥ W l ∥ 2 √ m E ( w ) 1 /α · I α   , where E ( w ) =

corresponding to the linear operator associated with the l th convolutional layer, w i, : is the i th row of a matrix w , α is either 2 or 2 / 3 , I α = L if α = 2 and I α = 1 o.w. and u l are 'reference' matrices of the same dimensions as w l .

In general, neither our bounds nor those in [32] and [47] are inherently superior; with each being better in different cases. The main difference between their bounds and our bound, is that while their bounds include both multiplicative complexity term and additive complexity term, our bound includes only a multiplcative complexity term. For instance, the bound in Theorem 17 in [32] features both ∏ L l =1 ∥ w l ∥ 2 and E ( w ) , whereas our bound exclusively contains the multiplicative term ρ ( w ) = ∏ L l =1 ∥ w l ∥ F . For certain cases, this works in favor of our bound, but at the same time, our term ∏ L l =1 ∥ w l ∥ F is comparably larger than ∏ L l =1 ∥ w l ∥ 2 due to the smaller norms used in the former. As an example of a case where our bound is superior, consider the case described after Theorem 3.2, where each convolutional layer operates on non-overlapping patches of size 2 and the channel dimension is 1 at each layer. We choose u l = 0 for all l ∈ [ L -1] (which is a standard choice of reference matrices). We notice that ∥ W l ∥ 2 = ∥ w l ∥ 2 = ∥ w l ∥ F since W l is a block matrix and w l is a vector. In addition, for any matrix A , we have rank ( A ) ≥ ∥ A ⊤ ∥ 2 , 1 ∥ A ∥ 2 ≥ ∥ A ∥ F ∥ A ∥ 2 ≥ 1 and rank ( A ) ≥ ∥ A ∥ 2 max i ∥ A i, : ∥ 2 ≥ 1 (see [17]). Therefore, the bound in [32] scales as at least ∏ L l =1 ∥ w l ∥ 2 √ m · L 3 / 2 , while ∏ L l =1 ∥ w l ∥ 2 √ √

Vacuous bounds? A uniform convergence bound for a class F is an upper bound on the generalization gap that uniformly holds for all f ∈ F , i.e., sup f ∈F | err P ( f ) -err S ( f ) | ≤ ϵ ( m, F ) (typically tends to 0 as m → ∞ ). The Rademacher complexity bound in Lemma 2.2 is a form of uniform convergence bound. The issue with these bounds is that in interpolation regimes, where there exists a function f ∈ F G that fits any labeling of the samples { x i } i m =1 , uniform convergence bounds are provably vacuous.

While the derivation of the bound in Theorem 3.2 follows the application of Rademacher complexities, we emphasize that it is not a uniform convergence bound and is not necessarily vacuous. Throughout the proof, we sliced the class F G into subsets F G,ρ = { f ∈ F G | ρ ( w ) ≤ ρ } (for ρ ∈ N ) and applied Lemma 2.2 for each of these subsets. This approach yields a bound that is proportional to O ( ρ/ √ m ) for each of the slices F G,ρ . We then apply a union bound to combine all of them to obtain a bound that scales as O ( ρ ( w ) / √ m ) . This does not give a uniform convergence bound across all members of F G , since the bound is individualized for each member f w ∈ F G based on the norm ρ ( w ) . For example, for w = 0 , the bound will be 0 which is non-vacuous.

When the learning algorithm minimizes ρ ( w ) and the minimal norm required to fit the training labels is small, a tight bound can be achieved with a network that perfectly fits the training data. For example, suppose we have a dataset S = { ( x i , y i ) } i m =1 , a target function y ( x ) = ⟨ w ∗ , x ⟩ , and a hypothesis class F = {⟨ w,x ⟩ | w ∈ R d } . A classic VC-theory bound scales as O ( √ d/m ) , which is vacuous when d ≫ m . However, a norm-based bound scales as O ( ∥ w ∥ / √ m ) , which is non-vacuous for any ∥ w ∥ ≤ ∥ w ∗ ∥ (as long as m > ∥ w ∗ ∥ 2 ). In addition, the function y can be realized by {⟨ w,x ⟩ | ∥ w ∥ ≤ ∥ w ∗ ∥} ⊂ F . Specifically, for smaller ∥ w ∗ ∥ , we need fewer samples to ensure that the bound is non-vacuous for a minimal norm model that perfectly fits the training data.

3.1 Proof Sketch

We propose an extension to a well-established method for bounding the Rademacher complexity of norm-bounded deep networks. This approach, originally developed by [13], utilizes a 'peeling' argument, where the complexity bound for a depth L network is reduced to a complexity bound for a depth L -1 network and applied repeatedly. Specifically, the l th step bounds the complexity bound for depth l by using the product of the complexity bound for depth l -1 and the norm of the l th layer. By the end of this process, we obtain a bound that depends on the term E ξ g ( | ∑ m i =1 ξ i x i | )

( g ( x ) = x in [13] and g = exp in [14]), which can be further bounded using max x ∈ X ∥ x ∥ 2 . The final bound scales with ˜ ρ ( w ) · max x ∈ X ∥ x ∥ . Our extension further improves the tightness of these bounds by incorporating additional information about the network's sparsity.

To bound R X ( F G,ρ ) using ρ ( w ) , we notice that each neuron operates on a small subset of the neurons from the previous layer. Therefore, we can bound the contribution of a certain constituent function z l j ( x ) = w l j v l -1 j ( x ) in the network using the norm ∥ w l j ∥ F and the complexity of v l -1 j ( x ) instead of the full layer v l -1 ( x ) . To explain this process, we provide a proof sketch of Proposition 3.1 for convolutional networks G = ( V, E ) with non-overlapping patches. For simplicity, we assume that d 0 = 2 L , c l = 1 , and the strides and kernel sizes at each layer are k = 2 . In particular, the network f w can be represented as a binary tree, where the output neuron is computed as f w ( x ) = z L j 0 ( x ) = w L · σ ( z L -1 1 ( x ) , z L -1 2 ( x )) , z L -1 1 ( x ) = w L -1 · σ ( z L -2 1 ( x ) , z L -2 2 ( x )) and z L -1 2 ( x ) = w L -1 · σ ( z L -2 3 ( x ) , z L -2 4 ( x )) and so on. Similar to [14], we first bound the Rademacher complexity using Jensen's inequality,

where λ > 0 is an arbitrary parameter. As a next step, we rewrite right-hand side as follows:

We notice that each z L -1 j ( x ) is itself a depth L -1 binary-tree neural network. Therefore, intuitively we would like to apply the same argument L -1 more times. However, in contrast to the above, the networks σ ( z L -1 1 ( x )) = σ ( w L -1 ( z L -2 1 ( x ) , z L -2 2 ( x ))) and σ ( z L -1 2 ( x )) = σ ( w L -1 ( z L -2 3 ( x ) , z L -2 4 ( x ))) end with a ReLU activation. To address this issue, [13, 14] proposed a 'peeling process' based on Equation 4.20 in [51] that can be used to bound terms of the form E ξ sup f ′ ∈F ′ ,W : ∥ W ∥ F ≤ R exp ( α ∥ ∥ ∑ m i =1 ξ i · σ ( Wf ′ ( x i )) ∥ ∥ ) . However, this bound is not directly applicable when there is a sum inside the square root, as in equation 3 which includes a sum over j = 1 , 2 . Therefore, a modified peeling lemma is required to deal with this case.

Lemma 3.4 (Peeling Lemma) . Let σ be a 1-Lipschitz, positive-homogeneous activation function which is applied element-wise (such as the ReLU). Then for any class of vector-valued functions F ⊂ { f = ( f 1 , . . . , f q ) | ∀ j ∈ [ q ] : f j : R d → R p } , and any convex and monotonically increasing function g : R → [0 , ∞ ) ,

By applying this lemma L -1 times with g = exp and f representing the neurons preceding a certain neuron at a certain layer, we can bound the term in equation 3 as follows:

where the last inequality follows from standard concentration bounds (see the proof for details). Finally, by equation 2 and properly adjusting λ , we can finally bound R X ( F G,ρ ) .

4 Experiments

In this section, we empirically evaluate the generalization bounds derived in section 3. In each experiment, we compare our bound with alternative bounds from the literature. We focus on simple

0

100

200

300

400

500

Epoch

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization gap

0

100

200

300

400

500

Epoch

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization gap

0

100

200

300

400

500

Epoch

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization gap

CONV-3-200

CONV-3-600

CONV-3-1000

200

400

600

800

1000

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

200

400

600

800

1000

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

200

400

600

800

1000

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

CONV-4-

H

CONV-6-

H

CONV-8-

H

convolutional neural networks trained on MNIST and investigate the behavior of the bound when varying different hyperparameters. Each experiment was averaged across five runs. For additional experimental details, please refer to the appendix.

Network architecture. We used convolutional networks with L layers and H channels per layer denoted by CONVL -H . The networks consist of a stack of L 2 × 2 convolutional layers with a stride of 1, 0 padding, and H output channels, utilizing ReLU activations followed by a fully-connected layer. The overall number of trainable parameters is at least 4 H +4( L -1) H 2 .

Optimization process. Each model was trained using SGD for MSE-loss minimization between the logits of the network and the one-hot encodings of the training labels. We applied weight normalization [52] to all trainable layers, except for the last one, which is left un-normalized. In order to regularize the weight parameters, we used weight decay for each one of the layers of the network with the same regularization parameter λ > 0 . To train each model, we used an initial learning rate of µ = 0 . 01 that is decayed by a factor of 0 . 1 at epochs 60, 100, 300, batch size 32, momentum of 0 . 9 , and λ = 3e -3 by default.

Experiments. Weconducted several experiments to compare our bound to alternative bounds from the literature, when applied for neural networks trained on the MNIST dataset for classification. Throughout these experiments, we compared our bound to the one in Theorem 1 of [14], the third inequality in Theorem 2.1 of [33], Theorem 16 of [32], and Theorem 3.5 in [47] (explicitly as mentioned in their Table 3). To compute the bound in [14] for multi-class classification, we adopted a modified version based on the technique we utilized in the proof of Proposition 3.1, which allows us to extend these bounds for multi-class classification. Since [33] did not provide an explicit value for their coefficient C , we assumed it to be 1 . In the first experiment we trained three models of different widths for MNIST classification. As can be seen in Figure 1, our bound is significantly smaller than the alternative bounds and is surprisingly close to 1, indicating its tightness. In the second experiment, we compared our bound with alternative bounds from previous literature, focusing on convolutional neural networks with varying depths and widths. As can be seen in Figure 2, even as

4

5

6

7

8

9

Depth

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

4

5

6

7

8

9

Depth

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

4

5

6

7

8

9

Depth

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

CONV-

L

-

200

CONV-

L

-

600

CONV-

L

-

1000

8e-4

9e-4

1e-3

2e-3

3e-3

4e-3

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

8e-4

9e-4

1e-3

2e-3

3e-3

4e-3

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

8e-4

9e-4

1e-3

2e-3

3e-3

4e-3

2

17

2

12

2

7

2

2

2 3

2 8

2 13

2 18

2 23

2 28

2 33

2 38

2 0

Ledent et al. 2021

Golowich et al. 2020

Long et al. 2020

Our bound

Train error

Test error

Generalization Gap

H

= 200

H

= 600

H

= 1000

networks become highly overparameterized at large widths, the width does not appear to influence the results of any of the bounds. In Figure 3, we showcase the results for convolutional networks of different depths. Our bound increases exponentially with depth but rises more slowly than the bound from [14]. As an ablation study, in in Figure 4, we compared our bound with alternative bounds, this time varying the regularization coefficient λ . It is evident that our bound and some alternative bounds decrease concurrently with the generalization gap of the neural network, as desired outcome.

5 Conclusions

We studied the question of why certain deep learning architectures, such as convolutional networks and MLP-mixers, perform better than others on real-world datasets. To tackle this question, we derived Rademacher complexity generalization bounds for sparse neural networks, which are orders of magnitude better than a naive application of standard norm-based generalization bounds for fullyconnected networks. In contrast to previous papers [33, 32], our results do not rely on parameter sharing between filters, suggesting that the sparsity of the neural networks is the critical component to their success. This sheds new light on the central question of why certain architectures perform so well and suggests that sparsity may be a key factor in their success. Even though our bounds are not practical in general, our experiments show that they are quite tight for simple classification problems, unlike other bounds based on parameter counting and norm-based bounds for fully-connected networks.

References

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition , CVPR '16, pages 770-778. IEEE, June 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, jun 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529:484-489, 2016.

Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective, 2019. cite arxiv:1902.01724.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.

Vladimir N. Vapnik. Statistical Learning Theory . Wiley-Interscience, 1998.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203 - 248, 2021.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Peter Gr¨ unwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 1376-1401, Paris, France, 03-06 Jul 2015. PMLR.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA , 9(2):473-504, 05 2020.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. In J. Mach. Learn. Res. , 2001.

Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension bounds for piecewise linear neural networks. ArXiv , abs/1703.02930, 2017.

Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems , volume 30. Curran Associates Inc., 2017.

Behnam Neyshabur, Srinadh Bhojanapalli, David A. McAllester, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. ArXiv , abs/1707.09564, 2018.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks, 2019.

Amit Daniely and Elad Granot. Generalization bounds for neural networks via approximate description length. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation, 2019.

Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2019.

Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, and Tuo Zhao. On tighter generalization bound for deep neural networks: Cnns, resnets, and beyond, 2018.

Fengxiang He, Tongliang Liu, and Dacheng Tao. Why resnet works? residuals generalize, 2019.

Vaishnavh Nagarajan and Zico Kolter. Deterministic PAC-bayesian generalization bounds for deep networks via generalizing noise-resilience. In International Conference on Learning Representations , 2019.

Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Nonvacuous generalization bounds at the imagenet scale: a PAC-bayesian compression approach. In International Conference on Learning Representations , 2019.

Sanae Lotfi, Marc Anton Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and Andrew Gordon Wilson. PAC-bayes compression bounds so tight that they can explain generalization. In Advances in Neural Information Processing Systems , volume 35. Curran Associates Inc., 2022.

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks, 2019.

Dominic Richards and Ilja Kuzborskij. Stability & generalisation of gradient descent for shallow neural networks without the neural tangent kernel. In Advances in Neural Information Processing Systems , volume 34. Curran Associates Inc., 2021.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Antoine Ledent, Waleed Mustafa, Yunwen Lei, and Marius Kloft. Norm-based generalisation bounds for deep multi-class convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence , 35(9):8279-8287, May 2021.

Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations , 2020.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 24261-24272. Curran Associates, Inc., 2021.

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems , 2(4):303-314, 1989.

Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks , 4:251-257, 1991.

Hrushikesh N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Comput. , 8(1):164-177, 1996.

Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. NEUROCOMPUTING , 25:81-91, 1999.

Vitaly Maiorov, Ron Meir, and Joel Ratsaby. On the approximation of functional classes equipped with a uniform measure using ridge functions. J. Approx. Theory , 99(1):95-111, 1999.

Vitaly Maiorov. On best approximation by ridge functions. J. Approx. Theory , 99(1), 1999.

Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width, 2017.

Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. When and why are deep networks better than shallow ones? In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence , AAAI'17, page 2343-2349. AAAI Press, 2017.

Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceedings of the National Academy of Sciences , 117(48):30039-30045, 2020.

T. Poggio. Foundations of deep learning: Compositional sparsity of computable functions. CBMM memo 138 , 2022.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations , 2020.

Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research , 6:0024, 2023.

Florian Graf, Sebastian Zeng, Bastian Rieck, Marc Niethammer, and Roland Kwitt. On measuring excess capacity in neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 10164-10178. Curran Associates, Inc., 2022.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, Cambridge, MA, 2 edition, 2018.

Shai Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge eBooks, 2014.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Michel Ledoux and Michel Talagrand. Probability in Banach spaces . Springer Berlin, Heidelberg, 1991.

Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks, 2016.

G´ abor Lugosi St´ ephane Boucheron, Olivier Bousquet. Theory of classification: a survey of some recent advances. ESAIM: Probability and Statistics , 9:323-375, 3 2010.

Tong Zhang. Statistical analysis of some multi-category large margin classification methods. J. Mach. Learn. Res. , 5:1225-1251, dec 2004.

Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007.

Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles, editors, Algorithmic Learning Theory , pages 3-17, Cham, 2016. Springer International Publishing.

St´ ephane Boucheron, G´ abor Lugosi, and Pascal Massart. Concentration Inequalities - A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

6 Limitations

Our work demonstrates how a simple modification to the proof in [14] can yield tighter generalization bounds for convolutional networks. Empirically, we have observed that our bound is reasonably tight in basic learning settings (e.g., training 3-layer neural networks on MNIST), which is promising. However, this bound is anticipated to be generally loose for more complex learning settings. One drawback of our bound is its dependence on depth. For instance, our factor ρ ( w ) represents a product of L weight norms, which could increase as the depth L grows. Additionally, our bound contains a product ∏ L -1 l =1 k l , which increases exponentially with depth. As demonstrated in section 3, this indicates that the bound is likely reasonable only when L depends logarithmically on the input's dimension. Consequently, our bound may not be suitable for very deep convolutional networks. Finally, building upon the analysis in [14], our bounds are specific to a certain class of ReLU neural networks and do not readily extend to architectures with different activation functions.

7 Broader Impact

This paper delves into the theoretical aspects of deep learning, specifically examining the reasons behind the strong generalization capabilities of sparse neural networks. Such insights may enrich our understanding of the success of deep learning and specific architectures, such as convolutional networks [30] and MLP-mixers [34]. However, since this is a theoretical study, we do not anticipate any direct negative societal impacts stemming from our research.

8 Additional Experimental Details

Compute. Each of the runs was done using a single GPU for at most 20 hours on a computing cluster with several available GPU types (e.g., GeForce RTX 2080, GeForce RTX 2080 Ti, Quadro RTX 6000, Tesla V-100, GeForce RTX A6000, A100, and GeForce GTX 1080 Ti).

Calculating our bound. Before discussing the calculation of the bound, let us first define how we model a convolutional neural network within the framework outlined in section 2.

Wefocus on ReLU convolutional networks which comprise a fully-connected layer on top of a series of convolutional layers. The l th convolutional layer possesses a kernel size of k l . Each neuron in this l th layer accepts k l neurons from its predecessor layer as input (therefore its degree is deg ( G ) l = k l ), each with dimension c l -1 , and outputs a single output node of dimension c l . The top linear layer L following the series of convolutional layers receives a set of k L nodes as input (this represents the image resultant from applying multiple convolutional layers), where each node is of the output channel dimension of the previous layer. The output of this layer is of dimension k .

The bound derived in Theorem 3.2 of the main text can be written as follows:

To compute this bound, we set δ = 0 . 001 (the bound holds uniformly with probability 1 -δ ) and γ = 1 . The norm ρ ( w ) is calculated as the product of the norms of each one of the network's layers ∥ w l ∥ F . The norm of each of these layers is simply the Frobenius norm of the kernel tensor of the layer. For the last fully-connected layer we calculate the norm of the full weight matrix. For the l th convolutional layer, k l = | pred ( l, j l ) | = deg ( G ) l is the kernel's size. For instance, if the layer is a 3 × 3 or 2 × 2 convolutional layer, the size is 9 or 4 , respectively. It is worth noting that the bound is independent of the size of the image k L that the fully-connected layer receives as input.

Finally, to calculate max j 1 ,...,j L ∑ m i =1 ∥ z 0 j 0 ( x i ) ∥ 2 2 , with j l -1 ∈ pred ( l, j l ) . It is important to note that this term is bounded by max j 0 ∑ m i =1 ∥ z 0 j 0 ( x i ) ∥ 2 2 , where the maximum is taken over all input patches. To calculate this quantity, we sum the squared norms of the j 0 th pixel across all samples and return the maximum value across all possible j 0 choices.

9 Proofs

Lemma 2.2. Let P be a distribution over R c 0 d 0 × [ C ] and F ⊂ { f ′ : X → R C } . Let S = { ( x i , y i ) } i m =1 be a dataset of i.i.d. samples selected from P and X = { x i } i m =1 . Then, with probability at least 1 -δ over the selection of S , for any f w ∈ F , we have

Proof. By Lemma 3.1 in [17], with probability at least 1 -δ over the selection of S , for any f w ∈ F , we have

̸

where F | γ = { ( x, y ) ↦→ ℓ γ ( -M ( f w ( x ) , y )) | f w ∈ F} , ℓ γ ( u, y ) := min(1 , max(0 , 1 + M ( u,y ) γ )) is the ramp loss function and M ( u, j ) = u j -max i = j u i (these quantities are standard [53, 54, 55]). Next, by Lemma A.3 in [17], ℓ γ ( M ( · , r )) is a (2 /γ ) -Lipschitz function. Hence, by Corollary 1 in [56],

which completes the proof.

Lemma 3.4 (Peeling Lemma) . Let σ be a 1-Lipschitz, positive-homogeneous activation function which is applied element-wise (such as the ReLU). Then for any class of vector-valued functions F ⊂ { f = ( f 1 , . . . , f q ) | ∀ j ∈ [ q ] : f j : R d → R p } , and any convex and monotonically increasing function g : R → [0 , ∞ ) ,

Proof. Let W ∈ R h × p be a matrix and let w 1 , . . . , w h be the rows of the matrix W . Define a function Q j ( v ) := ( ∑ m i =1 ξ i · σ ( v ⊤ ∥ v ∥ 2 f j ( x i )) ) 2 taking a vector v ∈ R p . We notice that

For any w j 1 , . . . , w jh , we have

̸

which is obtained for ˆ w j 1 , . . . , ˆ w jh , where ˆ w ji = 0 for all i = r ∗ and ˆ w jr ∗ of norm R for some r ∗ ∈ [ h ] . Together with the fact that g is a monotonically increasing function, we obtain

where the last equality follows from the symmetry in the distribution of the ξ i random variables. By Equation 4.20 in [51] and Cauchy-Schwartz, we have the following:

Proposition 3.1. Let G be a neural network architecture of depth L and let ρ > 0 . Let X = { x i } i m =1 be a set of samples. Then,

where the maximum is taken over j 0 , j 1 , . . . , j L , such that, j l -1 ∈ pred ( l, j l ) for all l ∈ [ L ] .

Proof. We denote by f w an arbitrary member of F G,ρ and w l j l the weights of the j l th neuron of the l th layer. Due to the homogeneity of the ReLU function, each function f w ∈ F G,ρ can be rewritten as f ˆ w , where ˆ w L := ρ w L ∥ w L ∥ F and for all l < L and j l ∈ [ d l ] , ˆ w l j l := w l j l max j ∈ [ d l ] ∥ w l j ∥ F . In particular, we have F G,ρ ⊂ ˆ F G,ρ := { f w | ∥ w L ∥ F ≤ ρ and ∀ i < L, j l ∈ [ d l ] : ∥ w l j l ∥ F ≤ 1 } since the ReLU function is homogeneous.

We first consider that

where the last inequality follows from moving the norm of ˆ w L to the vector ˆ w L rj L -1 for maximizing the term ∑ j L -1 ∑ C r =1 ( ˆ w L rj L -1 ) ⊤ ∑ m i =1 ξ ir z L -1 j L -1 ( x i ) and applying the Cauchy-Schwartz inequality. Next, we apply Jensen's inequality,

where the supremum is taken over the weights ˆ w l j l ( l ∈ [ L ] , j l ∈ [ d l ] ) that are described above. Next, we use Lemma 3.4,

where the supremum is taken over the parameters of f ˆ w and j 1 ∈ pred ( L, j 0 ) . By applying this process recursively L times, we obtain the following inequality,

where the maximum is taken over j 0 , . . . , j L , such that, j l -1 ∈ pred ( l, j l ) and r ∈ [ C ] . We notice that

Following the proof of Theorem 1 in [14], by applying Jensen's inequality and Theorem 6.2 in [57] we obtain that for any α > 0 ,

Hence, by combining equations 5-7 with α = λ · √ ∏ L -1 l =1 | pred ( l, j l ) | , we obtain that

Theorem 3.2. Let P be a distribution over R c 0 d 0 ×{± 1 } . Let S = { ( x i , y i ) } i m =1 be a dataset of i.i.d. samples selected from P . Then, with probability at least 1 -δ over the selection of S , for any f w ∈ F G ,

where the maximum is taken over j 0 , . . . , j L , such that, j l -1 ∈ pred ( l, j l ) for all l ∈ [ L ] .

Proof. Let t ∈ N ∪{ 0 } and G t = F G,t . By Lemma 2.2, with probability at least 1 -δ t ( t +1) , for any function f w ∈ G t , we have

By Proposition 3.1 in the main text, we have

because of the union bound over all t ∈ N , equation 1 holds uniformly for all t ∈ N and f w ∈ G t with probability at least 1 -δ . For each f w with norm ρ ( w ) we then apply the bound with t = ⌈ ρ ( w ) ⌉ since f w ∈ G t , we have

which proves the desired bound.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the derivation of generalization bounds for sparsely connected neural networks. The authors extend the peeling arguments to establish a bound on the Rademacher complexity of these networks based on the number of connections between each neuron and its predecessors and norms of the network weights. The proposed bound is then applied to the specific case of Convolutional Neural Networks (CNNs), and the authors validate their bounds through experimental evaluations.\n\nSTRENGTHS:\n1) The paper demonstrates a clear and well-structured writing style.\n2) The related work is thoroughly cited, indicating a comprehensive understanding of the existing literature.\n3) The extended approach presented in the paper is interesting.\n\nWEAKNESSES:\n1) The provided bounds exhibit exponential scaling in the network depth, even when disregarding the exponential dependence of $\\rho(w)$ on depth. \n\n2) The nature of the bound shares similarities with parameter counting bounds, as it explicitly relies on the number of connections between each neuron and its predecessors (e.g., filter size in CNNs). This explicit dependence remains present even if the learned network deviates minimally from its initialization.\n\n3) In comparisons of related work, the authors use unrealistic cases to demonstrate the superiority of their approach. For instance,  input dimension is 2^L, filter size =2.\n\nQUESTIONS:\n1) Is it possible to adapt the proof technique to mitigate or eliminate the exponential dependence on $\\rho(w)$ (e.g.,  Lipschitz augmentation techniques mentioned in [1])?\n\n2) Can the explicit dependence on the filter size (number of connections) be replaced with an implicit dependence, for example, through incorporating weight norms? \n\n[1] Wei, C., & Ma, T. (2019). Data-dependent sample complexity of deep neural networks via Lipschitz augmentation. Advances in Neural Information Processing Systems, 32.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a bound on the Rademacher complexity of sparse neural networks, which include in particular convolutional neural networks. This bound improves over previously-known bounds for CNNs, and applies more broadly since it does not assume weight sharing, but only sparsity. The bounds are illustrated by numerical experiments.\n\nSTRENGTHS:\nThe paper tackles the important question of the understanding of the advantages of the convolution structure from a statistical point of view. The proposed bounds improve over known bounds by leveraging modern proof techniques for the Rademacher complexity of deep neural networks. The paper is well-written and easy to follow. I checked the proofs partially, and the mathematical details are globally well-written as well.\n\nWEAKNESSES:\nI am not convinced by the numerical experiments as they are, because I do not think they back the claim of the authors on the tightness of their generalization bound. There are several orders of magnitude between the bound and the reported generalization gap. Of course, I am well aware of the difficulty of decreasing this spread, but I think it would be interesting to complement the experiments to prove the point by\n+ reporting the value of other generalization bounds in the same setting, in order to illustrate that the proposed bound is indeed tighter than others. \n+ varying the amount of regularization (I guess, the lambda parameter) described in lines 334-342. If rho(w) does indeed play a role in the generalization power of the neural network, I think you should observe a negative correlation between the regularization strength and the generalization gap. This would be more interesting than just reporting the value of rho(w) as in Figure 1, since this value in itself does not have a strong meaning.\n\nQUESTIONS:\n+ Line 124 (and elsewhere in the paper): you mention that you study the overparametrized regime, but your bounds hold independently of the under/overparameterization. I suggest to rephrase to avoid confusion.\n+ Lines 126-129: the interpretation of Rademacher complexity is misleading. It is not the expected performance of the class, but of the worse case error in the class. This is very different.\n+ Line 161: shouldn’t it be v_i^l and pred(l, i) instead of v_i^{l-1} and pred(l-1,i)? To fit with the definition of v_i^l two lines above.\n+ Line 197 and 205: I find the numbering of j_1 … j_L extremely confusing, I think it would be much more intuitive to reverse the order, in order to make pred(l, j_l) appear in the formula instead of pred(l, j_{L-l}). There is a related issue in the proof, line 134 of the Appendix, I believe it should be j_{L-l+1} instead of j_{l+1}. Also, an explanation on what’s going on (and a drawing?) would be most welcome. You’re basically enumerating the paths in the graph from the input to the output, right?\n+ Lines 261 to 282: ok, but I believe these explanations are not specific to your bound, but are true for any norm-based bound. This should be written more explicitely.\n+ Line 274: I don’t think the bound of Theorem 3.2 is equal to 0 if rho(w)=0? Could you elaborate on this?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an adaptation of Radamacher complexity analysis for deep neural networks specifically suited to those that are compositionally sparse, as in convolutional architectures. The authors characterize the specific model parametrization by a directed acyclic graph, and provide an upper bound to the Radamacher complexity of neurons that have only a few parents. By adapting the required lemmas to these cases, the authors provide a bound on the generalization gap that scales better with the norm of the weights for convolutional networks. Lastly, the authors demonstrate their bounds in practice for a couple of cases (CIFAR and MNIST).\n\nSTRENGTHS:\n- The majority of the generalization bounds for neural network does not make explicit use of the structure in convolutional filters, and this paper seeks to fill this important gap\n- The numerical evaluation of their bounds is highly appreciated\n- The writing and presentation are clear\n\nWEAKNESSES:\n- To my understanding, some claims are overstated or incorrect (see below).\n- Some further numerical evaluation would support their claims better (see below).\n\nQUESTIONS:\nI liked this paper, and enjoyed reading it very much. \n\n1. My main concerns regard the authors depiction of the scaling of their bound with the depth of the network, L: when discussing the implications of theorem 3.2 (main result), the authors conclude that their bound is $\\mathcal O( \\rho \\sqrt{ L \\beta / m})$, and they therefore comment that their bound has a mild dependency on depth $O(\\sqrt{L})$. While I might be missing something, to my understanding this is incorrect for two independent reasons: \n\n    a) the bound on the network norm, $\\rho$, scales exponentially with $L$ in general (line 185). It is true that we don’t know what the correct scaling for this norm is, but since this norm is the product of Frobenius norms (of the filters), the only way for $\\rho$ not to grow exponentially with $L$ is for these norms to decrease exponentially with $L$. Naturally, this is not a reasonable assumption (unless the authors can demonstrate this numerically). Their bound, like all other works based on Radamacher complexities, have an exponential dependence on L.\n\n    b) on line 212, the authors explain that for some \"constant\" $\\beta$, one can upper bound $\\prod^L k_l max_j \\sum_m ||z_j(x_i)||^2 \\leq \\beta \\sum_m ||x_i||^2$. I do not believe it is possible for this upper bound to hold for a constant beta (i.e., $\\beta$ that is independent of $L$), right? If $L$ increases, then necessarily $\\beta$ should increase (and exponentially so) for the bound to hold.\n    \n    Therefore, by presenting their overall bound as $O( \\rho \\sqrt{ L \\beta / m})$, the dependence on L seems incorrect. I’d be happy to learn what I am missing if I'm incorrect about these points above.\n    \n\n1. The inclusion of numerical results that evaluate the obtained bounds is highly appreciated. While it is true that these bounds seem interesting and beginning to be reasonable, they do not provide non-vacuous bounds as stated in the abstract. I believe this should be rephrased. Indeed, in the conclusions the authors do mention, accurately, that their bounds are not yet practical.\n\n1. I am curious about how the authors choose the value for the network norm, $\\rho(w)$, in order to evaluate their bound. As presented, in Theorem 3.2, this value characterizes the hypothesis class and is fixed (i.e. chosen independent of data). In the experiments, the authors mention that they control the value of this norm via regularization, and obtain trained model $w*$. But then, what value for $\\rho(w)$ is chosen at the end to instantiate the bound? Naturally, the value $\\rho(w*)$ cannot be chosen, as it is data dependent. Alternatively, the authors could extend their results to hold uniformly over the choice of $\\rho$, but this comes at the expense of an additional statistical cost (which I don’t believe the authors are taking into account). Could you clarify?\n\n 1. The authors have done a reasonable job of comparing with existing literature in generalization theory. However, the sensitivity analysis/(modified layer-peeling process) bares resemblance to prior work (such as Singla et al) that studies the Lipschitz constant of convolutional networks. I encourage the authors to cite these works as well.\n\n2. The authors contend that the computed bounds are non-uniform as they rely on the norm-based measure $\\rho(w)$. However, the bounds *are* uniform for all similarly norm-bounded networks. This presents a limitation of the Rademacher analysis as the worst-case deviation between networks with $\\rho(w) < \\rho*$ could be much larger than the generalization gap of the trained predictor. While this phenomena is not specific to this work, I would encourage the authors to include/contextualize this in their discussion of vacuous bounds.\n\n**Smaller observations and typos:**\n\ni) In Eq.(1), the (-0.5) exponent should be (0.5).\n\nii) The numerical results (Figs 1 and 2) seem out of place on Page 5. Please move closer to Sec. 4.\n\niii) \\mathcal{F}_G should be \\mathcal{F}_{G,\\rho} in line 204\n\niv) It’s unclear to me why the authors stress so much that their bounds also hold when the network perfectly fits the data. This is true, but not particularly enlightening given that the networks they use, in practice, do not.\n\nv) “we rewrite right-hand” → “we rewrite the right-hand..”\n\nvi) The sentence “Furthermore, since our bound…” seems incomplete.\n\nvii) In the last sentence of their paper, the authors state that “..our experiments show that they are quite tight for simple classification problems, unlike other bounds based on parameter counting, suggesting that the underlying theory is sound and does not need a basic reformulation.” I have troubles with this, since every paper that states “this theory will never need a reformulation” is wrong. It’s clear that this paper makes some nice contribution to understand generalization in deep networks, but the claim seems an exaggeration (e.g., it suffices to note that the bounds are still vacuous!).\n\nviii) Subjective: The notation z_j^l(x) is a bit unnatural to me, as z_j^l was defined as node in a DAG instead of a function of the input. Maybe a footnote would clarify this.\n\nix) in line 127, the authors colloquially defined the Radamacher complexity of a function class as “the expected performance of the class on random data”. I understand that this is just a colloquial description, but it’s not quite accurate: this notion of complexity measures the *maximal* correlation of any function in the function class to random labeling of the empirical sample. \n\nx) line 145: “om” → “on”?\n\nxi) line 134: “the Rademacher complexity has the added advantage that it is data-dependent and can be measured from finite samples.” This is incorrect (unless the authors can solve the maximization over all f in the function class). What the authors likely meant is that it can be upper bounded based on a finite sample.\n\nLIMITATIONS:\nTheir limitations have received little attention. It would be beneficial to expand on these. For example, their bounds (as they do mention) are not yet practical. What do the authors recognize as the main limitation in their analysis that prevents this from hapening?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a generalization bound for a class of neural networks that consist of sparsely connected neurons. The class includes the class of convolutional neural networks and networks with weight sharing.  \t\n* The generalization bound (Theorem 3.2) is based on Rademacher complexity analysis (Proposition 3.1) that builds mainly on the work of Golowich et al (“Size independent sample complexity…”) and Peeling lemma (Lemma 3.4).  The Peeling lemma from Golowich et al cannot be directly used as there is a sum inside the square root in (3).\n* The bound has linear dependence on the number of layers and depends on the product of the Frobenius norms of the kernel matrices (with the maximum taken per layer).  \n* The dimension dependence of the bound is via the notion of predecessor $\\text{pred}(l,i)$, which capture how the neurons are connected across layers. \n\n* As a conclusion, the authors provide a generalization bound for convolutional neural networks (Corollary 3.3).  \n* The experiments are done for MNIST and CIFAR-5m datasets, and for 4-layer convolutional networks. The main contribution of the paper is the generalization bound for sparse neural networks and its special case for convolutional networks.\n\nTo summarize, the paper considers an interesting setting, and the results provide interesting insights about the impact of neuron connectivity and channel dimensions. However, it seems to me that the authors can do a better job in supporting these claims experimentally, formulate their claims more precisely and report the limitations clearly.\n\nSTRENGTHS:\nThe paper is in general well written (see my comments below). Obtaining non-vacuous generalization bounds for convolutional networks is an important and challenging topic. I find the formalization of DNNs as DAG is quite elegant and can model convolutional networks with their feature maps nicely. It is an interesting fact that the number of channels $H$ do not impact the bound (see Section 4.2 in the supplementary materials).\n\nWEAKNESSES:\n* Even if the predecessor set is minimally chosen to have the cardinality 2, the term $\\prod_{l=1}^L |\\text{pred}(l,j_{L-l})|=2^L$, which means exponential dependence on the number of layers. This issue is circumvented by assuming that the model is $\\beta-$balanced (see Page 6 – line 211-212), and assuming $d_0 = 2^L$. This is a strong assumption, since  $d_0$, which is purely data dependent, becomes architecture dependent. It does not seem to me that this assumption is satisfied in practice, and is just used to remove the exponential dependence on the number of layers. The choice needs more justification. \n* Note that in the plots, the term term $\\prod_{l=1}^L |\\text{pred}(l,j_{L-l})|$ is ignored, and only $\\rho(w)/\\sqrt{m}$ is plotted. Therefore, the claimed tightness and non-vacuity can be questionable. \n* The proof seems to be a straightforward modification of the proof in Golowich et al, where only the Peeling lemma is updated. It seems to me that the technical novelty of the proof is limited, and the contribution lies mainly on the extension to the new case of sparse networks. This is fine, if the authors could provide better experiments. The considered architecture and datasets are simpler than what is typically tried in the generalization literature (see for example [44]; or in [17], AlexNet for cifar10 is considered). The message of the paper can become more compelling with better experiments.\n* The plots only show the correlation between the generalization error and the bound. To show that the true generalization error does indeed scale with $\\prod_{l=1}^L |\\text{pred}(l,j_{L-l})|$, it is good to have additional experiments.\n* I wonder about the message of Figure 2. The training and test errors saturate rapidly, and the plotted bound (red curve) does not seem to capture the correlation with the true generalization error (Figure 2  on supplementary materials seems to be even worse in terms of correlation). This might be because the true generalization error is already too small, and the authors might want to look for better visualizations.\n* There are certain imprecise statements in the text that harm the overall presentation. For example, it is claimed that the bound is “quite tight”, although it is not clear how the tightness is measured and with respect to which baseline this judgment is formed. As another example, in the abstract it is claimed that “these bounds may be orders of magnitude better than standard norm based generalization bounds”, although no numerical comparison with many existing norm based bounds is done. Also the claim about the bound being non-vacuous is not always true based on the numerical results.  Admittedly, these limitations are addressed in the supplementary materials, Section 1. It might be a good idea to transfer this to the main part.\n\nQUESTIONS:\n* I am curious about the connection of this paper with two other works (see below). Arora et al also consider convolutional neural networks, and their analysis applies as well to sparse neural networks. Vardi et al prove a bound on a single hidden layer convolutional neural networks that is dependent on the spectral norm of the kernel, better than the bound in the paper, and the patch overlaps. They also need only Lipschitz  functions, which is also more general than the paper. I think it is useful to include this discussion and add these references to the main paper.\n\n  * Arora, Sanjeev, Rong Ge, Behnam Neyshabur, and Yi Zhang. “Stronger Generalization Bounds for Deep Nets via a Compression Approach.” In International Conference on Machine Learning, 254–63, 2018.\n  * Vardi, Gal, Ohad Shamir, and Nati Srebro. “The Sample Complexity of One-Hidden-Layer Neural Networks.” Advances in Neural Information Processing Systems 35 (2022): 9139–50.\n\n## Comments and suggestions:\n* The paper assumes 1-Lipschitz, positive-homogeneous activation functions (Lemma 3.4). Please clarify this throughout the paper. \n* Adding a visualization of predecessor sets can be useful.\n* I feel that the results on approximation guarantees (page 2) are not related to the generalization error and just motivate the case for sparse networks. I suggest moving it to the introduction.\n* In page 2, it is mentioned that the MSE loss is used for training the network to get tighter bounds. The authors justify this choice by claiming that the weight norms are higher for “the cross-entropy loss which implicitly maximizes the network’s weight norms once the network perfectly fits the training data.” It would be good to add a reference for this, for instance:\n** Daniel Soudry  et al, The Implicit Bias of Gradient Descent on Separable Data, JMLR 2018\n* Lemma 2.2, which is Lemma 5.1, is quite standard. It would be enough to mention it in the supplementary materials. The proof is also quite standard and can be removed in my opinion.\n\nLIMITATIONS:\nThe authors addressed some of the limitations in the supplementary materials, although this can be done more clearly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a generalization bound for sparse neural networks, such as convolutional networks. The bound is based on a Rademacher complexity bound derived with a layer-peeling argument. Empirically, the bound is shown to be non-vacuous for a 3 layer network trained on MNIST.\n\nSTRENGTHS:\nThis is a good paper with a strong contribution. As far as I know, it presents the first generalization bound based on Rademacher complexities that is non-vacuous on MNIST! However, I have some doubts (see weaknesses).\n\nThe paper is well written and easy to follow. The results are put into context and compared with prior work [1,2,3]. (For references see end of § Questions)\n\nWEAKNESSES:\nAs already mentioned, I have doubt if the bound is really vacuous.  \n- The bound is evaluated on MNIST, i.e. a multi-class dataset. However, it seems that the bound only applies to binary classification (see Theorem 3.2). Extending the bound to multi-class settings can be done, for example with results from [5], but this yields an additional factor 10 = number of classes.\n- In the provided source code, the maximum pixel sum is computed on the raw data, whereas training occurs on a normalized version of the training data from the data loader. This omits a factor $1/\\sqrt{0.13} \\approx 2.7$.\n- Overall, this yields a factor of $\\approx 27$ which results in a (comparably small but) vacuous bounds again.\n- To clarify, even if the bound is vacuous, it still contributes a novel and significant contribution.\n\nComparisons with prior work is presented, but I have some additional questions, particularly regarding the scaling with network depth, which appears to be unfavorable in this work. See § Questions.\n\nThroughout the paper it is claimed that the derived bounds are (or may be) *orders of magnitude* smaller than existing ones. However, this is not supported by empirical evidence, as existing bounds are not computed in the experiments. A table which lists numerical values of the bounds from [1,2,3,4] evaluated on the models from the MNIST experiment together with a short discussion would be insightful.\n\nThe bound is limited to homogeneous elementwise activation function (e.g. ReLU) and thus cannot handle some popular network components such as pooling layers. This is a weakness that is shared by all bounds proven with a layer-peeling argument, so this is only a minor concern, as long as it is clearly communicated and pointed out that existing bounds for conv nets are not limited in this regard (e.g. [2,3,4]).\n\nQUESTIONS:\n**Interpretation of the bound.**\n\nCould you add a short paragraph after Proposition 3.1 which describes the different factors in the bound. Specifically, how should the maximum term inside the square root be interpreted? And, how does $z^0_{j_L}(x_i)$ differ from $x_i$?  \n\nMinor question: Its not entirely clear how to read the product. I guess it's $\\big( \\prod_l |\\mathrm{pred}(l,j_{L-1})| \\big) \\sum_i z(x_i)$ and not $\\big( \\prod_l |\\mathrm{pred}(l,j_{L-1})|  \\sum_i z(x_i) \\big)$\n\n**Comparison with prior work \\& scaling with depth.**\n\nThe paragraph starting at line 224 presents a comparison with [1] and shows that under specific architecture constraints, your bound is superior compared to [1] when applied to \nconv nets. What about the other direction? When applying both bounds to an MLP, will both bounds be equal? Is yours still better or is it even worse?\nAt first sight, it seems that your bound has a malign exponential scaling with the network depth via $\\prod_{l=1}^L |\\mathrm{pred}(l,j_{L-1})|$ and will be worse for deep networks.\n\nHow does this exponential scaling affect the comparison with [3]? You state that your bound has a \"*significantly better*\" (explicit) dependence on depth (cf. l.253).\nI also want to remark that one of the strengths of [3] is its benign scaling with the number of classes, which, as far as I see, Corollary 3 does not possess.\n\nI would like to see a comparison with the generalization bound (♣) for conv nets from [4]. Both seem to scale similarly with a product of norms and a log term that is related to the number of parameters. \nMoreover, the capacity reduction experiments in this reference are quite similar to your training procedure with a decomposition into weight direction and magnitude, only that lipschitz norms are controlled instead. Yet empirically the computed bound values in [4] are far from vacuous. I wonder why. Is it due to a strict better scaling of your bound? Or is it due to the training procedure, e.g., the network architecture or the mse loss?\n\nYou state that your bound and [3] \"*cannot be directly compared, with each being better in different cases*\". Could you specify such a setting where your bound is superior to [3] and [4]? As I already mentioned, I would like to see numerical values for the bounds from [1,2,3,4] in the MNIST experiment. Presumably, the MNIST experiment is already a setting where your bound is superior, but empirical evidence is needed.\n\n**Notation.**\nI did not understand the notation at the top of page 8 and in the proof of Proposition 5.3. What is $w^L \\cdot \\sigma (z^{L-1}_1(x), z^{L-1}_1(x))$ Is it \n$w^ \\cdot \\sigma (z^{L-1}_1(x)) + w^L \\cdot \\sigma (z^{L-1}_2(x))$?\n\n**Typo.** \n\"om\" in line 145.\n\n**References.**  \n\n[1] Golowich et al., Size-independent sample complexity of neural networks, COLT 2018  \n[2] Long & Sedghi, Generalization bounds for deep convolutional neural networks, IMCL 2020  \n[3] Ledent et al., Norm-based generalisation bounds for deep multi-class convolutional neural networks. AAAI 2021  \n[4] Graf et al., On Measuring Excess Capacity in Neural Networks, NeurIPS 2021  \n[5] Maurer,  A vector-contraction inequality for Rademacher complexities, ALT 2016\n\nLIMITATIONS:\nThere is a section called \"limitations\" in the supplementary material, but therein, limitations are hardly discussed, only that the proposed bound \"is generally loose and may be vacuous for certain settings\". I think the opposite would have been more honest, i.e., that the bound is non-vacuous for certain settings but should be expected to be vacuous in general.\n\nA limitation of the bound is that it only holds for certain activation functions and thus is not applicable to many network architectures, e.g. AlexNet. This is not sufficiently emphasized in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}