Generalization bounds for deep convolutional neural networks

We prove bounds on the generalization error of convolutional networks . The bounds are in terms of the training loss , the number of parameters , the Lipschitz constant of the loss and the distance from the weights to the initial weights . They are independent of the number of pixels in the input , and the height and width of hidden feature maps . We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network , comparing our bounds with practical generalization gaps . 1 INTRODUCTION . Recently , substantial progress has been made regarding theoretical analysis of the generalization of deep learning models ( see Neyshabur et al. , 2015 ; Zhang et al. , 2016 ; Dziugaite and Roy , 2017 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ; 2018 ; Arora et al. , 2018 ; Golowich et al. , 2018 ; Neyshabur et al. , 2019 ; Wei and Ma , 2019a ; Cao and Gu , 2019 ; Daniely and Granot , 2019 ) . One interesting point that has been explored , with roots in ( Bartlett , 1998 ) , is that even if there are many parameters , the set of models that can be represented using weights with small magnitude is limited enough to provide leverage for induction ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2018 ) . Intuitively , if the weights start small , since the most popular training algorithms make small , incremental updates that get smaller as the training accuracy improves , there is a tendency for these algorithms to produce small weights . ( For some deeper theoretical exploration of implicit bias in deep learning and related settings , see ( Gunasekar et al. , 2017 ; 2018a ; b ; Ma et al. , 2018 ) . ) Even more recently , authors have proved generalization bounds in terms of the distance from the initial setting of the weights instead of the size of the weights ( Dziugaite and Roy , 2017 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2019 ; Nagarajan and Kolter , 2019 ) . This is important because small initial weights may promote vanishing gradients ; it is advisable instead to choose initial weights that maintain a strong but non-exploding signal as computation flows through the network ( see LeCun et al. , 2012 ; Glorot and Bengio , 2010 ; Saxe et al. , 2013 ; He et al. , 2015 ) . A number of recent theoretical analyses have shown that , for a large network initialized in this way , accurate models can be found by traveling a short distance in parameter space ( see Du et al. , 2019b ; a ; Allen-Zhu et al. , 2019 ; Zou et al. , 2018 ; Lee et al. , 2019 ) . Thus , the distance from initialization may be expected to be significantly smaller than the magnitude of the weights . Furthermore , there is theoretical reason to expect that , as the number of parameters increases , the distance from initialization decreases . This motivates generalization bounds in terms of distance from initialization ( Dziugaite and Roy , 2017 ; Bartlett et al. , 2017 ) . Convolutional layers are used in all competitive deep neural network architectures applied to image processing tasks . The most influential generalization analyses in terms of distance from initialization have thus far concentrated on networks with fully connected layers . Since a convolutional layer has an alternative representation as a fully connected layer , these analyses apply in the case of convolutional networks , but , intuitively , the weight-tying employed in the convolutional layer constrains the set of functions computed by the layer . This additional restriction should be expected to aid generalization . ∗Authors are ordered alphabetically . In this paper , we prove new generalization bounds for convolutional networks that take account of this effect . As in earlier analyses for the fully connected case , our bounds are in terms of the distance from the initial weights , and the number of parameters . Additionally , our bounds independent of the number of pixels in the input , or the height and width of the hidden feature maps . Our most general bounds apply to networks including both convolutional and fully connected layers , and , as such , they also apply for purely fully connected networks . In contrast with earlier bounds for settings like the one considered here , our bounds are in terms of a sum over layers of the distance from initialization of the layer . Earlier bounds were in terms of product of these distances which led to an exponential dependency on depth . Our bounds have linear dependency on depth which is more aligned with practical observations . As is often the case for generalization analyses , the central technical lemmas are bounds on covering numbers . Borrowing a technique due to Barron et al . ( 1999 ) , these are proved by bounding the Lipschitz constant of the mapping from the parameters to the loss of the functions computed by the networks . ( Our proof also borrows ideas from the analysis of the fully connected case , especially ( Bartlett et al. , 2017 ; Neyshabur et al. , 2018 ) . ) Covering bounds may be applied to obtain a huge variety of generalization bounds . We present two examples for each covering bound . One is a standard bound on the difference between training and test error . Perhaps the more relevant bound has the flavor of “ relative error ” ; it is especially strong when the training loss is small , as is often the case in modern practice . Our covering bounds are polynomial in the inverse of the granularity of the cover . Such bounds seem to be especially useful for bounding the relative error . In particular , our covering bounds are of the form ( B/ ) W , where is the granularity of the cover , B is proportional to the Lipschitz constant of a mapping from parameters to functions , and W is the number of parameters in the model . We apply a bound from the empirical process literature in terms of covering bounds of this form due to Giné and Guillou ( 2001 ) , who paid particular attention to the dependence of estimation error on B . This bound may be helpful for other analyses of the generalization of deep learning in terms of different notions of distance from initialization . ( Applying bounds in terms of Dudley ’ s entropy integral in the standard way leads to an exponentially worse dependence on B . ) Related previous work . Du et al . ( 2018 ) proved bounds for CNNs in terms of the number of parameters , for two-layer networks . Arora et al . ( 2018 ) analyzed the generalization of networks output by a compression scheme applied to CNNs . Zhou and Feng ( 2018 ) provided a generalization guarantee for CNNs satisfying a constraint on the rank of matrices formed from their kernels . Li et al . ( 2018 ) analyzed the generalization of CNNs under other constraints on the parameters . Lee and Raginsky ( 2018 ) provided a size-free bound for CNNs in a general unsupervised learning framework that includes PCA and codebook learning . Related independent work . Ledent et al . ( 2019 ) proved bounds for CNNs that also took account of the effect of weight-tying . ( Their bounds retain the exponential dependence on the depth of the network from earlier work , and are otherwise qualitatively dissimilar to ours . ) Wei and Ma ( 2019b ) obtained bounds for fully connected networks with an improved dependence on the depth of the network . Daniely and Granot ( 2019 ) obtained improved bounds for constant-depth fully-connected networks . Jiang et al . ( 2019 ) conducted a wide-ranging empirical study of the dependence of the generalization gap on a variety of quantities , including distance from initialization . Notation . If K ( i ) is the kernel of convolutional layer number i , then op ( K ( i ) ) refers to its operator matrix 1 and vec ( K ( i ) ) denotes the vectorization of the kernel tensor K ( i ) . For matrix M , ‖M‖2 denotes the operator norm of M . For vectors , || · || represents the Euclidian norm , and || · ||1 is the L1 norm . For a multiset S of elements of some set Z , and a function g from Z to R , let ES [ g ] = 1m ∑m t=1 g ( zt ) . We will denote the function parameterized by Θ by fΘ . 2 BOUNDS FOR A BASIC SETTING . In this section , we provide a bound for a clean and simple setting . 1Convolution is a linear operator and can thus be written as a matrix-vector product . The operator matrix of kernel K , refers to the matrix that describes convolving the input with kernel K. For details , see ( Sedghi et al. , 2018 ) . 2.1 THE SETTING AND THE BOUNDS . In the basic setting , the input and all hidden layers have the same number c of channels . Each input x ∈ Rd×d×c satisfies ‖ vec ( x ) ‖ ≤ 1 . We consider a deep convolutional network , whose convolutional layers use zero-padding ( see Goodfellow et al. , 2016 ) . Each layer but the last consists of a convolution followed by an activation function that is applied componentwise . The activations are 1-Lipschitz and nonexpansive ( examples include ReLU and tanh ) . The kernels of the convolutional layers are K ( i ) ∈ Rk×k×c×c for i ∈ { 1 , . . . , L } . Let K = ( K ( 1 ) , ... , K ( L ) ) be the L× k × k × c× c-tensor obtained by concatening the kernels for the various layers . Vector w represents the last layer ; the weights in the last layer are fixed with ||w|| = 1 . Let W = Lk2c2 be the total number of trainable parameters in the network . We let K ( 1 ) 0 , . . . , K ( L ) 0 take arbitrary fixed values ( interpreted as the initial values of the kernels ) subject to the constraint that , for all layers i , || op ( K ( i ) 0 ) ||2 = 1 . ( This is often the goal of initialization schemes . ) Let K0 be the corresponding L× k× k× c× c tensor . We provide a generalization bound in terms of distance from initialization , along with other natural parameters of the problem . The distance is measured with ||K −K0||σ def = ∑L i=1 || op ( K ( i ) ) − op ( K ( i ) 0 ) ||2 . For β > 0 , define Kβ to be the set of kernel tensors within || · ||σ distance β of K0 , and define Fβ to be set of functions computed by CNNs with kernels in Kβ . That is , Fβ = { fK : ||K −K0||σ ≤ β } . Let ` : R× R→ [ 0 , 1 ] be a loss function such that ` ( · , y ) is λ-Lipschitz for all y . An example is the 1/λ-margin loss . For a function f from Rd×d×c to R , let ` f ( x , y ) = ` ( f ( x ) , y ) . We will use S to denote a set { ( x1 , y1 ) , . . . , ( xm , ym ) } = { z1 , . . . zm } of random training examples where each zt = ( xt , yt ) . Theorem 2.1 ( Basic bounds ) . For any η > 0 , there is a C > 0 such that for any β , δ > 0 , λ ≥ 1 , for any joint probability distribution P over Rd×d×c × R , if a training set S of n examples is drawn independently at random from P , then , with probability at least 1− δ , for all f ∈ Fβ , Ez∼P [ ` f ( z ) ] ≤ ( 1 + η ) ES [ ` f ] + C ( W ( β + log ( λn ) ) + log ( 1/δ ) ) n and , if β ≥ 5 , then Ez∼P [ ` f ( z ) ] ≤ ES [ ` f ] + C √ W ( β + log ( λ ) ) + log ( 1/δ ) n and otherwise Ez∼P [ ` f ( z ) ] ≤ ES [ ` f ] + C ( βλ √ W n + √ log ( 1/δ ) n ) . If Theorem 2.1 is applied with the margin loss , then Ez∼P [ ` f ( z ) ] is in turn an upper bound on the probability of misclassification on test data . Using the algorithm from ( Sedghi et al. , 2018 ) , || · ||σ may be efficiently computed . Since ||K −K0||σ ≤ || vec ( K ) − vec ( K0 ) ||1 ( Sedghi et al. , 2018 ) , Theorem 2.1 yields the same bounds as a corollary if the definition of Fβ is replaced with the analogous definition using || vec ( K ) − vec ( K0 ) ||1 .

Summary: This paper studied the generalization power of CNNs and showed several upper bounds of generalization errors. Their results have two characteristics. First, the bounds are in terms of the quantity that is independent of the input dimension (size-free). Second, the upper bounds involve the distance between initial and learned parameters. These results improved the upper bounds that we can derive by naively applying the results of Bartlett et al. (2017) or Neushubar et al. (2017), because the dominant term of the existing upper bounds contained $l_{2, 1}$ or $l_2$ norms, which could depend on the input dimensions in the worst case. The authors empirically showed that there is a correlation between the generalization error of learned CNNs and the dominant term of the upper bound (i.e., the product of the parameter size and the distance from the set of initial parameters).