On Tighter Generalization Bounds for Deep Neural Networks: CNNs, ResNets, and Beyond

1 INTRODUCTION . We aim to provide a theoretical justification for the enormous success of deep neural networks ( DNNs ) in real world applications ( He et al. , 2016 ; Collobert et al. , 2011 ; Goodfellow et al. , 2016 ) . In particular , our paper focuses on the generalization performance of a general class of DNNs . The generalization bound is a powerful tool to characterize the predictive performance of a class of learning models for unseen data . Early studies investigate the generalization ability of shallow neural networks with no more than one hidden layer ( Bartlett , 1998 ; Anthony & Bartlett , 2009 ) . More recently , studies on the generalization bounds of deep neural networks have received increasing attention ( Dinh et al. , 2017 ; Bartlett et al. , 2017 ; Golowich et al. , 2017 ; Neyshabur et al. , 2015 ; 2017 ) . There are two major questions of our interest in these analysis of the generalization bounds : ( Q1 ) Can we establish tighter generalization error bounds for deep neural networks in terms of the network dimensions and structure of the weight matrices ? ( Q2 ) Can we develop generalization bounds for neural networks with special architectures ? For ( Q1 ) , ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ; Golowich et al. , 2017 ) have established results that characterize the generalization bounds in terms of the depth D and width p of networks and norms of rank-r weight matrices . For example , Neyshabur et al . ( 2015 ) provide an exponential bound on D based on the Frobenius norm ‖Wd‖F , where Wd is the weight matrix of d-th layer ; Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) provide a polynomial bound on p and D based on ‖Wd‖2 ( spectral norm ) and ‖Wd‖2,1 ( sum of the Euclidean norms for all rows of Wd ) . Golowich et al . ( 2017 ) provide a nearly size independent bound based on ‖Wd‖F . Nevertheless , the generalization bound depends on other than the spectral norms of the weight matrices may be too loose . In specific , ‖Wd‖F ( ‖Wd‖2,1 ) is in general √ r ( r ) times larger than ‖Wd‖2 . Given m training data points and suppose ‖Wd‖2 = 1 for ease of discussion , Bartlett et al . ( 2017 ) and Neyshabur et al . ( 2017 ) demonstrate generalization error bounds as Õ ( √ D3pr/m ) , and Golowich et al . ( 2017 ) achieve a bound Õ ( rD/2 min ( m−1/4 , √ D/m ) ) , where Õ ( · ) represents the rate by ignoring logarithmic factors . In comparison , we show a tighter generalization error bound as Õ ( √ Dpr/m ) , which is significantly smaller than existing results and achieved based on a new Lipschitz analysis for DNNs in terms of both the input and weight matrices . We notice that some recent results characterize the generalization bound in more structured ways , e.g. , by considering specific error-resilience parameters ( Arora et al. , 2018 ) , which can achieve empirically improved generalization bounds than existing ones based on the norms of weight matrices . However , it is not clear how the weight matrices explicitly control these parameters , which makes the results less interpretable . Thus , we do not compare with these types of results . We summarize the comparison between existing norm based generalization bounds with our results in Table 1 , as well as the results when ‖Wd‖2 = 1 for more explicit comparison in terms of the network sizes ( i.e , depth and width ) . For ( Q2 ) , we consider several widely used architectures to demonstrate , including convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ) , residual networks ( ResNets ) ( He et al. , 2016 ) , and hyperspherical networks ( SphereNets ) ( Liu et al. , 2017b ) . By taking their structures of weight matrices into consideration , we provide tight characterization of their resulting capacities . In particular , we consider orthogonal filters and normalized weight matrices , which show good performance in both optimization and generalization ( Mishkin & Matas , 2015 ; Xie et al. , 2017 ) . This is closely related with normalization frameworks , e.g. , batch normalization ( Ioffe & Szegedy , 2015 ) and layer normalization ( Ba et al. , 2016 ) , which have achieved great empirical performance ( Liu et al. , 2017a ; He et al. , 2016 ) . Take CNNs as an example . By incorporating the orthogonal structure of convolutional filters , we achieve Õ ( ( k s ) D 2 √ Dk2/ √ m ) , while Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) achieve Õ ( ( k s ) D−1 2 √ D3p2/ √ m ) and Golowich et al . ( 2017 ) achieve Õ ( p D 2 min { 1 4√m , √ D m } ) ( rank ( Wd ) = p in CNNs ) , where k is the filter size that satisfies k p and s is stride size that is usually of the same order with k ; see Section 4.1 for details . Here we achieve stronger results in terms of both depth D and width p for CNNs , where our bound only depend on k rather than p. Some recent result achieved results that is free of the linear dependence on the weight matrix norms by considering networks with bounded outputs ( Zhou & Feng , 2018 ) . We can achieve similar results using bounded loss functions as discussed in Section 3.2 , but do not restrict ourselves to this scenario in general . Analogous improvement is also attained for ResNets and SphereNets . In addition , we consider some widely used operations for width expansion and reduction , e.g. , padding and pooling , and show that they do not increase the generalization bound . Further numerical evaluation is provided for quantitative comparison in Section 4.5 . Our tighter bounds result in potentially stronger expressive power , hence higher training/testing accuracy for the DNNs . In particular , when achieving the same order of generalization errors , we allow the choice of a larger parameter space with deeper/wider networks and larger matrix spectral norms . We further show numerically that a larger parameter space can lead to better empirical performance . Quantitative analysis for the expressive power of DNNs is of great interest on its own , which includes ( but not limited to ) studying how well DNNs can approximate general class of functions and distributions ( Cybenko , 1989 ; Hornik et al. , 1989 ; Funahashi , 1989 ; Barron , 1993 ; 1994 ; Lee et al. , 2017 ; Petersen & Voigtlaender , 2017 ; Hanin & Sellke , 2017 ) , and quantifying the computation hardness of learning neural networks ; see e.g. , Shamir ( 2016 ) ; Eldan & Shamir ( 2016 ) ; Song et al . ( 2017 ) . We defer our investigation toward this to future efforts . Notation . Given an integer n > 0 , we define [ n ] = { 1 , . . . , n } . Given a matrix A ∈ Rn×m , we denote ‖A‖ as a generic norm , ‖A‖2 as the spectral norm , ‖A‖F as the Frobenius norm , and ‖A‖2,1 = ∑n i=1 ‖Ai∗‖2 . We use the standard notations O ( · ) , Θ ( · ) , and Ω ( · ) to denote limiting behaviors ignoring constants , and Õ ( · ) , Θ̃ ( · ) and Ω̃ ( · ) to further ignore logarithmic factors . 2 PRELIMINARIES . We provide a brief description of the DNNs . Given an input x ∈ Rp0 , the output of a D-layer network is defined as fDf ( WD , x ) = fWD ( · · · fW1 ( x ) ) ∈ RpD , where fWd ( y ) = σd ( Wd · y ) : Rpd−1 → Rpd with an entry-wise activation function σd ( · ) . We specify σd as the rectified linear unit ( ReLU ) activation ( Nair & Hinton , 2010 ) for ease of discussion . The extension to more general activations , e.g. , Lipschitz continuous functions , is straightforward . Then we denote DNNs with bounded weight matricesWD = { Wd ∈ Rpd×pd−1 } D d=1 and ranks as FD , ‖·‖ = { f ( WD , x ) | ∀d ∈ [ D ] , Wd ∈ WD , ‖Wd‖ ≤ Bd , rank ( Wd ) ≤ rd } , ( 1 ) where x ∈ Rp0 is an input , and { Bd } are real positive constants . We will specify the norm ‖·‖ and the corresponding upper bounds Bd , e.g. , ‖·‖2 and Bd,2 , or ‖·‖F and Bd , F , when necessary . Given a loss function g ( · , · ) , we denote a class of loss functions measuring the discrepancy between a DNN ’ s output f ( WD , x ) and the corresponding observation y ∈ Ym for a given input x ∈ Xm as G ( FD , ‖·‖ ) = { g ( f ( WD , x ) , y ) ∈ R | x ∈ Xm , y ∈ Ym , f ( · , · ) ∈ FD , ‖·‖ } , where the sets of bounded inputs Xm and the corresponding observations Ym are Xm = { xi ∈ Rp0 | ‖xi‖2 ≤ R for all i ∈ [ m ] } ⊂ X and Ym = { yi ∈ [ pD ] for all i ∈ [ m ] } ⊂ Y . Then the empirical Rademacher complexity ( ERC ) of G ( FD , ‖·‖ ) given Xm and Ym is Rm ( G ( FD , ‖·‖ ) ) = E ∈ { ±1 } m [ sup f ( · , · ) ∈FD , ‖·‖ ∣∣∣∣ 1m m∑ i=1 i · g ( f ( WD , xi ) , yi ) ∣∣∣∣ ∣∣∣∣∣ Xm , Ym ] , ( 2 ) where { ±1 } m ∈ Rm is the set of vectors only containing entries +1 and−1 , and ∈ Rm is a vector with Rademacher entries , i.e. , i = +1 or −1 with equal probabilities . Take the classification as an example . For multi-class classification , suppose pD = Nclass is the number of classes . Consider g with bounded outputs , namely the ramp risk . Specifically , for an input x belonging to class y ∈ [ Nclass ] , we denote ν = ( f ( WD , x ) ) y −maxi 6=y ( f ( WD , x ) ) i . For a given real value γ > 0 , the class of ramp risk functions with parameter γ is Gγ ( FD , ‖·‖ ) = { gγ ( f ( WD , x ) , y ) |fD ∈ FD , ‖·‖ } , where gγ is 1 γ -Lipschitz continuous , defined in ( 3 ) . For conve- gγ ( f ( WD , x ) , y ) = 0 , ν > γ1− νγ , ν ∈ [ 0 , γ ] 1 , ν < 0 , ( 3 ) nience , we denote gγ ( f ( WD , x ) , y ) as gγ ( f ( WD , x ) ) ( or gγ ) in the rest of the paper . Then the generalization error bound for PAC learning ( Bartlett et al. , 2017 ) ( Lemma 3.1 ) states the following . Given any real δ ∈ ( 0 , 1 ) and gγ , with probability at least 1 − δ , we have that for any f ( · , · ) ∈ FD , ‖·‖ , the generalization error is upper bounded with respect to ( w.r.t . ) the ERC satisfies E [ gγ ( f ( WD , x ) ) ] − 1m ∑m i=1 gγ ( f ( WD , xi ) ) ≤ 2Rm ( Gγ ( FD , ‖·‖ ) ) + 3 √ log ( 2δ ) 2m . ( 4 ) The right hand side of ( 4 ) is viewed as a guaranteed error bound for the gap between the testing and the empirical training performance . Since the ERC is generally the dominating term in ( 4 ) , a small Rm is desired for DNNs given the loss function gγ . Analogous results hold for regression tasks ; see e.g. , Kearns & Vazirani ( 1994 ) ; Mohri et al . ( 2012 ) for details .

Summary: The paper provides a generalization bound for multi-layered deep neural networks in terms of dimensions rather than norms. The bound is derived by controlling Rademacher complexity of the Ramp loss under the Lipschitzness of the network as a parametric function in Depth * Width ^2 number of parameters, and then using standard L-2 covering and Dudley Integral. They extend this technique for CNNs, Resnets, Hyper-spherical Networks, etc and provide specialized bounds for each case. In the end, the authors provide comparisons to the existing bounds.