Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

1 INTRODUCTION . Deep learning has shown quite successful results in wide range of machine learning applications . such as image recognition ( Krizhevsky et al. , 2012 ) , natural language processing ( Devlin et al. , 2018 ) and image synthesis tasks ( Radford et al. , 2015 ) . The success of deep learning methods is mainly due to its flexibility , expression power and computational efficiency for large dataset training . Due to its significant importance in wide range of application areas , its theoretical analysis is also getting much important . For example , it has been known that the deep neural network has universal approximation capability ( Cybenko , 1989 ; Hornik , 1991 ; Sonoda & Murata , 2015 ) and its expressive power grows up in an exponential order against the number of layers ( Montufar et al. , 2014 ; Bianchini & Scarselli , 2014 ; Cohen et al. , 2016 ; Cohen & Shashua , 2016 ; Poole et al. , 2016 ; Suzuki , 2019 ) . However , theoretical understandings are still lacking in several important issues . Among several topics of deep learning theories , a generalization error analysis is one of the biggest issues in the machine learning literature . An important property of deep learning is that it generalizes well even though its parameter size is quite large compared with the sample size ( Neyshabur et al. , 2019 ) . This can not be well explained by a classical VC-dimension type theory ( Harvey et al. , 2017 ) which suggests that overparameterized models cause overfitting and thus result in poor generalization ability . For this purpose , norm based bounds have been extensively studied so far ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017b ; Neyshabur et al. , 2017 ; Golowich et al. , 2018 ) . These bounds are beneficial because the bounds are not explicitly dependent on the number of parameters and thus are useful to explain the generalization error of overparameterized network ( Bartlett , 1998 ; Neyshabur et al. , 2015 ; 2019 ) . However , these bounds are typically exponentially dependent on the number of layers and thus tends to be loose for deep network situations ( Dziugaite & Roy , 2017 ; Arora et al. , 2018 ; Nagarajan & Kolter , 2019 ) . As a result , Arora et al . ( 2018 ) reported that a simple VC-dimension bound ( Li et al. , 2018 ; Harvey et al. , 2017 ) can still give sharper evaluations than these norm based bounds in some practically used deep networks . Wei & Ma ( 2019 ) improved this issue by involving a data dependent Lipschitz constant as performed in Arora et al . ( 2018 ) ; Nagarajan & Kolter ( 2019 ) . On the other hand , compression based bound is another promising approach for tight generalization error evaluation which can avoid the exponential dependence on the depth . The complexity of deep neural network model is regulated from several aspects . For example , we usually impose explicit regularization such as weight decay ( Krogh & Hertz , 1992 ) , dropout ( Srivastava et al. , 2014 ; Wager et al. , 2013 ) , batch-normalization ( Ioffe & Szegedy , 2015 ) , and mix-up ( Zhang et al. , 2018 ; Verma et al. , 2018 ) . Zhang et al . ( 2016 ) reported that such explicit regularization does not have much effect but implicit regularization induced by SGD ( Hardt et al. , 2016 ; Gunasekar et al. , 2018 ; Ji & Telgarsky , 2019 ) is important . Through these explicit and implicit regularizations , deep learning tends to produce a simpler model than its full expression ability ( Valle-Perez et al. , 2019 ; Verma et al. , 2018 ) . To measure how “ simple ” the trained model is , one of the most promising approaches currently investigated is the compression bounds ( Arora et al. , 2018 ; Baykal et al. , 2019 ; Suzuki et al. , 2018 ) . These bounds measure how much the network can be compressed and characterize the size of the compressed network as the implicit effective dimensionality . Arora et al . ( 2018 ) characterized the implicit dimensionality based on so called layer-cushion quantity and suggested to perform random projection to obtain a compressed network . Along with a similar direction , Baykal et al . ( 2019 ) proposed a pruning scheme called Corenet and derived a bound of the size of the compressed network . Suzuki et al . ( 2018 ) has developed a spectrum based bound for their compression scheme . Unfortunately , all of these bounds guarantee the generalization error of only the compressed network , not the original network . Hence , it does not give precise explanations about why large network can avoid overfitting . In this paper , we derive a unified framework to obtain a compression based bound for a noncompressed network . Unlike the existing researches , our bound is valid to evaluate the original network before compression , and thus gives a direct explanation about why deep learning generalizes despite its large network size . The difficulty to apply the compression bound to the original network lies in evaluation of the population L2-bound between the compression network and the original network . A naive evaluation results in the VC-bound which is not preferable . This difficulty is overcome by developing novel data dependent capacity control technique using local Rademacher complexity bounds ( Mendelson , 2002 ; Bartlett et al. , 2005 ; Koltchinskii , 2006 ; Giné & Koltchinskii , 2006 ) . Then , the bound is applied to some typical situations where the network is well compressed . Our analysis stands on the implicit bias hypothesis ( Gunasekar et al. , 2018 ; Ji & Telgarsky , 2019 ) that claims deep learning tends to produce rather simple models . Actually , Gunasekar et al . ( 2018 ) ; Ji & Telgarsky ( 2019 ) showed gradient descent results in ( near ) low rank parameter matrices in each layer in linear network settings . Martin & Mahoney ( 2018 ) evaluated the eigenvalue decays of the weight matrix through random matrix theories and several numerical experiments . These observations are also supported by the flat minimum analysis ( Hochreiter & Schmidhuber , 1997 ; Wu et al. , 2017 ; Langford & Caruana , 2002 ) , that is , the product of the eigenvalues of the Hessian around the SGD solution tends to be small , which means SGD converges to a flat minimum and possess stability against small perturbations leading to good generalization . Based on these observations , we make use of the eigenvalue decay of the weight matrix and the covariance matrix among the nodes in each layer ( this assumption is actually verified by numerical experiments in Appendix D ) . The eigenvalue decay speed characterizes the redundancy in each layer and thus is directly relevant to compression ability . Our contributions in this paper are summarized as follows : • We give a unified framework to obtain a compression based bound for non-compressed network which properly explains that a compressible network can generalizes well . The bound can convert several existing compression based bounds to that for non-compressed one in a unifying manner . The bound is applied to near low rank models as concrete examples . • We develop a data dependent capacity control technique to bound the discrepancy between the original network and compressed network . As a result , we obtain a sharp generalization Other related work Recently , the role of over-parameterization for two layer networks has been extensively studied ( Neyshabur et al. , 2019 ; Arora et al. , 2019 ) . These are for the shallow network and the generalization error is essentially given by the norm based bounds . It is not obvious that these bounds also give sharp bounds for deep models . PAC-Bayes bound is also applied to obtain a non-vacuous compression based bound ( Zhou et al. , 2019 ) . However , the bound is still for the compressed ( quantized ) models and it is not obvious that that bound can be converted to that for the original network . Relation between compression and learnability was traditionally studied in a different framework as in Littlestone & Warmuth ( 1986 ) and minimum description code length ( Hinton & Van Camp , 1993 ) . Our bound would share the same spirits with these studies but give a new analysis by incorporating recent observations in deep learning researches . 2 PRELIMINARIES : PROBLEM FORMULATION AND NOTATIONS . In this section , we give the problem setting and notations that will be used in the theoretical analysis . We consider the standard supervised leaning formulation where data consists of input x ∈ Rd and output ( or label ) y ∈ R. We consider a single output setting , i.e. , the output y is a 1-dimensional real value , but it is straight forward to generalize the result to a multiple output case . Suppose that we are given n i.i.d . observations Dn = ( xi , yi ) ni=1 distributed from a probability distribution P . To measure the performance of a trained function f , we use a loss function ψ : R× R→ R and define a training error and its expected one as Ψ̂ ( f ) : = 1 n n∑ i=1 ψ ( yi , f ( xi ) ) , Ψ ( f ) : = E [ ψ ( Y , f ( X ) ) ] , where the expectation is taken with respect to ( X , Y ) ∼ P . Basically , we are interested in the generalization error Ψ ( f̂ ) − Ψ̂ ( f̂ ) for an estimator f̂ . We denote the empirical L2-norm by ‖f‖n : =√∑n i=1 f ( zi ) 2/n for an empirical observation zi = ( xi , yi ) ( i = 1 , . . . , n ) . The population L2norm is denoted by ‖f‖L2 : = √ EZ∼P [ f ( Z ) 2 ] . This paper deals with deep neural networks as a model . The activation function is denoted by η which will be assumed to be 1-Lipschitz as satisfied by ReLU ( Assumption 1 ) . Let the depth of the network be L and the width of the ` -th internal layer be m ` ( ` = 1 , . . . , L + 1 ) where we set m1 = d ( dimension of input ) and mL+1 = 1 ( dimension of output ) for convention . Then , the set of networks having depth L and width m = ( m1 , . . . , mL ) with norm constraint as NN ( m , R′2 , R ′ F ) : = { f ( x ) = G ◦ ( W ( L ) η ( · ) ) ◦ ( W ( L−1 ) η ( · ) ) ◦ · · · ◦ ( W ( 1 ) x ) | W ( ` ) ∈ Rm ` ×m ` +1 , ‖W ( ` ) ‖2 ≤ R′2 , ‖W ( ` ) ‖F ≤ R′F } . where ‖W‖2 : = supu : ‖Wu‖6=0 ‖Wu‖/‖u‖1 is the operator norm ( the maximum singular value ) , ‖W‖F : = √∑ i , jW 2 i , j is the Frobenius norm , and G is the “ clipping ” operator that is defined by G ( x ) = max { −M , min { x , M } } for a constant M . The reason why we put the clipping operator G on top of the last layer is because the clipping operator restricts the L∞-norm by a constant M and then we can avoid unrealistically loose generalization error . Note that the clipping operator does not change the classification error for binary classification . We express F to represent the “ full model ” : F = NN ( m , R2 , RF ) for a given R2 , RF > 0 . Here , we implicitly suppose that R2 is close to 1 so that the norm of the output from internal layers is not too much amplified , while RF could be moderately large . The Rademacher complexity is the typical tool to evaluate the generalization error on a function class F ′ , which is denoted by R̂n ( F ′ ) : = E [ supf∈F ′ 1 n ∑n i=1 if ( zi ) | Dn ] where Dn = ( zi ) ni=1 = ( xi , yi ) n i=1 , and i ( i = 1 , . . . .n ) is an i.i.d . Rademacher sequence ( P ( i = 1 ) = P ( i = −1 ) = 1/2 ) . This is also called conditional Rademacher complexity because the expectation is taken conditioned on fixed Dn . Its expectation with respect to Dn is denoted by R̄n ( F ′ ) : = EDn [ R̂n ( F ′ ) ] . Roughly speaking the Rademacher complexity measures the size of the model and it gives an upper bound of the generalization error ( Vapnik , 1998 ; Mohri et al. , 2012 ) . The main difficulty in generalization error analysis of deep learning is that the Rademacher complexity of the full model F is quite large . One of the successful approaches to avoid this difficulty is the compression based bound ( Arora et al. , 2018 ; Baykal et al. , 2019 ; Suzuki et al. , 2018 ) which measures how much the trained network f̂ can be compressed . If the network can be compressed to much smaller one , then its intrinsic dimensionality can be regarded as small . To describe it more precisely , suppose that the trained network f̂ is included in a subset of the neural network model : f̂ ∈ F̂ ⊂ F . For example , F̂ can be a set of networks with weight matrices that have bounded norms and are near low rank ( Sec . 3.1 or Sec . 3.2 ) . We do not assume a specific type of training procedure , but we give a uniform bound valid for any estimator f̂ that falls into F̂ and satisfies the following compressibility condition . We suppose that the network f̂ is easy to compress , that is , f̂ can be compressed to a smaller network ĝ which is included in a submodel : ĝ ∈ Ĝ . For example , Ĝ can be a set of networks with a smaller size than f̂ . How small the trained network f̂ can be compressed has been characterized by several notions such as “ layer-cushion ” ( Arora et al. , 2018 ) . Typical compression based bounds give generalization errors of the compressed model ĝ , not the original network f̂ . Our approach converts an error bound of ĝ to that of f̂ and eventually obtains a tighter evaluation . 1In this paper , ‖ · ‖ denotes the Euclidean norm : ‖u‖ = √∑ i u 2 i . The biggest difficulty for transforming the compression bound to that of f̂ lies in evaluation of the population L2-norm between f̂ and ĝ . Basically , the compression based bounds are given as Ψ ( ĝ ) ≤ Ψ̂ ( f̂ ) + ‖f̂ − ĝ‖n + CR̄n ( Ĝ ) , ( 1 ) for a constant C > 0 under some assumptions ( Table 1 ) . The term ‖f̂ − ĝ‖n appears to adapt the empirical error of f̂ to that of ĝ , that is called “ compression error ” which can be seen as a bias term . We see that , in the right hand side , there appears the complexity of Ĝ which is assumed to be much smaller than that of the full model F . However , the left hand side is not the expected error of f̂ but that of ĝ . One way to transfer this bound to that of f̂ is that we have |Ψ ( ĝ ) − Ψ ( f̂ ) | ≤ ‖ĝ − f̂‖L2 by assuming Lipschitz continuity of the loss function and then convert the bound ( 1 ) to Ψ ( f̂ ) ≤ Ψ̂ ( f̂ ) + ( ‖f̂ − ĝ‖n + ‖f̂ − ĝ‖L2 ) + R̄n ( Ĝ ) . However , to bound the term ‖f̂ − ĝ‖n + ‖f̂ − ĝ‖L2 , there typically appears the complexity of the model F̂ which is larger than the compressed model Ĝ like ‖f̂ − ĝ‖n ≤ √ ‖f̂ − ĝ‖2L2 +Op ( R̄ ( F̂ ) ) , which results in slow convergence rate . To overcome this difficulty , we need to carefully control the difference between the training and test error of f̂ and ĝ by utilizing the local Rademacher complexity technique ( Mendelson , 2002 ; Bartlett et al. , 2005 ; Koltchinskii , 2006 ; Giné & Koltchinskii , 2006 ) . The local Rademacher complexity of a model F ′ with radius r > 0 is defined as Ṙr ( F ′ ) : = R̄n ( { f ∈ F ′ | ‖f‖L2 ≤ r } ) . The main difference from the standard Rademacher complexity is that the model is localized to a set of functions satisfying ‖f‖L2 ≤ r. As a result , we obtain a tighter error bound . Throughout this paper , we always assume the following assumptions . Let PX and PY denote the marginal distribution of x and that of y respectively . Assumption 1 ( Lipschitz continuity of loss and activation functions ) . The loss function ψ is 1- Lipschitz continuous with respect to the function output : |ψ ( y , u ) − ψ ( y , u′ ) | ≤ |u− u′| ( ∀y ∈ supp ( PY ) , u , u′ ∈ R ) . The activation function η is also 1-Lipschitz continuous : ‖η ( u ) − η ( u′ ) ‖ ≤ ‖u − u′‖ ( ∀u ∈ Rd′ ) where d′ is any positive integer . Assumption 2 . The norm of input is bounded by Bx > 0 : ‖x‖ ≤ Bx ( ∀x ∈ supp ( PX ) ) . Assumption 3 . The L∞-norms of all elements in F̂ and Ĝ are bounded by M ≥ 1 : ‖f‖∞ , ‖g‖∞ ≤ M for all f ∈ F̂ and g ∈ Ĝ . This assumption can be ensured by applying the clipping operator G to the output of the functions . In this paper , all the variables L , m ` , R2 , RF , M , Bx are supposed to be o ( n ) . What we will derive in the following is a bound which has mild dependency on the depth L and depends on the width ( m ` ) L ` =1 in a sub-linear order by using the compression based approach . Existing bounds for no-compressed network Here we give a brief review of the generalization error bound for non-compressed models . ( i ) VC-bound : The Rademacher complexity of the full model F can be bounded by a naive VC-dimension bound ( Harvey et al. , 2017 ) which is R̄n ( F ) = O ( √ L ∑n ` =1m ` m ` +1 n log ( n ) ) . In this bound , there appears the number of parame- ters ∑n ` =1m ` m ` +1 in the numerator . However , the number of parameters is often larger than the sample size n in practical use . Hence , this bound is not appropriate to evaluate generalization ability of overparameterized networks . ( ii ) Norm-based bound : Golowich et al . ( 2018 ) showed the norm based bound which is given as R̄n ( F ) = O ( √ LRLF n ) . However , this is exponentially dependent on the depth as RLF resulting in quite loose bound . Neyshabur et al . ( 2017 ) showed a norm based bound of R̄n ( F ) = O ( √ L3 ( max ` m ` ) R2F/R 2 2 n ) which avoids the exponential de- pendency . However , there is still dependency on the width , ∑ ` m ` R 2 F , which is larger than the linear order of the width since RF could be moderately large . Bartlett et al . ( 2017b ) showed R̄n ( F ) = O ( RL2√ n ( L R 2/3 2→1 R 2/3 2 ) 3/2 ) . The norm constraint on R2→1 implicitly assumes sparsity on the weight matrix and R2→1 typically depends on the width linearly . Wei & Ma ( 2019 ) improved the exponential dependency RL2 appearing in this bound ( Bartlett et al. , 2017b ) to obtained a bound O ( 1√ n ( 1 + Lκ 4 3R 2/3 2→1 + Lκ 2 3R 2/3 1→1 ) 3/2 ) where κ is the Lipschitz continuity between layers . We can see that R22→1 and R 2 1→1 can depend on the width linearly and quadratically respectively even though R2 is bounded .

Summary: This paper obtains a compression-based generalization bound (Theorem 1) for the original network, while prior work gives bounds for the compressed network. The general bound given by Theorem 1 is further applied to networks with low-rank weight matrices (Theorem 2 and Corollary 1) or low-rank covariance matrices (Theorem 3 and 4). In some cases, the bound given by Theorem 1 for the original network could be better than the bound for the compressed network.