Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach

1 INTRODUCTION . A pivotal question in machine learning is why deep networks perform well despite overparameterization . These models often have many more parameters than the number of examples they are trained on , which enables them to drastically overfit to training data ( Zhang et al. , 2017a ) . In common practice , however , such networks perform well on previously unseen data . Explaining the generalization performance of neural networks is an active area of current research . Attempts have been made at adapting classical measures such as VC-dimension ( Harvey et al. , 2017 ) or margin/norm bounds ( Neyshabur et al. , 2018 ; Bartlett et al. , 2017 ) , but such approaches have yielded bounds that are vacuous by orders of magnitudes . Other authors have explored modifications of the training procedure to obtain networks with provable generalization guarantees ( Dziugaite & Roy , 2017 ; 2018 ) . Such procedures often differ substantially from standard procedures used by practitioners , and empirical evidence suggests that they fail to improve performance in practice ( Wilson et al. , 2017 ) . We begin with an empirical observation : it is often possible to “ compress ” trained neural networks by finding essentially equivalent models that can be described in a much smaller number of bits ; see Cheng et al . ( 2018 ) for a survey . Inspired by classical results relating small model size and generalization performance ( often known as Occam ’ s razor ) , we establish a new generalization bound based on the effective compressed size of a trained neural network . Combining this bound with off-the-shelf compression schemes yields the first non-vacuous generalization bounds in practical problems . The main contribution of the present paper is the demonstration that , unlike many other measures , this measure is effective in the deep-learning regime . Generalization bound arguments typically identify some notion of complexity of a learning problem , and bound generalization error in terms of that complexity . Conceptually , the notion of complexity we identify is : complexity = compressed size− remaining structure . ( 1 ) The first term on the right-hand side represents the link between generalization and explicit compression . The second term corrects for superfluous structure that remains in the compressed representation . For instance , the predictions of trained neural networks are often robust to perturbations of the network weights . Thus , a representation of a neural network by its weights carries some irrelevant information . We show that accounting for this robustness can substantially reduce effective complexity . Our results allow us to derive explicit generalization guarantees using off-the-shelf neural network compression schemes . In particular : • The generalization bound can be evaluated by compressing a trained network , measuring the effective compressed size , and substituting this value into the bound . • Using off-the-shelf neural network compression schemes with this recipe yields bounds that are state-of-the-art , including the first non-vacuous bounds for modern convolutional neural nets . The above result takes a compression algorithm and outputs a generalization bound on nets compressed by that algorithm . We provide a complementary result by showing that if a model tends to overfit then there is an absolute limit on how much it can be compressed . We consider a classifier as a ( measurable ) function of a random training set , so the classifier is viewed as a random variable . We show that the entropy of this random variable is lower bounded by a function of the expected degree of overfitting . Additionally , we use the randomization tests of Zhang et al . ( 2017a ) to show empirically that increased overfitting implies worse compressibility , for a fixed compression scheme . The relationship between small model size and generalization is hardly new : the idea is a variant of Occam ’ s razor , and has been used explicitly in classical generalization theory ( Rissanen , 1986 ; Blumer et al. , 1987 ; MacKay , 1992 ; Hinton & van Camp , 1993 ; Rasmussen & Ghahramani , 2001 ) . However , the use of highly overparameterized models in deep learning seems to directly contradict the Occam principle . Indeed , the study of generalization and the study of compression in deep learning has been largely disjoint ; the later has been primarily motivated by computational and storage limitations , such as those arising from applications on mobile devices ( Cheng et al. , 2018 ) . Our results show that Occam type arguments remain powerful in the deep learning regime . The link between compression and generalization is also used in work by Arora et al . ( 2018 ) , who study compressibility arising from a form of noise stability . Our results are substantially different , and closer in spirit to the work of Dziugaite & Roy ( 2017 ) ; see Section 3 for a detailed discussion . Zhang et al . ( 2017a ) study the problem of generalization in deep learning empirically . They observe that standard deep net architectures—which generalize well on real-world data—are able to achieve perfect training accuracy on randomly labelled data . Of course , in this case the test error is no better than random guessing . Accordingly , any approach to controlling generalization error of deep nets must selectively and preferentially bound the generalization error of models that are actually plausible outputs of the training procedure applied to real-world data . Following Langford & Caruana ( 2002 ) ; Dziugaite & Roy ( 2017 ) ; Neyshabur et al . ( 2018 ) , we make use of the PAC-Bayesian framework ( McAllester , 1999 ; Catoni , 2007 ; McAllester , 2013 ) . This framework allows us to encode prior beliefs about which learned models are plausible as a ( prior ) distribution π over possible parameter settings . The main challenge in developing a bound in the PAC-Bayes framework bound is to articulate a distribution π that encodes the relative plausibilities of possible outputs of the training procedure . The key insight is that , implicitly , any compression scheme is a statement about model plausibilities : good compression is achieved by assigning short codes to the most probable models , and so the probable models are those with short codes . 2 GENERALIZATION AND THE PAC-BAYESIAN PRINCIPLE . In this section , we recall some background and notation from statistical learning theory . Our aim is to learn a classifier using data examples . Each example ( x , y ) consists of some features x ∈ X and a label y ∈ Y . It is assumed that the data are drawn identically and independently from some data generating distribution , ( Xi , Yi ) iid∼ D. The goal of learning is to choose a hypothesis h : X → Y that predicts the label from the features . The quality of the prediction is measured by specifying some loss function L ; the value L ( h ( x ) , y ) is a measure of the failure of hypothesis h to explain example ( x , y ) . The overall quality of a hypothesis h is measured by the risk under the data generating distribution : L ( h ) = E ( X , Y ) ∼D [ L ( h ( X ) , Y ) ] . Generally , the data generating distribution is unknown . Instead , we assume access to training data Sn = { ( x1 , y1 ) , . . . , ( xn , yn ) } , a sample of n points drawn i.i.d . from the data generating distribution . The true risk is estimated by the empirical risk : L̂ ( h ) = 1n ∑ ( x , y ) ∈S L ( h ( x ) , y ) . The task of the learner is to use the training data to choose a hypothesis ĥ from among some prespecified set of possible hypothesisH , the hypothesis class . The standard approach to learning is to choose a hypothesis ĥ that ( approximately ) minimizes the empirical risk . This induces a dependency between the choice of hypothesis and the estimate of the hypothesis ’ quality . Because of this , it can happen that ĥ overfits to the training data : L̂ ( ĥ ) L ( ĥ ) . The generalization error L ( ĥ ) − L̂ ( ĥ ) measures the degree of overfitting . In this paper , we consider an image classification problem , where xi is an image and yi the associated label for that image . The selected hypothesis is a deep neural network . We mostly consider the 0 -1 loss , that is , L ( h ( x ) , y ) = 0 if the prediction is correct and L ( h ( x ) , y ) = 1 otherwise . We use the PAC-Bayesian framework to establish bounds on generalization error . In general , a PACBayesian bound attempts to control the generalization error of a stochastic classifier by measuring the discrepancy between a pre-specified random classifier ( often called prior ) , and the classifier of interest . Conceptually , PAC-Bayes bounds have the form : generalization error of ρ ≤ O ( √ KL ( ρ , π ) /n ) , ( 2 ) where n is the number of training examples , π denotes the prior , and ρ denotes the classifier of interest ( often called posterior ) . More formally , we write L ( ρ ) = Eh∼ρ [ L ( h ) ] for the risk of the random estimator . The fundamental bound in PAC-Bayesian theory is ( Catoni , 2007 , Thm . 1.2.6 ) : Theorem 2.1 ( PAC-Bayes ) . Let L be a { 0 , 1 } -valued loss function , let π be some probability measure on the hypothesis class , and let α > 1 , > 0 . Then , with probability at least 1− over the distribution of the sample : L ( ρ ) ≤ inf λ > 1 Φ−1λ/n { L̂ ( ρ ) + α λ [ KL ( ρ , π ) − log + 2 log ( log ( α2λ ) logα ) ] } , ( 3 ) where we define Φ−1γ as : Φ−1γ ( x ) = 1− e−γx 1− e−γ . ( 4 ) Remark 2.2 . The above formulation of the PAC-Bayesian theorem is somewhat more opaque than other formulations ( e.g. , McAllester , 2003 ; 2013 ; Neyshabur et al. , 2018 ) . This form is significantly tighter when KL/n is large . See Bégin et al . ( 2014 ) ; Laviolette ( 2017 ) for a unified treatment of PAC-Bayesian bounds . The quality of a PAC-Bayes bound depends on the discrepancy between the PAC-Bayes prior π— encoding the learned models we think are plausible—and ρ , which is the actual output of the learning procedure . The main challenge is finding good choices for the PAC-Bayes prior π , for which the value of KL ( ρ , π ) is both small and computable . 3 RELATIONSHIP TO PREVIOUS WORK . Generalization . The question of which properties of real-world networks explain good generalization behavior has attracted considerable attention ( Langford , 2002 ; Langford & Caruana , 2002 ; Hinton & van Camp , 1993 ; Hochreiter & Schmidhuber , 1997 ; Baldassi et al. , 2015 ; 2016 ; Chaudhari et al. , 2017 ; Keskar et al. , 2017 ; Dziugaite & Roy , 2017 ; Schmidt-Hieber , 2017 ; Neyshabur et al. , 2017 ; 2018 ; Arora et al. , 2018 ) ; see Arora ( 2017 ) for a review of recent advances . Such results typically identify a property of real-world networks , formalize it as a mathematical definition , and then use this definition to prove a generalization bound . Generally , the bounds are very loose relative to the true generalization error , which can be estimated by evaluating performance on held-out data . Their purpose is not to quantify the actual generalization error , but rather to give qualitative evidence that the property underpinning the generalization bound is indeed relevant to generalization performance . The present paper can be seen in this tradition : we propose compressibility as a key signature of performant real-world deep nets , and we give qualitative evidence for this thesis in the form of a generalization bound . The idea that compressibility leads to generalization has a long history in machine learning . Minimum description length ( MDL ) is an early formalization of the idea ( Rissanen , 1986 ) . Hinton & van Camp ( 1993 ) applied MDL to very small networks , already recognizing the importance of weight quantization and stochasticity . More recently , Arora et al . ( 2018 ) consider the connection between compression and generalization in large-scale deep learning . The main idea is to compute a measure of noise-stability of the network , and show that it implies the existence of a simpler network with nearly the same performance . A variant of a known compression bound ( see ( McAllester , 2013 ) for a PAC-Bayesian formulation ) is then applied to bound the generalization error of this simpler network in terms of its code length . In contrast , the present paper develops a tool to leverage existing neural network compression algorithms to obtain strong generalization bounds . The two papers are complementary : we establish non-vacuous bounds , and hence establish a quantitative connection between generalization and compression . An important contribution of Arora et al . ( 2018 ) is obtaining a quantity measuring the compressibility of a neural network ; in contrast , we apply a compression algorithm and witness its performance . We note that their compression scheme is very different from the sparsity-inducing compression schemes ( Cheng et al. , 2018 ) we use in our experiments . Which properties of deep networks allow them to be sparsely compressed remains an open question . To strengthen a naïve Occam bound , we use the idea that deep networks are insensitive to mild perturbations of their weights , and that this insensitivity leads to good generalization behavior . This concept has also been widely studied ( e.g. , Langford , 2002 ; Langford & Caruana , 2002 ; Hinton & van Camp , 1993 ; Hochreiter & Schmidhuber , 1997 ; Baldassi et al. , 2015 ; 2016 ; Chaudhari et al. , 2017 ; Keskar et al. , 2017 ; Dziugaite & Roy , 2017 ; Neyshabur et al. , 2018 ) . As we do , some of these papers use a PAC-Bayes approach ( Langford & Caruana , 2002 ; Dziugaite & Roy , 2017 ; Neyshabur et al. , 2018 ) . Neyshabur et al . ( 2018 ) arrive at a bound for non-random classifiers by computing the tolerance of a given deep net to noise , and bounding the difference between that net and a stochastic net to which they apply a PAC-Bayes bound . Like the present paper , Langford & Caruana ( 2002 ) ; Dziugaite & Roy ( 2017 ) work with a random classifier given by considering a normal distribution over the weights centered at the output of the training procedure . We borrow the observation of Dziugaite & Roy ( 2017 ) that the stochastic network is a convenient formalization of perturbation robustness . The approaches to generalization most closely related to ours are , in summary : Reference Structure Non-VacuousMNIST ImageNet Dziugaite & Roy ( 2017 ) Perturbation Robustness 3 7 Neyshabur et al . ( 2018 ) Perturbation Robustness 7 7 Arora et al . ( 2018 ) Compressibility ( from Perturbation Robustness ) 7 7 Present paper Compressibility and Perturbation Robustness 3 3 These represent the best known generalization guarantees for deep neural networks . Our bound provides the first non-vacuous generalization guarantee for the ImageNet classification task , the de facto standard problem for which deep learning dominates . It is also largely agnostic to model architecture : we apply the same argument to both fully connected and convolutional networks . This is in contrast to some existing approaches that require extra analysis to extend bounds for fully connected networks to bounds for convolutional networks ( Neyshabur et al. , 2018 ; Konstantinos et al . ; Arora et al. , 2018 ) . Compression . The effectiveness of our work relies on the existence of good neural network compression algorithms . Neural network compression has been the subject of extensive interest in the last few years , motivated by engineering requirements such as computational or power constraints . We apply a relatively simple strategy in this paper in the line of Han et al . ( 2016 ) , but we note that our bound is compatible with most forms of compression . See Cheng et al . ( 2018 ) for a survey of recent results in this field .

Summary: This paper tries to push forward in important directions the seemingly increasingly powerful approach of using PAC-Bayesian formalism to explain low risks of training neural nets on real-life data. They take an interesting approach to evaluate these bounds by setting up a prior distribution as a mixture of Gaussians centered on possible heuristic compressions of the net  and this prior's variances are obtained by doing a layerwise grid search. This seems to give good risk bounds on certain known compressible nets using image data sets.