Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience

1 INTRODUCTION . Modern deep neural networks contain millions of parameters and are trained on relatively few samples . Conventional wisdom in machine learning suggests that such models should massively overfit on the training data , as these models have the capacity to memorize even a randomly labeled dataset of similar size ( Zhang et al. , 2017 ; Neyshabur et al. , 2015 ) . Yet these models have achieved state-ofthe-art generalization error on many real-world tasks . This observation has spurred an active line of research ( Soudry et al. , 2018 ; Brutzkus et al. , 2018 ; Li & Liang , 2018 ) that has tried to understand what properties are possessed by stochastic gradient descent ( SGD ) training of deep networks that allows these networks to generalize well . One particularly promising line of work in this area ( Neyshabur et al. , 2017 ; Arora et al. , 2018 ) has been bounds that utilize the noise-resilience of deep networks on training data i.e. , how much the training loss of the network changes with noise injected into the parameters , or roughly , how wide is the training loss minimum . While these have yielded generalization bounds that do not have a severe exponential dependence on depth ( unlike other bounds that grow with the product of spectral norms of the weight matrices ) , these bounds are quite limited : they either apply to a stochastic version of the classifier ( where the parameters are drawn from a distribution ) or a compressed version of the classifier ( where the parameters are modified and represented using fewer bits ) . In this paper , we revisit the PAC-Bayesian analysis of deep networks in Neyshabur et al . ( 2017 ; 2018 ) and provide a general framework that allows one to use noise-resilience of the deep network on training data to provide a bound on the original deterministic and uncompressed network . We achieve this by arguing that if on the training data , the interaction between the ‘ activated weight matrices ’ ( weight matrices where the weights incoming from/outgoing to inactive units are zeroed out ) satisfy certain conditions which results in a wide training loss minimum , these conditions themselves generalize to the weight matrix interactions on the test data . After presenting this general PAC-Bayesian framework , we specialize it to the case of deep ReLU networks , showing that we can provide a generalization bound that accomplishes two goals simultaneously : i ) it applies to the original network and ii ) it does not scale exponentially with depth in terms of the products of the spectral norms of the weight matrices ; instead our bound scales with more meaningful terms that capture the interactions between the weight matrices and do not have such a severe dependence on depth in practice . We note that all but one of these terms are indeed quite small on networks in practice . However , one particularly ( empirically ) large term that we use is the reciprocal of the magnitude of the network pre-activations on the training data ( and so our bound would be small only in the scenario where the pre-activations are not too small ) . We emphasize that this drawback is more of a limitation in how we characterize noise-resilience through the specific conditions we chose for the ReLU network , rather than a drawback in our PAC-Bayesian framework itself . Our hope is that , since our technique is quite general and flexible , by carefully identifying the right set of conditions , in the future , one might be able to derive a similar generalization guarantee that is smaller in practice . To the best of our knowledge , our approach of generalizing noise-resilience of deep networks from training data to test data in order to derive a bound on the original network that does not scale with products of spectral norms , has neither been considered nor accomplished so far , even in limited situations . 2 BACKGROUND AND RELATED WORK . One of the most important aspects of the generalization puzzle that has been studied is that of the flatness/width of the training loss at the minimum found by SGD . The general understanding is that flatter minima are correlated with better generalization behavior , and this should somehow help explain the generalization behavior ( Hochreiter & Schmidhuber , 1997 ; Hinton & van Camp , 1993 ; Keskar et al. , 2017 ) . Flatness of the training loss minimum is also correlated with the observation that on training data , adding noise to the parameters of the network results only in little change in the output of the network – or in other words , the network is noise-resilient . Deep networks are known to be similarly resilient to noise injected into the inputs ( Novak et al. , 2018 ) ; but note that our theoretical analysis relies on resilience to parameter perturbations . While some progress has been made in understanding the convergence and generalization behavior of SGD training of simple models like two-layered hidden neural networks under simple data distributions ( Neyshabur et al. , 2015 ; Soudry et al. , 2018 ; Brutzkus et al. , 2018 ; Li & Liang , 2018 ) , all known generalization guarantees for SGD on deeper networks – through analyses that do not use noise-resilience properties of the networks – have strong exponential dependence on depth . In particular , these bounds scale either with the product of the spectral norms of the weight matrices ( Neyshabur et al. , 2018 ; Bartlett et al. , 2017 ) or their Frobenius norms ( Golowich et al. , 2018 ) . In practice , the weight matrices have a spectral norm that is as large as 2 or 3 , and an even larger Frobenius norm that scales with √ H where H is the width of the network i.e. , maximum number of hidden units per layer . 1 Thus , the generalization bound scales as say , 2D or HD/2 , where D is the depth of the network . At a high level , the reason these bounds suffer from such an exponential dependence on depth is that they effectively perform a worst case approximation of how the weight matrices interact with each other . For example , the product of the spectral norms arises from a naive approximation of the Lipschitz constant of the neural network , which would hold only when the singular values of the 1To understand why these values are of this order in magnitude , consider the initial matrix that is randomly initialized with independent entries with variance 1/ √ H . It can be shown that the spectral norm of this matrix , with high probability , lies near its expected value , near 2 and the Frobenius norm near its expected value which is √ H . Since SGD is observed not to move too far away from the initialization regardless of H ( Nagarajan & Kolter , 2017 ) , these values are more or less preserved for the final weight matrices . weight matrices all align with each other . However , in practice , for most inputs to the network , the interactions between the activated weight matrices are not as adverse . By using noise-resilience of the networks , prior approaches ( Arora et al. , 2018 ; Neyshabur et al. , 2017 ) have been able to derive bounds that replace the above worst-case approximation with smaller terms that realistically capture these interactions . However , these works are limited in critical ways . Arora et al . ( 2018 ) use noise-resilience of the network to modify and “ compress ” the parameter representation of the network , and derive a generalization bound on the compressed network . While this bound enjoys a better dependence on depth because its applies to a compressed network , the main drawback of this bound is that it does not apply on the original network . On the other hand , Neyshabur et al . ( 2017 ) take advantage of noise-resilience on training data by incorporating it within a PAC-Bayesian generalization bound ( McAllester , 1999a ) . However , their final guarantee is only a bound on the expected test loss of a stochastic network . In this work , we revisit the idea in Neyshabur et al . ( 2017 ) , by pursuing the PAC-Bayesian framework ( McAllester , 1999a ) to answer this question . The standard PAC-Bayesian framework provides generalization bounds for the expected loss of a stochastic classifier , where the stochasticity typically corresponds to Gaussian noise injected into the parameters output by the learning algorithm . However , if the classifier is noise-resilient on both training and test data , one could extend the PAC-Bayesian bound to a standard generalization guarantee on the deterministic classifier . Other works have used PAC-Bayesian bounds in different ways in the context of neural networks . Langford & Caruana ( 2001 ) ; Dziugaite & Roy ( 2017 ) optimize the stochasticity and/or the weights of the network in order to numerically compute good ( i.e. , non-vacuous ) generalization bounds on the stochastic network . Neyshabur et al . ( 2018 ) derive generalization bounds on the original , deterministic network by working from the PAC-Bayesian bound on the stochastic network . However , as stated earlier , their work does not make use of noise resilience in the networks learned by SGD . OUR CONTRIBUTIONS The key contribution in our work is a general PAC-Bayesian framework for deriving generalization bounds while leveraging the noise resilience of a deep network . While our approach is applied to deep networks , we note that it is general enough to be applied to other classifiers . In our framework , we consider a set of conditions that when satisfied by the network , makes the output of the network noise-resilient at a particular input datapoint . For example , these conditions could characterize the interactions between the activated weight matrices at a particular input . To provide a generalization guarantee , we assume that the learning algorithm has found weights such that these conditions hold for the weight interactions in the network on training data ( which effectively implies a wide training loss minimum ) . Then , as a key step , we generalize these conditions over to the weight interactions on test data ( which effectively implies a wide test loss minimum ) 2 . Thus , with the guarantee that the classifier is noise-resilient both on training and test data , we derive a generalization bound on the test loss of the original network . Finally , we apply our framework to a specific set up of ReLU based feedforward networks . In particular , we first instantiate the above abstract framework with a set of specific conditions , and then use the above framework to derive a bound on the original network . While very similar conditions have already been identified in prior work ( Arora et al. , 2018 ; Neyshabur et al. , 2017 ) ( see Appendix G for an extensive discussion of this ) , our contribution here is in showing how these conditions generalize from training to test data . Crucially , like these works , our bound does not have severe exponential dependence on depth in terms of products of spectral norms . We note that in reality , all but one of our conditions on the network do hold on training data as necessitated by the framework . The strong , non-realistic condition we make is that the pre-activation values of the network are sufficiently large , although only on training data ; however , in practice a small proportion of the pre-activation values can be arbitrarily small . Our generalization bound scales inversely with the smallest absolute value of the pre-activations on the training data , and hence in practice , our bound would be large . 2Note that we can not directly assume these conditions to hold on test data , as that would be ‘ cheating ’ from the perspective of a generalization guarantee . Intuitively , we make this assumption to ensure that under sufficiently small parameter perturbations , the activation states of the units are guaranteed not to flip . It is worth noting that Arora et al . ( 2018 ) ; Neyshabur et al . ( 2017 ) too require similar , but more realistic assumptions about pre-activation values that effectively assume only a small proportion of units flip under noise . However , even under our stronger condition that no such units exist , it is not apparent how these approaches would yield a similar bound on the deterministic , uncompressed network without generalizing their conditions to test data . We hope that in the future our work could be developed further to accommodate the more realistic conditions from Arora et al . ( 2018 ) ; Neyshabur et al . ( 2017 ) .

Summary: This paper presents a PAC-Bayesian framework that bounds the generalization error of the learned model. While PAC-Bayesian bounds have been studied before, the focus of this paper is to study how different conditions in the network (e.g. behavior of activations) generalize from training set to the distribution. This is important since prior work have not been able to handle this issue properly and as a consequence, previous bounds are either on the networks with perturbed weights or with unrealistic assumptions on the behavior of the network for any input in the domain.