Network-to-Network Regularization: Enforcing Occam's Razor to Improve Generalization

1 Introduction and Motivation . On the surface , the problem of learning to generalize well over unseen data seems an impossible task . Classification is inherently a problem in function estimation , and the finite information that the training data samples impart seems hardly enough to be able to correctly guess the behaviour of the function over the unseen data samples outside the training set . However , assumption of a structured ground truth label function , which is the unknown function which generates the ground truth label for any datapoint , leads to more optimistic outlook on the problem . Without the assumption of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . structuredness in the ground truth , learning is not guaranteed , as was observed in the no free lunch theorem [ 1 ] . As shown in that work , there is no universal learning algorithm which can generalize well for possible choices of ground truth label functions . Since one does not have any control over the true nature of the ground truth function in any classification problem , the other important parameter that decides the ability of a classifier to generalize , is the complexity of its function space itself . Over the past few decades , there have been multiple attempts at bounding the generalization error on the basis of various complexity measures [ 2 , 3 , 4 ] of the classifier ’ s function space F . The overall results of these theoretical developments indicate that generalization is primarily governed by metrics that are proportional to the size of F . Examples of complexity metrics in this regard include Rademacher complexity , VC-Dimension , and Local forms of Rademacher Complexity ( for more tight bounds ) . However , as observed in [ 5 ] these metrics lead to relatively loose bounds on the generalization performance for deep neural networks . This is because deep neural networks have high complexity spaces with flexibility to learn any label assignment on any set of training datapoints . This was perhaps most clearly observed in [ 6 ] , where even after random label assignments on the training data samples , the networks still were able to fit the training data labels with no errors . Thus , often other methods have been explored to establish the relationship between complexity measures and generalization performance [ 7 , 8 ] , such as causal relations . These studies lead to a natural question , which is , other than the metrics which relate to the size of the function space of a classifier , what other factors contribute towards its ability to generalize ? State-of-the-art deep neural networks , as observed above , will yield very high complexity spaces , which does not explain their remarkable ability to generalize well in complex high-dimensional supervised classification problems , such as in vision . We note , that metrics such as Rademacher complexity or VC dimension , which relate to whole function spaces F ( or subspaces within F which fit the training data well ) , essentially assign the same generalization gap to all functions within F ( or the subspace within F ) . However , there has been a longstanding understanding of the fact that usually among all functions which fit the training data well , simpler functions are expected to generalize better . This is an example of the Occam ’ s Razor principle , which states that among all hypotheses that explain a phenomenon , the simplest hypothesis is preferred [ 9 ] . In the context of neural networks , even for very high complexity deep neural network function spaces , there will always be some weight configurations of a deep neural network , which yield simpler functions . For example , all network weight configurations yielding input-output functions which can still be efficiently approximated by smaller , shallower networks , could be considered to be simpler . An extreme example of this would be where we assign all weight values to zero within a deep neural network . The function that results from this weight configuration is essentially the constant function f ( X ) = 0 . This observation points to the fact that even within a function space of a deep CNN , not all functions are equally complex , as some of them can still be approximated by shallower networks . To that end , a primary objective of this paper is to probe complexity measures which enable us to assign a level of complexity to individual functions within a function space . Subsequently , our objective would be to steer the learning process towards network configurations which yield less complex input-output functions . An important work which explored similar directions is [ 10 ] , where metrics from algorithmic complexity were developed to bias the learning process towards simpler functions . In this regard , measures of descriptional complexity [ 5 ] of functions have been proposed over the years , which quantify the level of complexity of an individual function , based on its shortest description . Although fundamental measures of descriptional complexity such as Kolmogorov Complexity [ 11 ] or Solomonoff Probability [ 12 ] are uncomputable , computable approximations to them have been developed [ 13 ] . In spite of this early work , there is a lack of concrete theoretical or empirical work to continue this line of investigation after [ 10 ] . Mainly , there is a lack of work that explores the relevance of such descriptional complexity measures to the generalizational ability of functions . That is , until recently , when interest in the descriptional complexity of a function was renewed as it was found that outputs of random maps tend to be biased towards simpler functions [ 14 ] . This result indirectly hinted that for classification tasks , the ground truth labelling function has a simple description with high probability . This was further investigated in [ 5 ] in the context of deep neural networks , where it was found that the Lempel-Ziv ( LZ ) complexity ( a form of descriptional complexity ) of most neural networks with random choices of weights are low ( shorter description ) , in accordance with the result in [ 14 ] . Empirically , it was found that network weight configurations which lead to functions of smaller Lempel-Ziv complexity show better generalization performance . These results and observations show that descriptional complexity measures may help to understand empirically the generalization behaviour of deep neural networks . In this paper , we advance this line of investigation by proposing a new theoretical framework for relating descriptional complexity measures to generalization performance . Subsequently , we also provide a computable method which improves generalization performance of neural networks by lowering their descriptional complexity . 2 Contributions . This paper makes the following contributions . 1 . First , we undertake a brief theoretical analysis for exploring the relevance of descriptional complexity measures of functions to their expected generalization error . We propose a novel measure of complexity called Kolmogorov Growth ( KG ) . Error bounds are estimated which portray the dependence of generalization error of a single function f toKG ( f ) . The bounds depict that functions of higher KG ( f ) will likely lead to a higher generalization gap . This result formalizes the Occam ’ s razor principle for classifiers , and also concurs with the empirical findings in [ 5 ] , where functions of higher LZ-Complexity showed worse generalization performance . 2 . Like Kolmogorov Complexity , KG is also uncomputable . Therefore , we propose computable approximations of KG for neural networks , based on the concept of teacher-student approximation ( similar to knowledge distillation [ 15 ] ) . Specifically , we show that neural network functions f which can be approximated well by smaller networks will have smaller empirical KG with high probability . 3 . Next , using this idea , we then develop a novel method for regularizing neural networks , called network-to-network ( N2N ) regularization . N2N regularization forces trained network configurations to be of low KG . We find that doing so not only improves the generalization performance across a range of training data sizes , but also helps in the case of label noise . For instance in MNIST , we see test error decrease by 94 % , reaching results competitive with benchmark methods . 4 . Finally , we study the evolution of empirical KG as networks are trained and observe that , in the standard classification scenario , networks show a sharp decrease in their KG as training progresses . However , this trend completely reverses when the training data labels are noise corrupted and N2N regularization is able to stem the undesirable increase of KG during training . 3 Kolmogorov Growth : Relevance to Generalization . Assume that our training data consists of m d-dimensional i.i.d samples and their labels S = [ ( z1 , y1 ) , ( z2 , y2 ) , .. , ( zm , ym ) ] drawn from some distribution Pm . We propose two growth measures for a single function f , namely Kolmogorov Growth KGm ( f ) and empirical Kolmogorov Growth K̂GS ( f ) . Note that , when appropriate , we may drop the subscript m and refer to KGm ( f ) as KG ( f ) . These measures are primarily motivated from the well-known growth function in statistics . The growth function Πm ( F ) is defined as Πm ( F ) = max z∈Rd | { ( f ( z1 ) , f ( z2 ) , ... , f ( zm ) ) : f ∈ F } | ( 1 ) Note that the growth function is defined for a space of functions and captures the maximum number of label assignments a function space F can generate on any m points in Rd . To define Kolmogorov Growth measures for a single function f , we first need to generate a function space from f , based on a description Df of f . This is done via noting all the parameters involved in the description Df ( other than the co-ordinates in Rd ) , and then assigning all possible values to those parameters to generate a space of functions F ( Df ) from the description Df . Next , we note that for a function f , we consider multiple descriptions ( D1f , D 2 f , D 3 f , ... ) , all of which faithfully generate the output of f over all points in Rd , with the additional constraint that each F ( Dif ) should fit any m datapoints sampled from Pm . Given these observations and denominations , we then can define the Kolmogorov Growth of a function f as follows : KGm ( f ) = min i log Πm ( F ( Dif ) ) m . ( 2 ) Note that KGm ( f ) requires the knowledge of distribution Pm of the training samples . For an instance of the datapoints in S , we define the empirical Kolmogorov Growth via the empirical growth function Π̂S ( F ) , which only computes the number of label assignments a function space can generate over the given m training data samples in S. Thus , Π̂S ( F ) = | { ( f ( z1 ) , f ( z2 ) , ... , f ( zm ) ) : f ∈ F } | . This leads to the following definition of the empirical Kolmogorov Growth of a function f : K̂GS ( f ) = min i log Π̂S ( F ( Dif ) ) m . ( 3 ) Remark : Kolmorogov growth is indirectly motivated from Kolmogorov complexity itself . However , unlike Kolmogorov complexity , which is the length of the shortest program that generates f , Kolmogorov Growth is concerned with the smallest function space that f can belong to , that can still fit the data well . Functions which have shorter descriptions usually require a smaller number of variables and are expected to have lower Kolmogorov Growth . Moreover , it turns out that Kolmogorov Growth allows us to directly comment on the error bounds for the function f ( see Section 3.1 ) . We believe that a possible direction of future work would be to do a deeper study of the relationship between Kolmogorov Growth and Kolmogorov Complexity itself . Remark : Note that , in the binary classification scenario , for a completely unstructured function f ( i.e. , f outputs random labels at every point X ∈ Rd ) , one expects KG ( f ) to be near its maximum value ( i.e. , log 2 ) . A structured f would generate shorter descriptions with fewer parameters and therefore lead to smaller KG ( f ) .

Summary: The paper proposes a novel and theoretically inspired regularization techniques for training deep neural networks. The technique involves regulating the network by regressing the output of the model to a simpler model but the simpler model also regresses towards the output of the base model. The experimental results show that the proposed techniques are effective at reducing the generalization gap and improving performance in presence of label noise.