The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

1 INTRODUCTION . Designing neural architectures that perform well can be a difficult process . In particular , the exploding / vanishing gradient problem has been a major challenge for building very deep neural networks at least since the advent of gradient-based parameter learning ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1997 ; Bengio et al. , 1994 ) . However , there is still no consensus about which metric should be used for determining the presence of pathological exploding or vanishing gradients . Should we care about the length of the gradient vector ( He et al. , 2015 ) , or about the size of individual components of the gradient vector ( Schoenholz et al. , 2017 ; Yang & Schoenholz , 2017 ; Glorot & Bengio , 2010 ) , or about the eigenvalues of the Jacobian ( Saxe et al. , 2014 ; Pascanu et al. , 2013 ; Pennington et al. , 2017 ) ? Depending on the metric used , different strategies arise for combating exploding and vanishing gradients . For example , manipulating the width of layers as suggested by e.g . Yang & Schoenholz ( 2018 ) ; Han et al . ( 2017 ) can greatly impact the size of gradient vector components but tends to leave the length of the entire gradient vector relatively unchanged . The popular He initialization for ReLU networks ( He et al. , 2015 ) is designed to stabilize gradient vector length , wheareas the popular Xavier initialization for tanh networks ( Glorot & Bengio , 2010 ) is designed to stabilize the size of gradient vector components . While the papers cited above provide much evidence that gradient explosion / vanishing when defined according to some metrics is associated with poor performance when certain architectures are paired with certain optimization algorithms , it is often unclear how general those results are . We make the following core contributions . 1 . We introduce the nonlinearity coefficient ( NLC ) , a gradient-based measurement of the degree of nonlinearity of a neural network ( section 3 ) . 2 . We show that the NLC , computed in the networks randomly initialized state , is a powerful predictor of test error and that attaining a right-sized NLC is essential for achieving an optimal test error , at least in fully-connected feedforward networks ( section 4 ) . 3 . We explain that , by design , the NLC is not susceptible to a range of confounders that render many other metrics unreliable , such as changes to input scale , width and bias ( section 6 ) . We demonstrate the properties of the NLC via an extensive empirical study covering a wide range of network architectures ( section 4 ) . The scope of our experiments exceeds that of the vast majority of related work . We conduct 43.000 full training runs . As the NLC is also conceptually simple and cheap to compute , it is a useful guide for architecture design and search . Architectures with a sub-optimal NLC can be discarded a priori and computational resources don ’ t have to be spent on training them . The NLC ( defined at the top of page 3 ) combines the Frobenius norm of the Jacobian of a neural network with the global variability of the input data and the global variability of the network ’ s outputs into a single metric . Despite its simplicity , it is tied to many important properties of the network . It is a remarkably accurate predictor of the network ’ s nonlinearity as measured by the relative diameter of the regions in input space that can be well-approximated by a linear function ( section 3 and figure 1 ) . It is closely related to the nonlinearity of the individual activation functions used in the network and the degree to which they can be approximated by a linear function ( section 5 ) . It is tied to the susceptibility of the network ’ s output to small random input perturbations . 2 NOTATION AND TERMINOLOGY . We define a neural network f as a function of the input x . Both x and the output f ( x ) are vectors of fixed dimensionality din and dout respectively . We assume a prediction framework , where the output is considered to be the prediction and the goal is to minimize the value of the ‘ error ’ e over this prediction and the label y , in expectation over some data distribution D . I.e. , we wish to minimize E ( x , y ) ∼D [ e ( f ( x ) , y ) ] . In this paper , e is always classification accuracy . During training , we replace D with the training set and ewith the surrogate loss function ` , which in this paper is always softmax plus cross-entropy . Let J ( x ) : = df ( x ) dx be the Jacobian of the output with respect to the input . x ( i ) and f ( x , j ) denote the i ’ th component of the input vector and the j ’ th component of the output vector respectively , where 1 ≤ i ≤ din and 1 ≤ j ≤ dout . Denote the component-wise standard deviation of a ( possibly multivariate ) random variable X as S [ X ] . Finally , let the ‘ quadratic expectation ’ Q of a random variable X be defined as Q [ X ] = √ ( E [ X2 ] ) , i.e . the generalization of the quadratic mean to random variables . 3 ON THE NONLINEARITY OF NEURAL NETWORKS - DERIVING THE NLC . Consider an arbitrary neural network f and assume it has a well-defined bounded domain D and bounded co-domain F. Then the Jacobian J taken at some x ∈ D defines a local linear approximation of the function f around x , i.e . f ( x + δ ) ≈ f ( x ) + J ( x ) δ for sufficiently small δ . The key insight behind the NLC and ultimately behind this paper is that there is a simple criterion for determining whether the approximation can be accurate for a given δ : If f ( x ) + J ( x ) δ is far away from F , then because f ( x+ δ ) ∈ F , f ( x ) +J ( x ) δ is far away from f ( x+ δ ) and thus it is inaccurate . We can use this insight to establish an approximate bound for the size of δ by referencing a well-known property of the Frobenius norm . Proposition 1 . Let A be an m×n matrix and u a random n-dimensional vector of fixed length and uniformly random orientation . Then ||A||F√ n ||u||2 = Qu||Au||2 . ( See section F for the short proof . ) As we also have f ( x ) ∈ F , we deduce that we must have ||J ( x ) ||F√ din ||δ||2 / diameter ( F ) for the local linear approximation at x to be accurate for a randomly oriented δ . In fact , an approximate bound for the diameter of the linearly approximable region at x in a random direction is √ dindiameter ( F ) ||J ( x ) ||F . How- ever , the size of this diameter by itself is not sufficient to judge the nonlinearity of f , because it does not take into account the size of the domain D. If D fits entirely within the linearly approximable region , then f is close to a linear function . If D is large compared to the linearly approximable region , we consider f highly nonlinear at x . Hence , we consider instead the approximate bound for the relative diameter of the linearly approximable region in a random direction , √ dindiameter ( F ) ||J ( x ) ||F diameter ( D ) . We posit the inverse of this value , ||J ( x ) ||F diameter ( D ) √ dindiameter ( F ) , as our nonlinearity metric for f at x . Of course , in practice , a network might not have a well-defined , bounded domain and co-domain . Here we make a final modeling assumption . We proxy the diameter with the quadratic expectation of the distance of two random points from the data distribution . Thus our nonlinearity estimate becomes ||J ( x ) ||FQx , x′∼D||x−x ′||2√ dinQx , x′∼D||f ( x ) −f ( x′ ) ||2 . And as we further show in section F , the quadratic expectation of this equals the NLC as defined below . Definition 1 . The nonlinearity coefficient ( NLC ) of a network f and inputs x ∼ D is NLC ( f , D ) : = ( Qx||J ( x ) ||F ) ( Qi ( Sxx ( i ) ) ) √ doutQj ( Sxf ( x , j ) ) . The terms Sxx ( i ) and Sxf ( x , j ) denote the standard deviation of the activation of the i ’ th input neuron and j ’ th output neuron under the data distribution respectively . Qi ( Sxx ( i ) ) and Qj ( Sxf ( x , j ) ) denote the quadratic expectation of these values across all input / output neurons respectively . We also have Qi ( Sxx ( i ) ) = √ Tr ( Cx ) din and Qj ( Sxf ( x , j ) ) = √ Tr ( Cf ) dout , where Cx and Cf are the covariance matrices of x and f ( x ) under the data distribution . Hence , the NLC can be re-written as√ Tr ( ExJJ T ) Tr ( Cx ) din Tr ( Cf ) . As a sanity check , let ’ s look at the NLC of a linear network . In that case , f ( x ) = Ax + b for some A and b . Then the NLC becomes √ Tr ( AAT ) Tr ( Cx ) din Tr ( ACxAT ) . This value equals 1 , for example , if Cx is a multiple of the identity or A is orthogonal . We further conjecture that this value is close to 1 for large , random matrices A as they occur in practice in randomly initialized neural networks , though this analysis goes beyond the scope of this paper . Note that an alternative interpretation of the NLC is that it represents the expected sensitivity of the network output with respect to small , randomly oriented changes to the input , normalized by the global variability of the input , output and √ dout . Finally , we refer readers interested in a pictorial illustration of the NLC to section A in the appendix . Computing the NLC It is worth noting that the NLC is cheap to ( approximately ) compute . Throughout this study , we compute Qi ( Sxx ( i ) ) and Qj ( Sxf ( x , j ) ) exactly via a single pass over the training set . Qx||J ( x ) ||F can be computed stochastically by backpropagating Gaussian random vectors from the output layer . See section G for details . 4 ON THE PREDICTIVE POWER OF THE NLC - LARGE-SCALE STUDY . Now we investigate the empirical properties of the NLC through a large-scale study . Architectures used We sampled architectures at random by varying the depth of the network , the scale of initial weights , scale of initial bias , activation function , normalization method , presence of skip connections , location of skip connections and strength of skip connections . We chose from a set of 8 activation functions ( table 1 ) , which were further modified by random dilation , lateral shift and debiasing . For now , we only considered fully-connected feedforward networks , as is ( perhaps unfortunately ) common in analytical studies of deep networks ( e.g . Saxe et al . ( 2014 ) ; Balduzzi et al . ( 2017 ) ; Schoenholz et al . ( 2017 ) ) . We have no reasons to suspect our results will not generalize to CNNs , and we plan to investigate this point in future work . See section C for the full details of our architecture sampling scheme . Datasets used We studied three datasets : MNIST , CIFAR10 and waveform-noise . All our results were highly consistent across these datasets . waveform-noise is from the UCI repository of datasets popular for evaluating fully-connected networks ( Klambauer et al. , 2017 ) . See section D for further details on dataset selection and preprocessing . We sampled 250 architectures per dataset , a total of 750. waveform-noise ( C ) 1 MNIST ( B ) 1 CIFAR10 ( A ) 1 Training protocol We trained each architecture with SGD with 40 different starting learning rates and selected the optimal one via a held-out validation set , independently for each architecture . During each run , we reduced the learning rate 10 times by a factor of 3 . All training runs were conducted in 64 bit floating point . See section E for further experimental details and section B.1 for an analysis of how the learning rate search and numerical precision contributed to the outcome of our study . Presentation of results The results of this study are shown in figures 1 , 2 , 3 , 4 , 7 , 8 , 9 and 10 . All figures except figure 7 are scatter plots where each point corresponds to a single neural architecture . In most graphs , we show the correlation and its statistical significance between the quantities on the x and y axis at the top . Note that if any quantity is plotted in log-scale , the correlation is also using the log of that quantity . For each architecture , we only studied a single random initialization . Given a limited computational budget , we preferred studying a larger number of architectures instead . Note that all values depicted that were computed after training , such as test error or ‘ NLC after training ’ in figure 3C , are based on the training run with the lowest validation error , as described above . The NLC measures nonlinearity First , we verify that the NLC is indeed an accurate measure of nonlinearity . In figure 1 , we plot the relative diameter of the linearly approximable regions of the network as discussed in section 3 and defined in section E.1 against the NLC in the randomly initialized state . We find a remarkably close match between both quantities . This shows empirically that our informal derivation of the NLC in section 3 leads to remarkably accurate nonlinearity estimates . The NLC predicts test error In figure 2 , we plot the NLC computed in the randomly initialized state , before training , against the test error after training . We find that for all three datasets , the test error is highly related to the NLC . Further , figure 2 indicates that one must start with an NLC in a narrow range , approximately between 1 and 3 , to achieve optimal test error , and the further one deviates from that range , the worse the achievable test error becomes . It is worth noting that some architectures , despite having an NLC in or close to this range , performed badly . One cause of this , high output bias , is explored later in this section . To verify that our results were not dependent on using the SGD optimizer , we re-trained all 250 waveform-noise architectures with Adam using the same training protocol . In figure 3F , we find that the results closely match those of SGD from figure 2C . A caveat is that we do not currently have a way to detect the ideal NLC range for a given dataset , except through observation , though we find this range to be consistent across our three datasets . NLC after training In figure 3B , we plot the value of the NLC before training versus after training . Both values were computed on the training set . We find that for the vast majority of architectures , the value of the NLC decreases . However , if the NLC is very large in the beginning , it remains so . Overall , the before-training NLC significantly predicts the after-training NLC . In figure 3C , we plot the after-training NLC versus test error . We find that unless the NLC lies in a narrow range , test error is close to random . Interestingly , the after-training NLC has a significantly lower correlation with test error than the before-training NLC . NLC before training T es t cl a ss ifi ca ti o n er ro r ( A d a m ) correlation=0.75 , p-value=9.16e-46 waveform-noise ( F ) correlation=-0.83 , p-value=1.87e-116 all datasets ( E ) correlation=0.25 , p-value=0.079 waveform-noise ( D ) correlation=0.55 , p-value=3.43e-21 CIFAR10 ( C ) correlation=0.83 , p-value=3.30e-63 CIFAR10 ( B ) correlation=0.42 , p-value=4.63e-12 CIFAR10 ( A ) 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC after training ( test set ) M a x . er ro rp re se rv in g p er tu rb . ( t es t se t ) 1e91e71e51e31e11e-1 1e1 1e-1 1e-3 1e-5 1e-7 1e-9 NLC before training T ra in in g cl a ss ifi ca ti o n er ro r 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC after training ( training set ) T es t cl a ss ifi ca ti o n er ro r 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC before training N L C a ft er tr a in in g ( t ra in in g se t ) 1e201e151e101e51e0 1e20 1e15 1e10 1e5 1e0 Bias : = Qj , xf ( x , j ) Qj ( Sxf ( x , j ) ) before training T es t cl a ss ifi ca ti o n er ro r 1e151e101e51e0 1 0.8 0.6 0.4 0.2 0 Figure 3 : Detailed results from our empirical study . See main text for explanation and section E.2 ( figures A/B/C/D/F ) and section E.3 ( figure E ) for further details . NLC predicts generalization , not trainability In figure 3D , we show the training error achieved for 50 randomly selected architectures for waveform-noise . We re-trained those architectures without using early stopping based on the validation error and considered an even larger range of starting learning rates . We depict the lowest training classification error that was achieved across all learning rates . Points are shown in red for visibility . We find that independently of the NLC , the vast majority of architectures achieve a zero or near-zero training error . This finding is the opposite of that of Schoenholz et al . ( 2017 ) and Xiao et al . ( 2018 ) , who claim that networks which are highly sensitive to small input perturbations are untrainable . The reason we were able to train these architectures was our extensive learning rate search as well as our decision to train with 64 bit precision . In fact , we found that architectures with high NLC often require very small learning rates and very small parameter updates to train successfully . One architecture required a learning rate as small as 5e-18 ! See section B.1 for further analysis on this point . In summary , we find that many architectures for which the NLC correctly predicts high test error are nonetheless trainable . Of course , one might expect that a high sensitivity leads to poor generalization . As a sanity check , we conducted an experiment where we corrupted the test set with small random perturbations and measured how large these perturbations could be before the test error increased significantly . We plot this in figure 3E . As expected , for the majority of high-NLC architectures , labels can be corrupted and the error increased with incredibly small perturbations . Summary We interpret the results observed so far as follows . To generalize , the network must attain a critical NLC after training . This is only possible in practice if the initial NLC is already close . In that case , the networks often learns automatically to adopt a more ideal NLC . However , unless the initial NLC is itself in the critical range , we can not attain optimal performance . Further predictors : bias and skip connections In figure 2 , we mark in red all points corresponding to architectures that have a very biased output , i.e . 1000Qj ( Sxf ( x , j ) ) < Qj , xf ( x , j ) . We note that all of these architecture attain a high test error , even if their NLC is small . In figure 3A , we plot the output bias before training against test error . We find that indeed , to achieve an optimal test error , a low initial bias is required . In section B.2 , we further show that just as the NLC , the bias also tends to decline throughout training and that attaining a very low bias after training is even more essential . In figure 2 , we plot in blue all points corresponding to architectures that have skip connections . Philipp et al . ( 2018 ) argued that skip connections reduce the gradient growth of general architectures as well as make a further contribution to performance . Correspondingly , we find that skip connections lead to lower NLC values and to lower test errors for a given NLC level . To enable a more convenient visual comparison , we plot results for architectures with and without skip connections separately in section B.3 .

Summary: In this paper the authors introduce a new quantity, the nonlinearity coefficient, and argue that its value at initialization is a useful predictor of test time performance for neural networks. The authors conduct a wide range of experiments over many different network architectures and activation functions to corroborate these results. The authors then extend their method to compute the local nonlinearity of activation functions instead.