The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

1 INTRODUCTION . Designing neural architectures that perform well can be a difficult process . In particular , the exploding / vanishing gradient problem has been a major challenge for building very deep neural networks at least since the advent of gradient-based parameter learning ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1997 ; Bengio et al. , 1994 ) . However , there is still no consensus about which metric should be used for determining the presence of pathological exploding or vanishing gradients . Should we care about the length of the gradient vector ( He et al. , 2015 ) , or about the size of individual components of the gradient vector ( Schoenholz et al. , 2017 ; Yang & Schoenholz , 2017 ; Glorot & Bengio , 2010 ) , or about the eigenvalues of the Jacobian ( Saxe et al. , 2014 ; Pascanu et al. , 2013 ; Pennington et al. , 2017 ) ? Depending on the metric used , different strategies arise for combating exploding and vanishing gradients . For example , manipulating the width of layers as suggested by e.g . Yang & Schoenholz ( 2018 ) ; Han et al . ( 2017 ) can greatly impact the size of gradient vector components but tends to leave the length of the entire gradient vector relatively unchanged . The popular He initialization for ReLU networks ( He et al. , 2015 ) is designed to stabilize gradient vector length , wheareas the popular Xavier initialization for tanh networks ( Glorot & Bengio , 2010 ) is designed to stabilize the size of gradient vector components . While the papers cited above provide much evidence that gradient explosion / vanishing when defined according to some metrics is associated with poor performance when certain architectures are paired with certain optimization algorithms , it is often unclear how general those results are . We make the following core contributions . 1 . We introduce the nonlinearity coefficient ( NLC ) , a gradient-based measurement of the degree of nonlinearity of a neural network ( section 3 ) . 2 . We show that the NLC , computed in the networks randomly initialized state , is a powerful predictor of test error and that attaining a right-sized NLC is essential for achieving an optimal test error , at least in fully-connected feedforward networks ( section 4 ) . 3 . We explain that , by design , the NLC is not susceptible to a range of confounders that render many other metrics unreliable , such as changes to input scale , width and bias ( section 6 ) . We demonstrate the properties of the NLC via an extensive empirical study covering a wide range of network architectures ( section 4 ) . The scope of our experiments exceeds that of the vast majority of related work . We conduct 43.000 full training runs . As the NLC is also conceptually simple and cheap to compute , it is a useful guide for architecture design and search . Architectures with a sub-optimal NLC can be discarded a priori and computational resources don ’ t have to be spent on training them . The NLC ( defined at the top of page 3 ) combines the Frobenius norm of the Jacobian of a neural network with the global variability of the input data and the global variability of the network ’ s outputs into a single metric . Despite its simplicity , it is tied to many important properties of the network . It is a remarkably accurate predictor of the network ’ s nonlinearity as measured by the relative diameter of the regions in input space that can be well-approximated by a linear function ( section 3 and figure 1 ) . It is closely related to the nonlinearity of the individual activation functions used in the network and the degree to which they can be approximated by a linear function ( section 5 ) . It is tied to the susceptibility of the network ’ s output to small random input perturbations . 2 NOTATION AND TERMINOLOGY . We define a neural network f as a function of the input x . Both x and the output f ( x ) are vectors of fixed dimensionality din and dout respectively . We assume a prediction framework , where the output is considered to be the prediction and the goal is to minimize the value of the ‘ error ’ e over this prediction and the label y , in expectation over some data distribution D . I.e. , we wish to minimize E ( x , y ) ∼D [ e ( f ( x ) , y ) ] . In this paper , e is always classification accuracy . During training , we replace D with the training set and ewith the surrogate loss function ` , which in this paper is always softmax plus cross-entropy . Let J ( x ) : = df ( x ) dx be the Jacobian of the output with respect to the input . x ( i ) and f ( x , j ) denote the i ’ th component of the input vector and the j ’ th component of the output vector respectively , where 1 ≤ i ≤ din and 1 ≤ j ≤ dout . Denote the component-wise standard deviation of a ( possibly multivariate ) random variable X as S [ X ] . Finally , let the ‘ quadratic expectation ’ Q of a random variable X be defined as Q [ X ] = √ ( E [ X2 ] ) , i.e . the generalization of the quadratic mean to random variables . 3 ON THE NONLINEARITY OF NEURAL NETWORKS - DERIVING THE NLC . Consider an arbitrary neural network f and assume it has a well-defined bounded domain D and bounded co-domain F. Then the Jacobian J taken at some x ∈ D defines a local linear approximation of the function f around x , i.e . f ( x + δ ) ≈ f ( x ) + J ( x ) δ for sufficiently small δ . The key insight behind the NLC and ultimately behind this paper is that there is a simple criterion for determining whether the approximation can be accurate for a given δ : If f ( x ) + J ( x ) δ is far away from F , then because f ( x+ δ ) ∈ F , f ( x ) +J ( x ) δ is far away from f ( x+ δ ) and thus it is inaccurate . We can use this insight to establish an approximate bound for the size of δ by referencing a well-known property of the Frobenius norm . Proposition 1 . Let A be an m×n matrix and u a random n-dimensional vector of fixed length and uniformly random orientation . Then ||A||F√ n ||u||2 = Qu||Au||2 . ( See section F for the short proof . ) As we also have f ( x ) ∈ F , we deduce that we must have ||J ( x ) ||F√ din ||δ||2 / diameter ( F ) for the local linear approximation at x to be accurate for a randomly oriented δ . In fact , an approximate bound for the diameter of the linearly approximable region at x in a random direction is √ dindiameter ( F ) ||J ( x ) ||F . How- ever , the size of this diameter by itself is not sufficient to judge the nonlinearity of f , because it does not take into account the size of the domain D. If D fits entirely within the linearly approximable region , then f is close to a linear function . If D is large compared to the linearly approximable region , we consider f highly nonlinear at x . Hence , we consider instead the approximate bound for the relative diameter of the linearly approximable region in a random direction , √ dindiameter ( F ) ||J ( x ) ||F diameter ( D ) . We posit the inverse of this value , ||J ( x ) ||F diameter ( D ) √ dindiameter ( F ) , as our nonlinearity metric for f at x . Of course , in practice , a network might not have a well-defined , bounded domain and co-domain . Here we make a final modeling assumption . We proxy the diameter with the quadratic expectation of the distance of two random points from the data distribution . Thus our nonlinearity estimate becomes ||J ( x ) ||FQx , x′∼D||x−x ′||2√ dinQx , x′∼D||f ( x ) −f ( x′ ) ||2 . And as we further show in section F , the quadratic expectation of this equals the NLC as defined below . Definition 1 . The nonlinearity coefficient ( NLC ) of a network f and inputs x ∼ D is NLC ( f , D ) : = ( Qx||J ( x ) ||F ) ( Qi ( Sxx ( i ) ) ) √ doutQj ( Sxf ( x , j ) ) . The terms Sxx ( i ) and Sxf ( x , j ) denote the standard deviation of the activation of the i ’ th input neuron and j ’ th output neuron under the data distribution respectively . Qi ( Sxx ( i ) ) and Qj ( Sxf ( x , j ) ) denote the quadratic expectation of these values across all input / output neurons respectively . We also have Qi ( Sxx ( i ) ) = √ Tr ( Cx ) din and Qj ( Sxf ( x , j ) ) = √ Tr ( Cf ) dout , where Cx and Cf are the covariance matrices of x and f ( x ) under the data distribution . Hence , the NLC can be re-written as√ Tr ( ExJJ T ) Tr ( Cx ) din Tr ( Cf ) . As a sanity check , let ’ s look at the NLC of a linear network . In that case , f ( x ) = Ax + b for some A and b . Then the NLC becomes √ Tr ( AAT ) Tr ( Cx ) din Tr ( ACxAT ) . This value equals 1 , for example , if Cx is a multiple of the identity or A is orthogonal . We further conjecture that this value is close to 1 for large , random matrices A as they occur in practice in randomly initialized neural networks , though this analysis goes beyond the scope of this paper . Note that an alternative interpretation of the NLC is that it represents the expected sensitivity of the network output with respect to small , randomly oriented changes to the input , normalized by the global variability of the input , output and √ dout . Finally , we refer readers interested in a pictorial illustration of the NLC to section A in the appendix . Computing the NLC It is worth noting that the NLC is cheap to ( approximately ) compute . Throughout this study , we compute Qi ( Sxx ( i ) ) and Qj ( Sxf ( x , j ) ) exactly via a single pass over the training set . Qx||J ( x ) ||F can be computed stochastically by backpropagating Gaussian random vectors from the output layer . See section G for details . 4 ON THE PREDICTIVE POWER OF THE NLC - LARGE-SCALE STUDY . Now we investigate the empirical properties of the NLC through a large-scale study . Architectures used We sampled architectures at random by varying the depth of the network , the scale of initial weights , scale of initial bias , activation function , normalization method , presence of skip connections , location of skip connections and strength of skip connections . We chose from a set of 8 activation functions ( table 1 ) , which were further modified by random dilation , lateral shift and debiasing . For now , we only considered fully-connected feedforward networks , as is ( perhaps unfortunately ) common in analytical studies of deep networks ( e.g . Saxe et al . ( 2014 ) ; Balduzzi et al . ( 2017 ) ; Schoenholz et al . ( 2017 ) ) . We have no reasons to suspect our results will not generalize to CNNs , and we plan to investigate this point in future work . See section C for the full details of our architecture sampling scheme . Datasets used We studied three datasets : MNIST , CIFAR10 and waveform-noise . All our results were highly consistent across these datasets . waveform-noise is from the UCI repository of datasets popular for evaluating fully-connected networks ( Klambauer et al. , 2017 ) . See section D for further details on dataset selection and preprocessing . We sampled 250 architectures per dataset , a total of 750. waveform-noise ( C ) 1 MNIST ( B ) 1 CIFAR10 ( A ) 1 Training protocol We trained each architecture with SGD with 40 different starting learning rates and selected the optimal one via a held-out validation set , independently for each architecture . During each run , we reduced the learning rate 10 times by a factor of 3 . All training runs were conducted in 64 bit floating point . See section E for further experimental details and section B.1 for an analysis of how the learning rate search and numerical precision contributed to the outcome of our study . Presentation of results The results of this study are shown in figures 1 , 2 , 3 , 4 , 7 , 8 , 9 and 10 . All figures except figure 7 are scatter plots where each point corresponds to a single neural architecture . In most graphs , we show the correlation and its statistical significance between the quantities on the x and y axis at the top . Note that if any quantity is plotted in log-scale , the correlation is also using the log of that quantity . For each architecture , we only studied a single random initialization . Given a limited computational budget , we preferred studying a larger number of architectures instead . Note that all values depicted that were computed after training , such as test error or ‘ NLC after training ’ in figure 3C , are based on the training run with the lowest validation error , as described above . The NLC measures nonlinearity First , we verify that the NLC is indeed an accurate measure of nonlinearity . In figure 1 , we plot the relative diameter of the linearly approximable regions of the network as discussed in section 3 and defined in section E.1 against the NLC in the randomly initialized state . We find a remarkably close match between both quantities . This shows empirically that our informal derivation of the NLC in section 3 leads to remarkably accurate nonlinearity estimates . The NLC predicts test error In figure 2 , we plot the NLC computed in the randomly initialized state , before training , against the test error after training . We find that for all three datasets , the test error is highly related to the NLC . Further , figure 2 indicates that one must start with an NLC in a narrow range , approximately between 1 and 3 , to achieve optimal test error , and the further one deviates from that range , the worse the achievable test error becomes . It is worth noting that some architectures , despite having an NLC in or close to this range , performed badly . One cause of this , high output bias , is explored later in this section . To verify that our results were not dependent on using the SGD optimizer , we re-trained all 250 waveform-noise architectures with Adam using the same training protocol . In figure 3F , we find that the results closely match those of SGD from figure 2C . A caveat is that we do not currently have a way to detect the ideal NLC range for a given dataset , except through observation , though we find this range to be consistent across our three datasets . NLC after training In figure 3B , we plot the value of the NLC before training versus after training . Both values were computed on the training set . We find that for the vast majority of architectures , the value of the NLC decreases . However , if the NLC is very large in the beginning , it remains so . Overall , the before-training NLC significantly predicts the after-training NLC . In figure 3C , we plot the after-training NLC versus test error . We find that unless the NLC lies in a narrow range , test error is close to random . Interestingly , the after-training NLC has a significantly lower correlation with test error than the before-training NLC . NLC before training T es t cl a ss ifi ca ti o n er ro r ( A d a m ) correlation=0.75 , p-value=9.16e-46 waveform-noise ( F ) correlation=-0.83 , p-value=1.87e-116 all datasets ( E ) correlation=0.25 , p-value=0.079 waveform-noise ( D ) correlation=0.55 , p-value=3.43e-21 CIFAR10 ( C ) correlation=0.83 , p-value=3.30e-63 CIFAR10 ( B ) correlation=0.42 , p-value=4.63e-12 CIFAR10 ( A ) 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC after training ( test set ) M a x . er ro rp re se rv in g p er tu rb . ( t es t se t ) 1e91e71e51e31e11e-1 1e1 1e-1 1e-3 1e-5 1e-7 1e-9 NLC before training T ra in in g cl a ss ifi ca ti o n er ro r 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC after training ( training set ) T es t cl a ss ifi ca ti o n er ro r 1e201e151e101e51e0 1 0.8 0.6 0.4 0.2 0 NLC before training N L C a ft er tr a in in g ( t ra in in g se t ) 1e201e151e101e51e0 1e20 1e15 1e10 1e5 1e0 Bias : = Qj , xf ( x , j ) Qj ( Sxf ( x , j ) ) before training T es t cl a ss ifi ca ti o n er ro r 1e151e101e51e0 1 0.8 0.6 0.4 0.2 0 Figure 3 : Detailed results from our empirical study . See main text for explanation and section E.2 ( figures A/B/C/D/F ) and section E.3 ( figure E ) for further details . NLC predicts generalization , not trainability In figure 3D , we show the training error achieved for 50 randomly selected architectures for waveform-noise . We re-trained those architectures without using early stopping based on the validation error and considered an even larger range of starting learning rates . We depict the lowest training classification error that was achieved across all learning rates . Points are shown in red for visibility . We find that independently of the NLC , the vast majority of architectures achieve a zero or near-zero training error . This finding is the opposite of that of Schoenholz et al . ( 2017 ) and Xiao et al . ( 2018 ) , who claim that networks which are highly sensitive to small input perturbations are untrainable . The reason we were able to train these architectures was our extensive learning rate search as well as our decision to train with 64 bit precision . In fact , we found that architectures with high NLC often require very small learning rates and very small parameter updates to train successfully . One architecture required a learning rate as small as 5e-18 ! See section B.1 for further analysis on this point . In summary , we find that many architectures for which the NLC correctly predicts high test error are nonetheless trainable . Of course , one might expect that a high sensitivity leads to poor generalization . As a sanity check , we conducted an experiment where we corrupted the test set with small random perturbations and measured how large these perturbations could be before the test error increased significantly . We plot this in figure 3E . As expected , for the majority of high-NLC architectures , labels can be corrupted and the error increased with incredibly small perturbations . Summary We interpret the results observed so far as follows . To generalize , the network must attain a critical NLC after training . This is only possible in practice if the initial NLC is already close . In that case , the networks often learns automatically to adopt a more ideal NLC . However , unless the initial NLC is itself in the critical range , we can not attain optimal performance . Further predictors : bias and skip connections In figure 2 , we mark in red all points corresponding to architectures that have a very biased output , i.e . 1000Qj ( Sxf ( x , j ) ) < Qj , xf ( x , j ) . We note that all of these architecture attain a high test error , even if their NLC is small . In figure 3A , we plot the output bias before training against test error . We find that indeed , to achieve an optimal test error , a low initial bias is required . In section B.2 , we further show that just as the NLC , the bias also tends to decline throughout training and that attaining a very low bias after training is even more essential . In figure 2 , we plot in blue all points corresponding to architectures that have skip connections . Philipp et al . ( 2018 ) argued that skip connections reduce the gradient growth of general architectures as well as make a further contribution to performance . Correspondingly , we find that skip connections lead to lower NLC values and to lower test errors for a given NLC level . To enable a more convenient visual comparison , we plot results for architectures with and without skip connections separately in section B.3 .

Summary: I do not understand the denomination of nonlinearity coefficient provided in definition 1: although the quantity indeed does equal to 1 under whitened data distribution or orthogonal matrix, the conjecture that it should be close to 1 does not seem to be close at all just under any data distribution. Using a similar construction that section 6, we can rescale a whitened input data with a diagonal matrix D with components all equal to one except for a very large one \lambda and also multiply the input weights by D^{-1} to compensate (and have a similar function). If you look at such construction for the linear case with identity initialization of A, the NLC is sqrt((\lambda^2 + n - 1) (\lambda^{-2} + n - 1)) / n which can grow arbitrarily large with \lambda *for a linear model*. However, because of its low capacity, we would expect a linear model to have reasonable generalization. This seems to compromise the initial NLC being low as a necessary condition for reasonable generalization.