Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks

1 INTRODUCTION . In the past decade , the emergence of practical deep neural networks arguably has had disruptive impact on applications of machine learning . Depth as such appears to be key to expressive models ( Raghu et al. , 2017 ) . However , depth also comes with challenges concerning training stability . Theoretical problems include vanishing and exploding gradients ( Hochreiter , 1991 ) , chaotic feedforward dynamics ( Poole et al. , 2016 ) , or decorrelation of gradients ( Balduzzi et al. , 2017 ) . In practice , a number of “ recipes ” are widely used , such as specific nonlinearities ( Glorot et al. , 2011 ; He et al. , 2015 ) , normalization methods such as batch normalization ( Ioffe & Szegedy , 2015 ) , shortcut architectures ( Srivastava et al. , 2015 ; He et al. , 2016a ; b ) , or multi-path architecture with ( Huang et al. , 2017 ) and without shortcuts ( Szegedy et al. , 2016 ) . Broadly speaking , a key research question is to understand how the shape of the network function , i.e. , the map from inputs and parameters to outputs , is affected by architectural choices . Our paper considers specifically the roughness of the weights-to-outputs function ( “ w-o function ” ) of nonlinear feed-forward networks . Motivated by the recent visualizations of ( Li et al. , 2018 ) , which show how depth increases roughness and residual connections smoothen the output again , our goal is to provide an analytical explanation of this effect , and study its implications on network design and trainability . To this end , we first formalize “ roughness ” as the decay-rate of the expected power spectrum of a function class . Our main contribution is to then apply harmonic distortion analysis to nonlinear feedforward networks , which predicts the creation of high-frequency “ harmonics ” ( thereby “ blueshifting ” the power spectrum ) by polynomial nonlinearities with large higher-order coefficients . Based on this model , we discuss how network depth increases blueshift and thus roughness , while shortcut connections , low-degree nonlinearities and parallel computation paths dampen it . In relation to trainability , we show an analytic link between blueshift and exploding gradients . Unlike the former model , the spectral view describes a more global behavior of the w-o function over regions in the parameter domain . Experiments confirm the theoretical predictions : We observe the predicted effects of depth , shortcuts and parallel computation on blueshift , and are able to differentiate different types of nonlinearities by the decay rate of coefficients of a polynomial approximation . The findings are in-line with known advantages in trainability of the different architectures . We further strengthen the evidence by training a large set of networks with a different amount of nonlinearity and depth , which shows a clear correlation between blueshift and training-problems , as well as a trade-off with expressivity . In summary , our paper explains how network architecture affects roughness , shows a connection to trainability , and thereby provides a new tool for analyzing the design of deep networks . 2 RELATED WORK . Vanishing or exploding gradients are a central numerical problem ( Hochreiter , 1991 ; Pascanu et al. , 2013 ; Yang et al. , 2019 ) : If the the magnitudes of the singular values of layer Jacobians deviates from one , subspaces are attenuated ( |σ| < 1 ) or amplified ( |σ| > 1 ) , potentially cascading exponentially over multiple layers ( Pennington et al. , 2017a ) . Formally , the behavior of stacks of matrices and nonlinear functions can be modeled by random matrix theory or Gaussian mean-field approximations ( Poole et al. , 2016 ; Pennington et al. , 2017b ; 2018 ) . The gist is that at initialization , orthogonal weight matrices are needed , which is challenging for convolutional architectures . A solution for tanh-networks is given by Xiao et al . ( 2018 ) ; for ReLU , there is a negative result ( Pennington et al. , 2017b ) . Using mean-field theory , it can be shown that batch normalization ( Ioffe & Szegedy , 2015 ) leads to exploding gradients at initialization ( Yang et al. , 2019 ) ( which equalize after a few steps , but that might be too late ( Frankle et al. , 2020 ) ) . A different route is taken by Balduzzi et al . ( 2017 ) , who observe an increasing decorrelation of gradients in the input space . Similar to our paper , they show that deeper networks lead to spectral whitening ( starting from brown noise ) ; however , the analysis is performed with respect to the inputs x , not weights W. The scale-space structure shown by our model might give further hints on the mechanisms behind training difficulties . By visualizing random slices of the loss surface , Li et al . ( 2018 ) observe that the loss surface of deep feedforward networks transitions between nearly convex to chaotic with increasing depth ; our work explains these observations by spectral analysis . Duvenaud et al . ( 2014 ) visualize pathologies on the landscape of deep gaussian processes that model deep wide-limit nonlinear networks . Fourier analysis of network functions ( Candès , 1999 ) wrt . input ( Rahaman et al. , 2019 ; Xu et al. , 2019 ; Xu , 2018 ; Basri et al. , 2019 ; Yang & Salman , 2019 ) has been used to show an inductive bias towards low-frequency functions ( wrt . input x ) , as well as a strong anisotropy of this spectrum . Wang et al . ( 2020 ) prove under some assumptions that all `` bad '' local minima of a deep residual network are very shallow . 3 HARMONIC DISTORTION . We now analyze the effect of a nonlinearity by relating the Fourier spectrum of a preactivation with that of its postactivation . Let f denote the preactivation of a single neuron of a neural network consisting of L-layers . We use x to denote the input to the whole network and thus to f , W to denote the weights , and φ to denote the employed nonlinearity . Li et al . ( 2018 ) visualize “ roughness ” using random 2D-slices in weight space . We follow their basic idea and consider 1D slices p ( t ) = f ( x , W + α−1 · t ·D ) ( 1 ) for random directions D and t ∈ [ 0 , 1 ] . D is initialized with entries from N0,1 and normalized to ||D||F = 1. α determines the path length . By varying D , this samples a ball of radius α around a point W in parameter space . For φ = id , the network f is multi-linear in W and thus p is polynomial in t ; empirically , this yields rather smooth functions ( Fig . 1 ) . To understand general nonlinearities φ better , we represent p by a complex Fourier series ( Appendix B.1 discusses convergence and approximation quality ) : p ( t ) = ∞∑ k=−∞ zk exp ( 2πikt ) , zk ∈ C. ( 2 ) Here , the sequence z : Z→ C contains the Fourier coefficients for p : [ 0 , 1 ] → R. As p is real , z is symmetric in the sense of zk = z−k . 3.1 FORMALIZING ROUGHNESS . The roughness of a class of random functions can be characterized by the statistics of their powerspectrum ( Musgrave , 1993 ) : Given a random process that generates functions p , we consider mean µk and variance σ2k of the Fourier coefficients zk . For paths with short length as used in our experiments , the means µk are empirically very close to zero except for z0 ( `` DC '' coefficient ) , which is excluded in all experiments . Therefore , we can focus on variance : In general , functions where the variance of high-frequency components drop off more quickly will appear smoother . A common model , which often fits natural data well , is fractal Brownian motion ( FBM- ) noise , where the σk drop off according to a power law : σk ∼ O ( 1/kh ) for some h > 0 . ( 3 ) The so-called “ fractal coefficient ” h describes the roughness of the noise function . A similar approach has been taken by Hoffer et al . ( 2017 ) , who modeled the loss surface by analyzing the dynamics of a random walk on a random potential . In our experiments , we estimate the average power-spectrum ED ( |zk|2 ) ( by sampling D uniformly on a unit sphere ) and fit a power-law to these spectra in order to quantify the roughness in a single number h. Experiments ( Section 4 ) and analytical arguments ( Appendix B.2 ) , show that the FBM/power-law model is a realistic model of the functions computed by the lower layers of a neural network . For higher layers the fit becomes worse , a phenomenon we will explore in the next chapter using harmonic distortion analysis . 3.2 WHY IS THE OUTPUT FUNCTION GETTING ROUGHER ? . Intuitively , applying ReLU to a function p is reminiscent of clipping an audio signal in amplitude , which is known to produce high-frequent ringing artifacts . We describe the effect of a single nonlinearity φ on the spectrum of a preactivation p ; Inductively , this describes the spectral shift of the whole network . For the analysis , we assume that φ is a K-th order polynomial : φ ( x ) = K∑ j=0 ajx j . ( 4 ) The effect of polynomial nonlinear maps on the spectrum of a function can be understood by harmonic distortion analysis ( see e.g . Feynman et al . ( 1965 ) , Ch . 50.8 ) . We can simply ( see Appendices B.1 – B.3 for details ) plug the Fourier expansion of p into the polynomial representation of φ : φ ( p ( t ) ) = K∑ j=0 aj [ ∞∑ k=−∞ zk exp ( 2πikt ) ] j . ( 5 ) The convolution theorem tells us that j-th power of functions corresponds to convolving the spectrum z of the function j-times with itself . We designate by z the vector containing all Fourier coefficients zk and by ⊗ the convolution operator . We can then write the spectrum of the output z′ as : z′ : = F ( φ ( p ) ) = K∑ j=0 aj j⊗ 1=1 z . ( 6 ) Discussion : We can make three important observations : • First , each repeated auto-convolutions in Eq . 6 broadens the spectrum by adding higher-frequency terms . We call this broadening effect blueshift . The exact magnitude is hard to quantify and thus left to the experiments ( Appendix B.4 gives informal arguments for a growth of O ( j1/2 ) rather than the trivial upper bound of O ( j ) ) . • Second and correspondingly , larger coefficients aj for larger orders j , increase the blueshift . • Third , j-fold convolutions correspond to j-th powers of the zk . Hence , larger magnitudes |zk| also increase the blueshift ( the nonlinearity becomes more visible with larger magnitude ) . 3.3 COMMON NONLINEARITIES . In practice , nonlinearities are usually not polynomial , and might not even have a globally convergent Taylor series . It is possible to approximate any continuous function by a polynomial ( Stone-Weierstrass theorem ) . We conjecture that a close polynomial approximation will be sufficient for a qualitative prediction of the blueshift effect ( our theoretical model does not guarantee convergence – we therefore validate this claim experimentally ) . We employ a Chebyshef approximation ( which is reasonably stable and non-oscillatory ) on the interval [ −5 , 5 ] and compare the speed at which higher order coefficients drop off : Fig . 2 shows the coefficient magnitudes for ReLU , tanh and softplus ( which we focus on in our experiments ) . softplus has the strongest drop-off , followed by tanh and ReLU . Appendix F.1 gives more details and covers several additional popular nonlinearities ( Figures 18 , 19 ) . Figure 3 , 17 show that this correlates with blueshift , as expected .

Summary: This paper proposes a new approach for how to analyze the ruggedness of the surface of the neural network loss. Specifically, the paper proposes to apply harmonic distortion on the weight-to-output (w-o) maps. That is, the method casts the w-o functions in the Fourier domain and then aggregate the surface characteristics by virtue of averaging the different order Fourier coefficients. The  paper shows that non-linearities are responsible for blueshifting with deeper layers, that is for "transferring more energy" on the higher frequencies. The consequence is rougher surfaces, as well as higher frequencies for gradients, which can lead to exploding gradients in the deeper layers. The remedy is with skip connections and feature averaging, which although are methods already known to improve trainability, the paper corroborates that they also make sense in terms of said approach. The paper conducts various empirical and ablation studies, providing evidence of the claims.