A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions

1 INTRODUCTION . It is undeniable that in recent years deep learning systems have found widespread success in their applications to a diverse and ever-expanding set of domains . The foundational results on many tasks such as image recognition ( Krizhevsky et al. , 2012 ) , speech recognition ( Hinton et al. , 2012 ) , and machine translation ( Wu et al. , 2016 ) , have begun to make their way into higher-level products that people interact with and rely upon in their daily lives . Whether these products generate a medical diagnosis , a navigation decision , or some other important output , it is crucial to understand the inner-workings of the deep learning algorithms that generate them . Unfortunately , our theoretical understanding of these deep learning algorithms continues to lag behind their impressive practical successes . One main challenge in building a fuller understanding stems from the fact that deep neural networks are complex nonlinear functions that employ millions or even billions ( Shazeer et al. , 2017 ) of parameters . Traditional wisdom would suggest that to this parameter complexity corresponds to an optimization difficulty . Recent work , however , suggests that as the width of a network ’ s hidden layers becomes large , the loss function simplifies and a theoretical analysis becomes tractable ( Jacot et al. , 2018 ; Mei et al. , 2018 ; Chizat & Bach , 2018a ; Mei et al. , 2019 ; Rotskoff et al. , 2019 ; Rotskoff & Vanden-Eijnden , 2018 ) . In some scenarios , the simplification is such that throughout training the parameters of the model stay within an infinitesimal radius of their initial ( random ) values , implying that much about neural network training can be understood by studying the random initialization ( Jacot et al. , 2018 ; Chizat & Bach , 2018b ; Lee et al. , 2019 ) . Another main challenge in building a rich understanding of deep learning systems stems from the fact that they are often trained on very large , complex datasets : even if the models themselves are very large , they may not be large in comparison to the number of constraints they are designed to satisfy . Indeed , many important phenomena may become apparent only by examining the high-dimensional regime in which the dataset size and width are both large and of the same order . In this work , we focus on the high-dimensional regime and analyze the performance of a regression model trained on the random features F = f ( WX +B ) for a random weight matrix W and random bias vector B . We obtain an exact formula for the training error on a noisy autoencoding task in the limit that the width and dataset size both go to infinity . The result is determined by the resolvent of the kernel matrix F tF , whose properties we analyze via the resolvent method from random matrix theory . Our analysis also provides an exact formula for the eigenvalue density of the kernel matrix , which may be of independent interest since it provides a characterization for how spectral properties of the data covariance matrix propagate through neural network layers at initialization . 1.1 OUR CONTRIBUTIONS . The main contribution of our work is an exact characterization of the training error of a ridgeregularized random feature regression model on a noisy autoencoder task in the high-dimensional regime . This is one of the first non-trivial models to be solved exactly in the joint limit of large data and large width and provides an interesting testing ground in which to analyze this regime . Some of our additional contributions include , • An exact characterization of the spectral density of the randm feature matrix F = f ( WX + B ) , extending prior results of Pennington & Worah ( 2017 ) to non-Gaussian data distributions and to non-zero bias distributions . • One interpretation of the random additive bias is that it induces a distribution of activation functions parameterized by B , i.e . f ( Z ; B ) : = f ( Z +B ) . Our analysis trivially extends to any distribution of activation functions f ( · ; B ) parameterized by B . • We show that there exists a non-trivial distribution over activation functions that outperforms the best possible single activation function on a noisy autoencoding task . • Our method of proof introduces a surrogate “ linearization '' of F , F lin , that possesses the same spectral information as F . F lin and its properties are likely to be of further interest and utility in analyzing neural networks in high dimensions . 1.2 RELATED WORK . Neural networks have been studied from the perspective of high-dimensional statistics in a number of recent works . Most prior work has focused on the bias-free case . Pennington & Worah ( 2017 ) studied the spectrum of the activation matrix f ( WX ) for iid Gaussian data and derived an analytic expression for the training error of a ridge-regularized random feature model trained on pure noise . It is natural to consider incorporating biases by appending a constant feature 1 to X . Unfortunately , this leads to biases that are the same order as the weights , and so the effect disappears in the large dataset limit . Moreover , this modification on the data violates the assumptions of Pennington & Worah ( 2017 ) . Hastie et al . ( 2019 ) study ridgeless interpolation in high-dimensional interpolation for linear features as well as nonlinear random features of iid Gaussian data . Louart et al . ( 2018 ) derived a deterministic equivalent for the resolvent of the kernel matrix F tF which allowed for a characterization of the asymptotic training and test performance of linear ridge regression of random feature models . Other work has investigated learning dynamics and generalization in the high-dimensional regime ( Liao & Couillet , 2018a ; Lampinen & Ganguli , 2018 ; Advani & Ganguli , 2016 ; Advani & Saxe , 2017 ) as well as the spectra of more complicated objects such as the Hessian ( Pennington & Bahri , 2017 ) and Fisher information matrix ( Pennington & Worah , 2018 ) . From the mathematical perspective , random matrix theory provides natural tools ( Silverstein & Bai , 1995a ) for analyzing the behavior of neural networks in the high-dimensional regime . Liao & Couillet ( 2018b ) examined spectra for data drawn from Gaussian mixture models ; see also ( El Karoui , 2010 ) on the spectra of random kernel matrices . 2 PRELIMINARIES . Consider a dataset X ∈ Rn0×m and the random feature matrix , F = f ( WX ; B ) , generated by a single hidden-layer network with iid Gaussian weights W ∈ Rn1×n0 ( Wak ∼ N ( 0 , σ2W /n0 ) ) , activation function f , and biases B = b1mT ∈ Rn1×m ( for b ∈ Rn1 ) . We regard the second argument of f as parametrizing ( continuously or discretely ) an ensemble of activation functions . We refer to B ( or b ) as the bias , in reference to the important special case f ( WX + B ) ( additive bias ) . In general , however , we only assume the parameters ba are such that the measure 1n1 ∑ a δba converges in distribution to some limiting distribution µB , effecting an arbitrary distribution over activation functions . We assume that E |f ( N ; b ) |k for N ∼ N ( 0 , σ ) is finite for all 1 ≤ k ≤ 3 , σ > 0 , and b ∈ support ( µB ) . When µB is a single Dirac mass at location b0 , the activation function can be written as f ( WX ; B ) = f ( WX ; b0 ) = g ( WX ) for some single-argument function g ( for an additive bias , g ( WX ) = f ( WX + b0 ) ) . When this is the case , we say the model has a single activation function , as opposed to a mixture or distribution of activation functions . The quantity of interest for the investigations below is the kernel matrix 1n1F tF , and in particular its resolvent , G ( z ) = ( 1 n1 F tF − zI ) −1 . ( 1 ) As we review in Sec . 4 , the optimal regression coefficients of a linear model on the random features F is a simple function of this resolvent . The high-dimensional regime that we study is the one in which the dataset size m , feature dimensionality n0 , and hidden layer width n1 all go to infinity at the same rate . In particular , as is standard in the random matrix literature , we assume that we can parameterize the limit in terms of the dataset size m in such a way that there exist two positive constants , φ : = lim m→∞ n0 ( m ) m and ψ : = lim m→∞ n0 ( m ) n1 ( m ) . ( 2 ) Note that the resolvent is a random matrix , but as m grows large , its normalized trace becomes a deterministic quantity . In the limit that m→∞ , this quantity is known as the Stieltjes transform1 , m ( z ) : = lim m→∞ 1 m trG ( z ) . ( 3 ) Together with an auxiliary transform m̃ ( z ) , defined below , these deterministic quantities completely characterize the asymptotic training error of kernel ridge regression on a noisy autoencoder task in this high-dimensional regime . The Stieltjes transform frequently arises in random matrix methods as a way to encode the spectra of matrices . In particular , if λi are the eigenvalues of 1n1F tF and the empirical distribution of eigenvalues converges in distribution to some deterministic limiting density as m→∞ , 1 m m∑ i=1 δλi → µ ( λ ) dλ , ( 4 ) then ( with appropriate technical assumptions ) , the limiting spectral density itself can be recovered from the Stieltjes transform m ( z ) via the inversion formula , µ ( λ ) = lim →0+ m ( λ− i ) −m ( λ+ i ) 2πi . ( 5 ) The Stieltjes transform then substitutes convergence in distribution for pointwise convergence for all z such that =z > 0 . 2.1 METHODS FOR COMPUTING THE STIELTJES TRANSFORM . We briefly review two standard methods for computing the Stieltjes transform m ( z ) , the resolvent method and the moments method . The resolvent method is an approach for computing the Stieltes transform based on the application of the Schur complement formula to the resolvent itself ( or to a closely-related block matrix ) . Intuitively , as the matrix size becomes large , the minors of the matrix are similar in distribution to the larger matrix , and , moreover , the Cauchy interlacing theorem guarantees that their Stieltjes transforms are 1The distinction between the Stieltjes transform m ( z ) and the dataset size m should be clear from context . close as well . This allows for the derivation of a self-consistent equation ( SCE ) in which the Stieltjes transform appears on the left-hand side as the trace of the resolvent , and on the right-hand side as the trace of one of its minors . The moments method is more combinatorial in nature and involves expanding the resolvent for large z and computing the traces of each term , m ( z ) = lim m→∞ 1 m trG ( z ) = − limm→∞ ∞∑ k=0 1 nk1 tr ( F tF ) k zk+1 . ( 6 ) The traces themselves are expended out as tr ( F tF ) k = ∑ Fa1α1Fa1α2 · · ·Fakα1 , ( 7 ) where the sum runs over matrix indices a1 , . . . , ak , α1 , . . . αk . The essence of the moment method involves analyzing the asymptotic contribution of each term in the sum based on its combinatorial type and the details of F , and resumming the results to obtain m ( z ) . We refer the reader to ( Erdos & Yau , 2017 ; Tao , 2012 ) for more details about these methods and additional background on random matrix theory .

Summary: This paper investigates the asymptotic spectral density of a random feature model F(Wx + B).  This is an extension of existing result that analyzed a model without the bias term, i.e., F(WX). This extension requires a modification of the proof technique. In addition to that, it analyzed a mixture of linear and non-linear activation functions, and show that mixture is better than single nonlinearity in terms of expected training error for ridge regression estimators.