A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions
Authors: Ben Adlam
Jake Levinson
Jeffrey Pennington
Venue:  (2020)

Abstract: One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions. While this paradigm has inspired significant research on the properties of large networks, relatively little work has been devoted to the fact that these networks are often used to model large complex datasets, which may themselves contain millions or even billions of constraints. In this work, we focus on this high-dimensional regime in which both the dataset size and the number of features tend to infinity. We analyze the performance of a simple regression model trained on the random features $F=f(WX+B)$ for a random weight matrix $W$ and random bias vector $B$, obtaining an exact formula for the asymptotic training error on a noisy autoencoding task. The role of the bias can be understood as parameterizing a distribution over activation functions, and our analysis actually extends to general such distributions, even those not expressible with a traditional additive bias. Intruigingly, we find that a mixture of nonlinearities can outperform the best single nonlinearity on the noisy autoecndoing task, suggesting that mixtures of nonlinearities might be useful for approximate kernel methods or neural network architecture design.

Under review as a conference paper at ICLR 2020
A RANDOM MATRIX PERSPECTIVE ON MIXTURES OF NONLINEARITIES IN HIGH DIMENSIONS
Anonymous authors Paper under double-blind review
ABSTRACT
One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions. While this paradigm has inspired significant research on the properties of large networks, relatively little work has been devoted to the fact that these networks are often used to model large complex datasets, which may themselves contain millions or even billions of constraints. In this work, we focus on this high-dimensional regime which both the dataset size and the number of features tend to infinity. We analyze the performance of simple regression model trained on the random features F f(W X B) for random weight matrix W and random bias vector B, obtaining an exact formula for the asymptotic training error on a noisy autoencoding task. The role of the bias can be understood as parameterizing a distribution over activation functions, and our analysis directly generalizes to such distributions, even those not expressible with a traditional additive bias. Intriguingly, we find that a mixture of nonlinearities can outperform the best single nonlinearity on the noisy autoeendoing task, suggesting that mixtures of nonlinearities might be useful for approximate kernel methods or neural network architecture design.
INTRODUCTION
is undeniable that in recent years deep learning systems have found widespread success in their applications to diverse and ever-expanding set of domains. The foundational results on many tasks such as image recognition (Krizhessyy al., 2012), speech recognition (Hinton et al.) (012). and machine translation (Wu et al.) 2016). have begun to make their way into higher-level products that people interact with and rely upon in their daily lives. Whether these products generate a medical diagnosis, navigation decision, inner-workings of the deep learning algorithms that generate them.
other important output, it is crucial understand the
Unfortunately, theoretical understanding of these deep learning algorithms continues lag behind their impressive practical successes. One main challenge in building a fuller understanding stems from the fact that deep neural networks are complex nonlinear functions that employ millions or even billions (Shazeer et al., (017)) of parameters. Traditional wisdom would suggest that to this parameter complexity corresponds an optimization difficulty. Recent work, however, suggests that as the width of a network's 's hidden layers becomes large, the loss function simplifies and theoretical analysis becomes tractable (Jacot et al., 2018 Mei et al. 2018 Chizat & Bach, 2018a| Mei et al.. [018)). In some scenarios, the simplification
2019, Rotskoff et al.) 2019 Rotskoff & is such that throughout training the parameters of the model stay within an infinitesimal radius of their initial (random) values, implying that much about neural network training can be understood by
studying the random initialization (Jacot et al. 2018: Chizat & Bach 2018b Lee et al. 2019).
Another main challenge in building rich understanding of deep learning systems stems from the fact that they are often trained on very large, complex datasets: even if the models themselves are very large, they may not be large in comparison Indeed, many important phenomena may become apparent only by examining the high- nigh-dimensional regime in which the dataset size and width are both large and of the same order.
the number of constraints they designed to satisfy.
In this work, we focus on the high- nigh-dimensional regime and analyze the performance of a regression model trained on the random features F = -(WX X + B) for random weight matrix W and random

---

= J[WX: B), Under review as a conference paper at ICLR 2020
bias vector B. We obtain an exact formula for the training error noisy autoencoding task in the limit that the width and dataset size both go to infinity. The result is determined by the resolvent of the kernel matrix F² F, whose properties we analyze via the resolvent method from random matrix theory. Our analysis also provides an exact formula for the eigenvalue density of the kernel matrix, which may be of independent interest since it provides characterization for how spectral properties of the data covariance matrix propagate through neural network layers at initialization.
1.1 OUR CONTRIBUTIONS
The main contribution of our work is an exact characterization of the training error of a ridge- regularized random feature regression model on a noisy autoencoder task in the high-dimensional regime. This is one the first non-trivial models to be solved exactly in the joint limit of large data and large width and provides an interesting testing ground in which to analyze this regime. Some of our additional contributions include,
An exact characterization of the spectral density of the randm feature matrix F = f(W - B), extending prior results of Pennington & Worah 2017 to non-Gaussian data distributions and to non-zero bias distributions.
One interpretation of the random additive bias is that it induces a distribution of activation functions parameterized by B, i.e. fZZ B) := f(Z + B). Our analysis trivially extends to any distribution of activation functions f(- B) parameterized by B.
We show that there exists a non-trivial distribution over activation functions that outperforms the best possible single activation function on noisy autoencoding task.
Our method of proof introduces a surrogate "linearization" of F, 1rlin, that possesses the same spectral information as F. Flin and its properties are likely to be of further interest and utility in analyzing neural networks in high dimensions.
1.2 RELATED WORK
Neural networks have been studied from the perspective of high- high-dimensional statistics in a number of recent works. Most prior work has focused on the bias-free case. [Pennington & Worah| (2017) studied the spectrum the activation matrix f ((WX) X) for iid Gaussian data and derived an analytic expression for the training error of ridge-regularized random feature model trained on pure noise. It is natural to consider incorporating biases by appending a constant feature to X. Unfortunately. this leads to biases that the same order as the weights, and so the effect disappears in the large dataset limit. Moreover, this modification on the data violates the assumptions of |Pennington & Worah (017)) |Hastie al. (2019) study ridgeless interpolation in high-dimensional interpo ation for linear features as well as nonlinear random features of iid Gaussian data. Louart et al. (2018) derived a deterministic equivalent for the resolvent of the kernel matrix 1s F which allowed for characterization of the asymptotic training and test performance of linear ridge regression of random feature models.
Other work has investigated learning dynamics and generalization in the high- -ich-smiensional regime (Liao Couillet 2018a Lampinen & Ganguli 2018 Advani & Ganguli 2016 Advani & Saxe 2017) as well as the spectra of more complicated objects such as the Hessian (Pennington & Bahri 2017) and Fisher information matrix (Pennington & Worah 2018). From the mathematical perspective, random matrix theory provides natural tools (Silverstein & Bai 1995a) for analyzing the behavior neural networks in the high- nigh-dimensional regime. Liao & Couillet (2018b) examined
spectra for data drawn from Gaussian mixture models; see also EEl Karoui 2010) the spectra of random kernel matrices.
2 PRELIMINARIES
Consider dataset X E R and the random feature matrix,
generated by a single hidden-layer network with iid Gaussian weights W E Rai X (Wak ~ (O /na)), activation function and biases B bl,"" T Rrixm (for R*1). We regard
2

---

Under review as a conference paper at ICLR 2020
the second argument of f as parametrizing (continuously or discretely) ensemble of activation functions. We refer to B (or b) as the bias, in reference to the important special case f(WX + B) (additive bias). In general, however, we only assume the parameters b, such that the measure 1 En 5b,, converges in distribution to some limiting distribution (AB, effecting an arbitrary distribution over activation functions.
We assume that E [IN; b) for N (N0, o) is finite for all k I 3, o 0, and b (upport(pB). ). When HA is a single Dirac mass at location bp. the activation function can be written as f(W x; B) = J(W X; bo) g( X) for some single-argument function g (for additive bias, g( x) = J(W + bn)). When this is the case, we say the model has a single activation function, as opposed mixture or distribution of activation functions.
The quantity of interest for the investigations below is the kernel matrix FFF. and in particular its resolvent,
G(2) = (
As we review in Sec. 4 the optimal regression coefficients of linear model on the random features F is simple function of this resolvent.
The nigh-dimensional regime that we study is the one in which the dataset size m, feature dimension- ality no. and hidden layer width nı all go to infinity the same rate. In particular, as is standard in the random matrix literature, we assume that we can parameterize the limit in terms of the dataset size in such a way that there exist two positive constants,
(m(m) and s lim (o n2 (m)
lim
(2)
m
Note that the resolvent is a random matrix, but as m grows large, its normalized trace becomes a deterministic quantity. In the limit that m + oc, this quantity known as the Stieltjes transform!
m(z) = lim 1rr
Together with auxiliary transform m=)), defined below, these deterministic quantities completely characterize the asymptotic training error of kernel ridge regression on a noisy autoencoder task in this high-dimensional regime.
The Stieltjes transform frequently arises in random matrix methods as a way to encode the spectra of matrices. In particular, if X; are the eigenvalues of _ -F F and the empirical distribution of n1 eigenvalues converges in distribution to some deterministic limiting density -> 0o,
Ex. - ((A)dA
(4)
then (with appropriate technical assumptions), the limiting spectral density itself can be recovered from the Stieltjes transform m(2) via the inversion formula,
m(A - iE) 10mmm+++ ic)
μA lim c
(5)
2Tic
The Stieltjes transform then substitutes convergence in distribution for pointwise convergence for all z such that 3z > O.
2.1 METHODS FOR COMPUTING THE STIELTJES TRANSFORM
We briefly review two standard methods for computing the Stieltjes transform m(2), the resolvent method and the moments method.
The resolvent method is an approach for computing the Stieltes transform based on the application of the Schur complement formula to the resolvent itself (or to closely-related block matrix). Intuitively, as the matrix size becomes large, the minors of the matrix are similar in distribution to the larger matrix, and, moreover, the Cauchy interlacing theorem guarantees that their Stieltjes transforms are
Thee distinction between the Stieltjes transform m22) and the dataset size should be clear from context.
3

---

Under review as a conference paper at ICLR 2020
Spectrum Gaussian Niases
Spectrum for Bemoull bjases
emprica
2-
1-
1-
2.0 2.5
0.5
Figure 1: We get excellent agreement of theory and simulation for spectral densities for any bias distribution. We set ó = 1.5, v = 0.8, ox ow L, and f = ReLU, Simulations are performed on matrices of size 214 Left Gaussian distribution over the biases with distribution. (N 1). Right Bernoulli distribution over biases with distribution 0.5.
close as well. This allows for the derivation of self-consistent equation (SCE) in which the Stieltjes transform appears on the left-hand side as the trace of the resolvent, and on the right-hand side as the trace of one of its minors.
The moments method is more combinatorial in nature and involves expanding the resolvent for large and computing the traces of each term,
tF(F'F)*
m (2)
lim 1 tr G(z) =- m 700000 mn
lim
(6)
n2
zkt
The traces themselves are expended out as
tr(F'FF** =
Faror Faio Fao,,Faroo Faros
Fogar
(7)
where the sum runs over matrix indices aj..... ag ai,,.... ag. The essence of the moment method involves analyzing the asymptotic contribution of each term in the based its combinatorial type and the details F, and resumming the results to obtain m(=).
We refer the reader to (Erdos & Yau 2017 Tao 2012) for more details about these methods and additional background on random matrix theory.
3 RESULT FOR STIELTJES TRANSFORM
3.1 MAIN THEOREM
We make the following assumptions the data matrix X and bias vector b;
1. 1 X.x.XaXa = Jasoz + O (1/ nno) for all a and ,
no
2. the empirical eigenvalue distribution of 1. X converges in distribution measure yx no
3. 1 E 8b. PB in distribution.
11
Theorem 1. Define az dwax and resolvent G(z) = (00))((1,))) m1
(1))
Then under the above
assumptions and for all such that 3z > O, the transforms
tr G(z) and 1 1tr L x' Xtr Xtr
(8)
converge in probability to the unique solution, m(2) and m((z), ) of the Eqn. that map C+ to C+:
S
mf (2) = Esnux Co(2)) SC,(2)
and m()) Esauxx Coll) + SC
(9)
where,
(BB - ((B)
{(B)
Co(2) == - -
C1(2) = EB
(10)
D(B)
D(B)
4

---

Under review as a conference paper at ICLR 2020
FIAARL
10
presidion emppical
15
10
spectral denstty
207
201
sparid
207
10a
10a
10
eigenvatues
olgar values
Gaussian
[iroodal input spectrum
2.0
SCE
emptrcal
1.5
PNL 8.8.0.8... pcl.1.
1.L11.,ppm 0.8
PML2.5_p=L_0 L..
PM_2.5_pxL_15
sigma 0.5.
sigmal b 2.0
0.5
0.5
Figure 2: Empirical spectral densities agree with our predictions for varied data distributions and shape parameters. Top left: One class from CIFAR (airplane), mean subtracted. Top right: Classes {0, 8) from MNIST, mean subtracted. Bottom left: Gaussian input data, varying the NN parameter settings and activation function. Bottom right: Input data with a bimodal spectrum. All plots used = ReLU, o 1.5, + 0.8 and ow oB 1, except for the indicated modified parameter. Empirical densities were smoothed using a Gaussian KDE.
D(B) 1 (C(B)m())++ ( ((B))
(11)
and where TBBB and ((B) are the Gaussian expectations
n(B)= = EN-N((.OE)) [f(N; and <(B) := :====ENN.00) BN)
(12)
The proof is quite involved and is presented in the supplementary material. The basic idea derive a multivariate Gaussian random matrix model with the same correlation structure as F, then derive self-consistent equation (SCE) using the resolvent method for this linearized version of F. Flinn
Remark 1. The self-consistent equations consist of two coupled equations involving the Stieltjes transform mz()) and an auxiliary object i=)z), (cf. (Paul & Silverstein 2009 Eq. (2))), which we will see in Cor. essentially measures the autoencoding capacity of the network.
Remark 2. Note that the self-consistent equations contain an expectation over the limiting spectral density of the input data. While the assumptions the data matrix X in Thm. Mare quite general, they may not be optimal. See Sec. 3.3 where we show strong agreement with empirical data from MNIST and CIFAR-10 and for range of synthetic distributions. This suggests that the theorem may hold for even more general data distributions.
3.2 ALTERNATE REPRESENTATION AND LIMITING RESULTS
When the data distribution is iid Gaussian, the expectations in Eqn. (9) can be expressed in closed form, though one must be careful to choose the correct branch of the resulting function. For simplicity and future reference, we focus the setting where < & &b < 1, in which case have the
5

---

Under review as a conference paper at ICLR 2020
Sandom, gwwls tramin MOP.
MMPRRNNOO tragin 1007
Hpap Higgnogge: Jranog poneeee
10
(b)
0.
0.5
Lauguee emo
0.4
0.4
0.0
19
101
107
101
127
307
100
10
10"
300
107
30"
100
10.
30:1
10'
107
Regulartastio paramitior
Regulariation parammcter
Repulanzation parameter
Figure 3: Comparisons of simulated ridge regression error and our theoretical prediction. We use ReLU with ox ow = 1 for all plots and vary the shape parameters ái and db. For simulations we use m 213 throughout. Note we also normalize the activation function so that Eg[n(6) = (a): Our predictions for ridge regression with random labels are solid lines. Simulated losses are the red stars. (b) Autoencoder error. (c) Noisy autoencoder error with dg = 0.2.
coupled algebraic equations,
m(2) C1 -Co
-C 2C,Ci
(13)
m(z) = 1 1 Com(2)
(14)
C1
When (g in Eqn. (TO) is trivial, i.e. single Dirac mass, then the result should reduce to the single
activation function case with F = f(W which was studied in (Pennington & Worah (017).
Indeed, writing
En [B]] and ( [(()]] for such a distribution, and eliminating m[z) from Eqns. 13) and (14). we find that m()) satisfies the following quartic polynomial:
= (=" )m())* + (22
))m(2) + (5 - 6)² + 2çΦ2 + 2no² mm))²
(15)
+ (So(p - の) + o² (z + n(w m)) 0
which agrees with the result in (Pennington & Worah 2017 upon identifying m(=) = -(1 6/v)/2 - o/vG(2).
3.3 SPECTRAL DENSITY ESTIMATES
The self-consistent equations in Thm. can be solved numerically by iterating Eqn. ( until con- vergence, using numerical integration. By utilizing the Stieltjes inversion formula, Eqn. (5). we can extract predictions for the limiting eigenvalue density of 1 F'F. The results show close agree- ment with empirical spectral simulations from several interesting practical datasets and synthetic distributions, see Figs. and 2
RIDGE-REGULARIZED NOISY AUTOENCODER
We consider the problem of kernel ridge regression with random features given by F (WWX; B) and noisy regression targets given by Y = AX + for some A x and independent Gaussian noise E 1 such that Eij (0, o²). As is common in the literature of high-dimensional statistics, we assume an isotropic prior on A such that A'A + 0 as m. 712 + co. Note that when σ = O, we have the pure memorization setting studied by Pennington & Worah (2017).
Corollary 1. Let W be the minimizer for regularized training loss
L(W2) 1
nym ||Y
W2F ngm ni 211211
(16)
with random features F = S(W X; B). Then the asymptotic training error converges in probability
Emiin
(0am(->) +
(17)
ngm
d-
In particular, that the derivative of the Stieltjes transform m' (- measures the capacity
to learn noisy labels, whereas m m-(-)) measures pure autoencoding capacity. See Fig. for comparison between these theoretical predictions and simulation.

---

Under review as a conference paper at ICLR 2020
Proof.
The optimal weights for the regularized loss given by
W
1 F for G(7) = F'F
791
resulting in training error
Exnin
ngm
l| W = = 2 tr (Y'YG(7)))
nam
- nom Anmm (X'XG(n))) t m (G())²)
= =~²0 d (m((-)) + + (m(->))
= -- (o'(mmm(((-)) + +0, mmm)(-
Remark 3. As in (Pennington & Worah 2017). there is a scaling homogeneity in the Firin:: an increase in the regularization constant ? can be compensated by decrease in scale W2, which, in turn, can be compensated by increasing the scale of F, which is equivalent to increasing T7(6) and ç(b). Owing to this homogeneity, we are free to choose a normalization of the activation function for which Eg-pn In(b)] =1.
4.1 NONLINEAR MIXTURES CAN OUTPERFORM SINGLE NONLINEARITIES
The bias term in our random feature model can be viewed as one way of defining a distribution over activation functions. The choice of distribution, in general, affects the performance of the model on given task - as quantified by the expectations in Eqn. (TO). A key benefit of our model and analytical approach is that it permits nontrivial distributions of nonlinear activation functions.
In this section, we build on our results for the training loss on noisy autoencoder tasks to examine the benefits of utilizing nonlinear mixtures. The goal here is not to identify "good" mixtures, since this will clearly be a dataset- and architecture-depedden question, nor is it to identify a large performance gap. Instead, merely seek to demonstrate a proof-of-principe distributions over nonlinearities that can provably outperform the best possible single nonlinearity.
namely that there exist non-trivial
For this analysis, consider the simplest possible nontrivial distribution over activation functions: Bernoulli mixture of two different functions. To each of these functions we associate two constants, but have B-dependence since each function is single
n and S, which derive from Eqn. (2 nonlinearity. Concretely, let
1 = ==EnNA(0.c2 [S(N)] and <= (= (NN)
(18)
For the two functions themselves, we utilize (i) a "pure linear" activation function with 7) = 1, i.e. the identity function and (ii) "pure nonlinear" (Hastie et al.),201 (019) activation function with 17 1 and < = O. (The particular purely nonlinear function in (ii) is irrelevant, as our theory predicts and our experiments confirm; see Fig. a)). To be precise, we define for p Bernoullipp).
fa( = Jxr
if pf
(19)
(g<= where 9<=0 is any function with 17 and Ç 0 (see, e.g., the functions in Fig. (3) of (Pennington & Worah 2017)).
if pf
The task of computing Emain for fp is cumbersome but purely algebraic. To see how to proceed, notice that the expectations in Eqn. (10) are simple for fp:
1-p
Cu(z) =22+ 1 \/)m(2))
C1(2) 1+ -1(2)= b/0
(20)
Plugging these equations into Eqns. 13) and (4) collecting terms and simplifying yields set of coupled polynomial equations for mn(3 =) and m ()). Taking the total derivative of these equations with respect to z yields two additional equations which can be solved to express m m²(z) and vi' vi'(2) in terms of m((2 and (z). Combining these results produces polynomial system whose solutior encodes Emain through Eqn. [T]. Fig. 4aa) shows the result of this calculation in solid lines for various values of y. while the lo error bars show empirical simulations with finite networks. The red stars in the figure show that for many values of y, the optimal mixture percentage is intermediate, i.e. 0 p 1.
SSpecial care must be taken in selecting the correct root of this equation, in accordance with the condition that m(2) ~ -1/2 2/22ff for large| [z].
7

---

Under review as a conference paper at ICLR 2020
(a) 10
(b)
Opeine
16
Ecrofn
Cytama
04
-0008
AEsun
AEtrain
0.00
000
0.2
0.4
0.5 0.5
1.0 0.0 0.2 0.4 0.5 0.5 1.0
0.001 0000 0.100
10
100
1000
Figure 4: Performance on ridge-regularized noisy autoencoder with 02 =, 1/2, and up 1/2. (a) Theoretical predictions for training error (solid lines) and error bars for empirical simulations of finite networks (no = 192, n, 384, m 384) for various values of ridge regularization constant y as the activation function varies. In the left panel, a single activation function f is used. the right panel, the non-linearity is fp. Bernoulli()) -miiture of purely linear (ç 1) and purely non-linear (C = 0) function. Each simulation uses a mandomly-chosee non-linearity having the specified values of Ç. demonstrating that Eusin depends the non-linearity solely through this constant. Red and blue stars denote minima. (b) Training error as a function of y for the optimal f and fp. as determined in (a). The bottom panel shows the difference in training error, demonstrating that the optimal Bernoullip)) (miilli -mixtur of non-linearities has smaller training error than the best single non-linearity.
The question is, does a non-trivial mixture actually outperform single nonlinearity? First, we must understand the performance of the optimal single nonlincarity. We note that owing to the homogeneity of the the training loss in 1). C. and y, we can assume without loss of generality that n 1. Therefore the entire effect of the nonlinearity should be encoded in the single constant Ç. In Fig. Ia) we plot our theoretical prediction for Eusin in solid lines and empirical simulations for finite networks as lo error bars. The activation function used for each simulation is chosen randomly, conditional on the value of Ç. So the good agreement in the left panel of Fig. aa demonstrates not just the correctness of our theoretical result but also the fact that Eusin depends on the activation function solely through the constant Ç. The blue stars in this figure indicate that the optimal single nonlincarity is neither purely linear (S 1), nor purely nonlinear (C 0), but rather something in between.
For this particular problem setup, the performance of the optimal single nonlinearity and the optimal Bernoulli mixture are rather close, indicated by the top panel of Fig. 4 However, owing to our precise analytical formulation, we can evaluate the training loss to high precision and observe that there is indeed difference in performance between the two models, as shown in the bottom panel of Fig. 4b). This result establishes that there are some problems for which even the best single nonlinearity is outperformed by a mixture of nonlinearities.
CONCLUSIONS
In this work we studied the feature matrix F = f:WX; B) where W a random matrix with iid Gaussian entries. Under mild assumptions on X and B, we obtained an exact analytic formula, Eqn. (D), that characterizes the Stieltjes transform of the spectral density of F. The result allowed us to describe the exact training loss of ridge-regularized noisy autoencoder in the high-dimensional, large-dataset limit, providing one of the first closed-fome solutions to non-trivial model in this limit. We found excellent agreement between the asymptotic predictions of Eqn. and variety of finteeddimensional empirical simulations.
We also advanced the interpretation of the bias B as one particular way of parameterizing distri- bution of activation functions. Indeed, our derivations proceed completely unchanged whether this distribution is of the traditional additive form f( + B) or the more general f(; B). By examining the latter, we showed that there are configurations in which a non-trivial distribution over activation functions provably outperforms the best possible single activation function. This opens the door to future investigations regarding optimal methods for parameterizing distributions over activation functions for approximate kernel methods, and suggests the possibility that mixtures of nonlinearities could be useful design consideration when constructing neural network architectures.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Madhu Advani and Surya Ganguli. Statistical mechanics of optimal convex inference in high
dimensions. Phys. Rev. X, 6:031034, Aug 2016. doi: 10. N0.100//hysReevx...333333 R...5..........1.33.... URL
https://iime aps. org/ do1/
6. 031034
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint 1TX0:1771003667, 2017.
Zhidong Bai and Wang Zhou. Large sample covariance matrices without independence structures in columns. Statistica Sinica, pp. 425-442, 2008.
Marwa Banna, Florence Merlevède, and Magda Peligrad. On the limiting spectral distribution for large class of symmetric random matrices with correlated entries. Stochastic Processes and their Applications, 125(7))2700-273) 2015.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterived models using optimal transport. In Advances in neural information processing systems, pp. 3036- 3046, 2018a.
Lenaic Chizat and Francis Bach. A note on lazy training in supervised differentiable programming. arXiv preprint wfXvv:112..07966, 2018b.
Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38, 01 2010. doi: 10. 10.1214/0--AOS644
László Erdos and Horng-Tzer Yau. A dynamical approach to random matrix theory. Courant Lecture Notes in Mathematics, 28, 2017.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint
2019.
Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-97, 2012.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems. NIPS' 18. pp. 3880-8589, USA. 2018. Curran Associates Inc. URL http: acm. org/ citat ion, C fm? ic 3327777.33277777 33277443888
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. (097-1105, 2012.
Andrew Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv:1809.10377 2018.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. arXiv preprint 10Xiv:190..06720, 2019.
Zhenyu Liao and Romain Couillet. The dynamics of learning: A random matrix approach. In 35th International Conference on Machine Learning, PMLR, 2018a.
Zhenyu Liao and Romain Couillet. On the Spectrum of Random Features Maps of High Dimensional Data. In International Conference on Machine Learning (ICML 2018), Stockholm, Sweden, July 2018b. URL Supplementary Material), 10 figure, ICML 2018.
hall ar wttp::/eww...e.........aaaaaaaa.
13 pages (with
Cosme Louart, Zhenyu Liao, Romain Couillet, et al. A random matrix approach to neural networks. The Annaly of Applied Probability, 280)):119-12288 2018.

---

Under review as a conference paper at ICLR 2020
Song Mei, Andrea Montanari, and Phan-Mini Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33) 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, arXiv preprint arXiv:1902.0601 2019.
Debashis Paul and Jack W Silverstein. No eigenvalues outside the support of the limiting empirical spectral distribution of separable covariance matrix. Journal of Multivariate Analysis, 100(1): 37-57, 2009.
Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International Conference on Machine Learning, pp. 2798-99-2222 2798-2806, 2017.
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning, In Advances in Neural Information Processing Systems, pp. 2637-2646, 2017.
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of single-hidden- layer neural network. In Advances in Neural Information Processing Systems, pp. 5410-5419, 2018
Grant Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden- Eijnden. Global convergence of neuron birth-death dynamics. arXiv preprint 1wX.iv:902.018420 2019.
Grant M Rotskoff and Eric Vanden-Eijnden. Neural networks interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. arXiv preprint arXiv: 0005500 2018.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural language models using sparsely gated mixtures of experts. ICLR, 2017. URL http:://amaa org/ hbc://bs//// httpaaas//11/1//./ 06538
Jack W. Silverstein and Z. D. Bai. On the empirical distribution of eigenvalues class of large dimensional random matrices, 1995a.
Jack W Silverstein and ZD Bai. On the empirical distribution of eigenvalues of a class of large dimensional random matrices. Journal of Multivariate analysis, 54(2):175-192, 1995b.
Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Society Providence, RI, 2012.
Yonghui WU, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08144, 2016.
10

---

Under review as a conference paper at ICLR 2020
Supplemental Material:
A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions
The main goal of the supplementary material prove Thm. ! This is achieved in three steps. The first step is to note that the limiting Stieltjes for the kernel matrix F*F is equivalent to multivariate a multivariate Gaussian random matrix model with an identical correlation structure. This was proven in great generality in Banna et al. (2015) (see Theorem 5.). Similar results for covariance matrices have been previously studied
Zhou (2008), but since does not have identically distributed rows (or columns) they are inadequate for our case. Specifically, Theorem 5 from Banna et al. (2015), implies that the two Stieltjes transforms have the same limit for & such that 3z >> almost surely.
The second step is to derive the leading order correlation structure (or kernel) for F (Lem. D and specify a multivariate Gaussian model with the same structure (Subsec. [A.I). We refer to this Gaussian model our random matrix ensemble. Without loss of generality, we may assume that Fa = for all a and a. This is because E.a. only depends a rank-1 matrix-a a transform that has no effect on the limiting Stieltjes transform as it represents an (l/mm) change in the spectrum (see for more detail Bai & Zhou (2008)).
linearized model, since it removes the nonlincarity from the random part of
(as we demonstrate in Sec. we center F using
The third a final step of the proof is a derivation of a self-consistent equation specifying the limiting Stieltjes transform of the Gaussian model. We do this using the resolvent method (see Erdos & Yau (2017) for an introduction). While the application of this method to multivariate Gaussian covariance matrices is standard, we note the form of the SCE is new and that we include the derivation here for completeness.
At the end of the supplementary material, we include some additional figures.
A CORRELATION STRUCTURE OF F
To begin we derive an asymptotic form for correlation structure of F. For simplicity of presentation, we derive the results for f(, B) = f(- B), but the argument generalizes directly. We recall that W is ny X ng random matrix with mean 0 and variance o /./no iid Gaussian entries. Define Coi8 == Σx Xka X no- The data matrix X and bias vector B are deterministic, so all expectations over W (and can be thought of as conditional on X and B). Moreover, X and B satisfy the following assumptions:
Ca3 6, O (1/ for all and R;
The empirical spectral distribution of 1 xx'X converges in distribution to some measure mx: 1 E 6b, + иB in distribution, or equivalently 1 mi Ex= g(ba) + EB=Pg for bounded, continuous functions
Observe that
Zea = WakXep and Zoo := Σ Wot
(21)
km
are jointly Gaussian. Moreover.
EZon and EZa. Zo. of capl (a = b).
(22)
Note in particular, that Zoo and Zao are independent if + b. For convenience, we normalize Cap and define Eap := ca./ox / and E., Ean
In order to specify the correlations, we employ transforms of the activation function f. The transforms Gaussian integrals of f at different locations and scales:
Ex(x) = o (k) (axow + x),
(23)
Thi (z) = Eo ++) ow
p (oxow + x) x2) +x),
(24)
and
(x)(x) = E iti it (() (ox =(x))()))
Eo :
(25)
11

---

Under review as a conference paper at ICLR 2020
Remark 4. For simplicity, while writing f() assume differentiability of f up ith order. However, this assumption can be relaxed, as since the derivatives are only used to take a Gaussian expectation. they can be expressed using Stein's lemma. For example, E (z) EN x +x). Thus, similar results should hold for rectified linear units for example, regardless of their lack of differentiability at 0.
Lemma 1. To leading order the correlation structure of F is
if + b if a = 3 and a = b if a + 3 and a = b
EF Fbg
(oo(bba) mi Caa)
(26)
Proof of Lem. 7
Before we prove Lem.
we state a simple lemma that is easy to show using Taylor
expansion.
Lemma 2. Suppose e is small constant and that N ~ No, 1), then
(2(
ENf (axow b) [b(b) 2
+ O(
(27)
ENS (exowv1+eN b)² (oo + (b) To2 E O(E²)
(28)
and
EN (oxowVI++)) f (oxowVI+EN + b)= = (i) + (b)b) + Eb(b) + O
(29)
Recall (x(x) = 0 by assumption. Using the observation in Eqn. 21 we repeatedly rewrite the expectations in Ft F
reduce the expectation over W expectation most two correlated Gaussian random variables. Let and N' denote independent, standard Gaussian random
variables. Using Lem. we see
(Z(o)))) = BB(ooVCo + ba) = (2(b,)
2 Eo + (EE2)
(30)
Again, using Lem.
2 we get
(f(zaa)² noo(ba) + (malba) + Muz (ba)) + 0(2)).
(31)
For the covariance calculation, define Pag cagl /Caa and recall that Pas is order 1/vm. Thus,
= + bu)s (awPasN' + ow v cBB Pag N + ba)
(32)
[Ex NCaN + ba)) [ENS (ow / Vag + +bb)))
(33)
+ PEa flow Rea ba)) [ENJ' (ow / c3g - PPap N + ba)
(34)
For the first term, we get
Ea t (b.).) (b(b))(2(bo)
+ 0(m 1) 0(m
(35)
For the second term, we have
(1 (1- En +
Eg
+ 0(m-1)
(<n(bn)) + E1 (bbb)) Ea(ba)
Ea + 0m--1)
Ga
(36)
2
2
x (ba) + (bba)) Es(ba)
Eg + 0mmml)
(37)
2
= (b,,) + 0(m ())
(38)
Thus, to second order we have
Ef(Zaa)f(Zen = (b. + O(n5')
(39)
n
12

---

Under review as a conference paper at ICLR 2020
A.1
LINEARIZED MODEL
Define
plin := + - -33-
(40)
where: O² and O are #1 x m matrices that have iid Gaussian entries with mean 0 and variance 1/n1: 2) V and C are n1 nı diagonal matrices with entries
v Too(b.) n((,)) and C² 7u (b.) C(ba),
(41)
for bias vector b; 3) Σ is matrix square root of 1
1gg
simple calculation shows that the entries of the matrix Flin match the first and second mixed
moments of F given in Lem. [ However, unlike F, Frln is linear in the random matrices O and O? in this sense, it is a lincarization of F.
B DERIVATION OF SELF-CONSISTENT EQUATION FOR FLIN
Theorem 2. Let (2) be the Stieltjes transform of 1 Flin, i.e. trG(22) Then with probabil-
ity I. nı sa. for all z such that 32 O, mmn (2) m(2), where m(z) is the solution of the coupled equations
S
m(z) = (B
and m((2) = Es-MP(e)
(42)
[oll)) + (C)
Co(z) + SC)
with
(n(B ((B)
{(B)
Co(2) == -==22BBBppp
C1(2) EBaEEn
(43)
D(B)
D(B)
D(B) =1 (C(B)m(z) + (n( ((B)) m(2))
(44)
We want to study the eigenvalues 1 prin piin Note without loss of generality it is sufficient to consdier diagonal Σ, since we can diagonalize Σ using some orthogonal matrices O and O' as the square root of positive definite matrix. Moreover, these orthogonal matrices, when applied either O 2 do change their distributions. Thus, diagonalizing E. so that Da / A where X are the eigenvalues of X X/no. results in an equivalent matrix ensemble in distribution (see Silverstein & Bai 1995b) for more detail). With this simplification, flin has independent entries given by
Flin C. e
+
V2 - CRO22
(45)
ec
To make the derivation easier, we can partly linearize the problem by studying the matrix
=1 Flin
ptin
H
(46)
-I
By the Schur complement formula, one easily finds
(m) = 1 TT lin piin /n1/m1an
1 m
Σ
Goa(z)
(47)
m
17
and (m) =
1
Tr Frlli plin't /nn
21)
1
Ti +m
Gaa(z),
(48)
n1
n1
where G the inverse H. Again by the Schur complement formula
1
ai
plin pain (la) m-tambb
(49)
Gaa
a,b=l
and
1
m
plin plnn (a) ac aB aB
(50)
Gao
cr,9=1
13

---

Under review as a conference paper at ICLR 2020
for E {1..... m and {m + 1...., m n1 and ((a) is the inverse of the minor H(a), Since G(a) is independent ( iar...., @ and iar..., mia we see by taking the expectation over these variables that
E
Flin Filin G(o)) ac m+t.m++
Σ ((b.)) + m(bb) ((ba))
G(a)
(51)
maaammaa"
0,b=1
Moreover, standard concentration inequalities and the Ward identity (see Erdos Yau 2017 show
R2
Flim Flin
G(a)
mm+
E
Flin Flinglo) G(o)
has m+a.m++
(52)
a.b=l
0,b=1
sC
1
Σ
(a) maamm+
1/2
SC
3Gla)
3Gm+a.m-t m-aama+a
1/2
50(/vm)
(53)
m
m
Oz
8.
with high probability. Similar bounds are easily obtained for Ea Flin plin G(o) an a9 ag
We may also replace ((a) with G at the expense of another small error that can be bounded using the Cauchy interlacing theorem: m -mm+ammta I O11mm. Using this control over these sums, we see
(a))
Flin pilin G(a) an
-
(54)
Gaa
a,b=1
2
1x ((ba) + (bba)) - ((bb)) Gmtamta + +0(1//m)
(55)
11
a=1
and
-1
lin F lin (aa aa a. 0g
(56)
Gaa
a
(1 ((ba) + m(baa) ((ba)) Gaa + O(1/vm/)
(57)
71
Finally, we invert Eqn.
56
multiply by Xx ((ba) + ((baa) ((ba), and average over to find
n2
((b.) - (bb)) ((b.)) G/a)
Gnttam+a
(58)
N ((ba) + 1/bb)) - ((ba)
(59)
71
1
(((bb))(() - (n(ba) ((ba)) mim (2)) ((1vmm)
((ba) + m(b.)) - ((b.)
+0(1//m)
(60)
(C(b.)m(z) (v(bb.) <(b,) ((b,))m(==)))))
x² (BB) C(B)
=-BB-N(0.03)
+ o())
(61)
I (C(B)n()) + (vr(B) - m(
where mm (z) En X2t Goa. where Taylor expanded in the second step, and where we used our assumption on B.
We can now invert Eqn.
54 and average over to find
1
(2) > Es-ax
(62)
- 2 EB-N(000))
14

---

Under review as a conference paper at ICLR 2020
Similarly,
S
mmm (2) + Es
(63)
(B
Note that the integral over S here is an integral over the limiting distribution of the data μx, which we assume
1
Σ 5
7
px
(64)
in distribution. In the case of iid Gaussian data, this is exactly given by the Marchenko-Pasee distribution.
C ADDITIONAL FIGURES
20
40
60
80
0
20
40 60
80
40
40
60
80
Figure 5: Comparison of right singular vectors of X and F for MNIST (left) and CIFARIO 10 (right). Entry ij shows x; fy. Although our theoretical results do not give predictions for how the singular vectors change, we found interesting behavior, with very little change to the largest singular vectors (which are nearly isolated the spectrum), but more mixing singular vectors in the dense part of the distribution.
Nay MAA
dww.gggeerrregeeo. TA
65-
13 mer
02
167
337
332
31
127
30
107
10
121
231
101
107
107
Wogularaation p0r0000T
Regabbrbzten
Regulotration awarnote
Figure 6: Comparisons of simulated ridge regression enror and our thcoretical prediction. Here vary the activation function by changing the slope a in leaky ReLU. In particular a = -1 is a linear function, 0 is regular ReLU, and is the scaled absolute value function. We normalize all functions so that E, [nb)] = 1. We get excellent agreement with theory from only a single sample.
15


Reviews:
[{"review_id": "rklsQVQCtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the asymptotic spectral density of a random feature model F(Wx + B).  This is an extension of existing result that analyzed a model without the bias term, i.e., F(WX). This extension requires a modification of the proof technique. In addition to that, it analyzed a mixture of linear and non-linear activation functions, and show that mixture is better than single nonlinearity in terms of expected training error for ridge regression estimators.\n\nPros:\n- This paper investigates an interesting problem and it successfully extends the existing work. The theoretical curve well matches the simulated curve.\n- The finding that mixture of nonlinearities gives better expected training error is interesting.\n\nCons:\n- The extension to the model with bias seems a bit incremental. In practice, we may consider an input with additional constant feature, X <- [X,1], to deal with both models in a unified manner. There should be more discussion about why this kind of trivial argument cannot be applied in the analysis.\n- The effect of mixture of activation functions is investigated in the \"training error,\" but I don't see much significance on investigating the training error thoroughly. Instead, people are interested in the test error. I guess there does not appear such a trade-off for the test error and the linear activation function would be always better because the true function is the linear model. Hence, more expositions about why the training error is investigated should be provided.\n\nMore minor comment:\n- I guess the definition of Etrain  (Eq.(17)) requires an expectation with respect to the training data.\n- Assumptions of the activation function f should be provided; is it just assumed to be differentiable?, ReLU is included?\n- The definition of G(\\gamma) in page 6 had better to be consistent to that in previous pages.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJgdT5JY5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzed the asymptotic training error of a simple regression model trained on the random features for a noisy autoencoding task and proved that a mixture of nonlinearities can outperform the best single nonlinearity on such tasks.\n\nComments:\n1.The paper is well written and provides sound derivation for the theories.\n\n2. Since this area is out of my expertise, I’m not sure whether merely extending the work of Pennington & Worah (2017) to non-Gaussian data distributions is significant enough or not.\n\n3. Except for Fig 4, the other figures seem out of the context. There is no explanation for the purpose of those figures in the main contents. It is a bit hard for the audience to figure out what to look at in the figures or what the figures try to prove. \n\n4. In “..., and our analysis actually extends to general such distributions, ... ”, “general” should be “generalize”.\n\n5. In “And whether these products generate a medical diagnosis or a navigation decision or some other important output, ..”, “whether” should be “no matter”.\n\n6. “..., they may not be large in comparison to the number of constraints they are designed asked satisfy.” should be “...  they are designed to satisfy”.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Bkl2KwYTtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors focus on the high-dimensional regime in which both the dataset size and the number of features tend to infinity. They analyze the performance of a simple regression model trained on the random features and revealed several interesting and important observations. I think it is a solid work and vote for acceptance. \nPros: \n(1) This paper has a solid theoretical foundation. Although I have not checked in detail, I think the deduction is clear and the contribution is well-established.\n(2) It extends some traditional bounds to more general cases. I think it will provide useful guidance to real applications, such as the network design in deep learning.\n(3) The authors have explained the results in a clear way. Thus, it will benefit the following readers and give deep insights about the related research areas.\nMinor comments:\n(1) I think some assumptions should be explained. For example, why the authors focus only on linear model. Due to the simplicity or the requirement from real applications?\n(2) More experimental results on large data sets should be added to validate the effectiveness.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, the authors focus on the high-dimensional regime in which both the dataset size and the number of features tend to infinity. They analyze the performance of a simple regression model trained on the random features and revealed several interesting and important observations.

Unfortunately, the reviewers could not reach a consensus as to whether this paper had sufficient novelty to merit acceptance at this time. Incorporating their feedback would move the paper closer towards the acceptance threshold.
Decision: Reject