Analysis of one-hidden-layer neural networks via the resolvent method
Authors: Vanessa Piccolo
Dominik Schröder
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural network
Resolvent method
Cumulant expansion

Abstract: In this work, we investigate the asymptotic spectral density of the random feature matrix $M = Y Y^*$ with $Y = f(WX)$ generated by a single-hidden-layer neural network, where $W$ and $X$ are random rectangular matrices with i.i.d. centred entries and $f$ is a non-linear smooth function which is applied entry-wise. We prove that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by [Pennington, Worah 2017] and [Benigni, Péché 2019] with the moment method. We extend the previous results to the case of additive bias $Y=f(WX+B)$ with $B$ being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. Our key finding is that in the case of additive bias it is impossible to choose an activation function preserving the layer-to-layer singular value distribution, in sharp contrast to the bias-free case where a simple integral constraint is sufficient to achieve isospectrality. To obtain the asymptotics for the empirical spectral density we follow the resolvent method from random matrix theory via the cumulant expansion. We find that this approach is more robust and less combinatorial than the moment method and expect that it will apply also for models where the combinatorics of the former become intractable. The resolvent method has been widely employed, but compared to previous works, it is applied here to non-linear random matrices.

Analysis of one-hidden-layer Neural Networks via the Resolvent Method
Vanessa Piccolo
Dominik Schröder Institute for Theoretical Studies ETH Zurich dschroeder@ethhzcco ch
ETH Zurich (current affiliation: ENS Lyon) vanessa, peceesseeeccoolees1y1ooooo
Abstract
In this work, we investigate the asymptotic spectral density of the random feature matrix M YY* with Y f[WX) generated by higge-hidden-layeer neural network, where W and X are random rectangular matrices with i.i.d. centred entries and
non-linear smooth function which applied entry-wiss -iiee We prove that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by Pennington and Worah (22) and Benigni and Péché (6) with the moment method. We extend the previous results to the case of additive bias = f(W X B) with B being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. Our key finding is that the case of additive bias it is impossible to choose activation function preserving the layer-to-layer singular value distribution, in sharp contrast to the bias-free case where simple integral constraint is sufficient achieve isospectrality. To obtain the asymptotics for the empirical spectral density we follow the resolvent method from random matrix theory via the cumulant expansion. We find that this approach is more robust and less combinatorial than the moment method and expect that it will apply also for models where the combinatorics of the former become intractable. The resolvent method has been widely employed, but compared to previous works, it is applied here non-linear random matrices.
Introduction
Machine learning has seen many successful achievements in recent years. Applications in face identification, object and speech recognition, translation, email spam filtering, navigation, medical diagnosis, etc. have proved the enormous potential of machine learning for day-to-day live [T6] [T]. Deep neural networks have turned out to be a particularly powerful machine learning method, and understanding the theoretical underpinning of their success has received tremendous attention in mathematics, physics and computer science.
fully-connected. feed-forward neural network with L hidden layers of dimensions n1. nı...., ny can be modelled follows:
[a( 3 s(w()) J(W(L-)) ((W(())))))) 1)) TR.d
where æ E mh denotes the input data vector and f: IR - R is non-linear activation function which is applied entry-wise. We denote the parameters of the network by 0 = (W(1).... ...., W BB) where W E 1 for < < I. and 3 E Rezxd are the matrices of the weights. In the classical setting of supervised learning, input feature vectors æ; E in with associated target vectors z; IRd. For example, æ; may encode the pixels of a photograph of animal and the target z; may label the species of the
given training set of (say) samples of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

animal in the image. Roughly speaking, the goal of supervised learning is to learn the mapping between the feature and the target vectors based on a given training set in order to predict the output of new unlabelled data. Let X = (a) Fm) E R be the matrix of the data and let Z = (z1... -m) E Rdxm be the target matrix. Then, the aim of the network is to find optimal parameters 8 such that (x) =
fo())) E Rdxm approximates the target Z optimally. During the training phase, weights are adjusted in order to minimize the empirical risk R(0)= = E Lifa( (X), 2), where ((:.) is a given loss function, usually involving some penalty for large weights 0 in order avoid over-fitting. Stochastic gradient descent (SGD) and its variants with propappropgation are the most commonly used algorithms for training multilayer networks by iteratively updating the parameters into the direction of the negative of the gradient of the empirical risk. For a much more complete survey, we refer the reader to
In the present paper, we will focus single-hidden-layer neural network of the form fx() B with Y = f(WX) ). This model was first studied by Louart, Liao, and Couillet (19) for the case where the data matrix X is deterministic and W is a matrix of random weights (in particular, the weights are given by functions of standard Gaussian random variables), and by Pennington and Worah (22) for the case where X and W are independent random matrices with both centred Gaussian entries. In both papers, the matrix / E Ik=xxxd xd is the only parameter to be learned and is chosen as the unique minimizer of the ridge-regularized least squares loss function
1 2dm
[(Jo(X),
11 - 3-1 + 1
where 7 > 0 is the learning rate. YG((?) where
The unique minimizing weight matrix 3 is then equal to B =
G(-~) = (-
is the resolvent of YY-Y. As proved in [19,2] 22 the expected training loss Euain is related to -7 G(-7), and thus also the Stieltjes transform of the limiting spectral measure of LY-Y. Here the Stieltjes transform my of a probability measure μ R is defined as mz,(z) = So (x ()==))= du(x) for E C such that 32 2 and for Mn, being the empirical probability measure of the n1 eigenvalues of m YYY Y is related to the resolvent via mpn, (z) = 1 Tr G(z). The performance of one-hidden-
m
layer neural networks depends on the asymptotic spectral properties of the matrix 1 LY-Y. Pennington and Worah [22] investigated the limiting spectral measure of the random matrix M = m derived the quartic self-consistent equation
m
LYY and
1 + zg (f())x (1 (1-
2 (1 + =2000)) (1+22ax)
Oz(l)
(1
16 Φ (1 ++2g00)
11 + 2020)
2b
(2, O2ll))
2%
1 11-2
(1 +1+ ++ )
2
where (=) = limmy 000 g(2) and g(zd((z)))))))) = 1 Tr G(2) is the Stieltjes transform, which is approxi-
ni
mately satisfied, (2) = g(2). by g(2) in case of Gaussian W, X. It is notable that the asymptotic
spectrum of y for large dimensions such that no/ m - Φ E (0, oo) and no/ni + 1b E (0. 20) depends on the non-linear function only through two integral parameters 0, (f) and 02(/), where (J) is the Gaussian mean of f and (z( is the square of the Gaussian mean of I' (c.f. (5). Benigni and Péché (6) then extended this model random matrices W and X with general i.i.d. centred entries, and obtained the same self-consistent equation 1]. We mention that ① may be reduced, for some special cases, to the quadratic equation that is satisfied by the Stieltjes trans- form Muze of the Marrhhnno-Passtiee distribution μMP (20). This means that for some activation functions, the non-linear random matrix model has the same limiting spectral distribution as that of sample covariance matrices XX* (in other cases, the equation can simplify to the cubic equation approximately satisfied by product Wishart matrices 10). This can be generalised to multilayer networks: [22] found experimentally that the singular value distribution is preserved through multiple prrchenko-Pasttur distribution in each layer. This conjecture was proved in 66] for the general case of bounded activation functions. Moreover, (19) performed a spectral analysis on the Gram matrix model with general training data and proved that, in the large dimensional regime, the resolvent of Y*Y has a similar behaviour that
layers by activation functions with 02( = and is given by the

---

observed in sample covariance matrix models. This was extended in [17] by considering Gaussian mixture of data. We also refer to the recent paper [18]. In the context of multilayer feedforward neural networks, Fan and Wang (9) analysed the eigenvalue distribution of the Gram matrix model, where the weights are random and the input vectors are assumed to be approximately pairwise orthogonal. In particular, they showed that the limiting spectral distribution converges to deterministic limit and, at each intermediate layer, this limit corresponds to the Marchenko-Pastee map of a linear transformation of that of the previous layer.
In recent years, there has been some progress in the asymptotic analysis of the eigenvalue distribution of another Gram matrix, the so-called Neural Tangent Kernel (NTK). Consider multilayer neural network and denote by J = (e) the Jacobian matrix of the network outputs with respect to the weights 9. Then, the NTK is the Gram matrix of J, defined by KNTK J. It was shown in (14) that the NTK at random initialization converges, in the infinite-widtl limit, to a deterministic kernel and it remains constant during the whole training time of the network. Subsequently, (23) analysed the spectrum of the sample covariance matrix JJ in a single-hidden hiddhh--ddddeerrrrr neural network, and provided an exact asymptotic characterization of the spectral distribution of JJ* with random Gaussian weights and data. Recently, (9) proved that the limiting spectral measure of the NTK converges to deterministic measure, which may be described by recursive fixed-point equations that extend the Marchenko-Pastie distribution.
The present paper is structured as follows. In the first part we consider the non-linear random matrix model studied in [6] and we compute its asymptotic spectral density. We follow the resolvent method via the cumulant expansion which, together with the moment method, is a standard approach to obtain the asymptotics for the empirical spectral density. In particular, we compute the self-consistent
equation that is approximately satisfied by the Stieltjes transform of the limiting spectral distribution. This is a quartic equation and is the same as that found in (6). In 66 22] the authors relied on the method of moments: they approximated general non-linear functions by polynomial ones and then computed the asymptotics of high moments E Tr(Y*) with Y = f(W to obtain the limiting measure via its moments. Conversely, we approach matters in a more robust and less combinatorial fashion by applying the resolvent method: we consider Y as random matrix with correlated entries and then we directly derive a self-consistent equation for its resolvent. In particular, we prove that the random matrix Y has cycle correlations, in the sense that the joint cumulant does not vanish when the random variables Vij's 's are joined by a cycle graph. We find that the variance of Yij is given by the parameter 0, (f), whereas for k the cumulants <<(i, Ya Vigiar.... Y i are powers of (f).). We note that in the random matrix literature matrices with general decaying correlations have been studied previously, see e.g. [3] 8 [ However, the cycle correlations of Y considered in the present paper are much stronger compared to these previous results. The second part of this paper concerns the additive bias case which is a more realistic model for machine learning applications. More precisely, we consider the random feature matrix Y f X B). where B is rectangular rank-one Gaussian random matrix, and derive a characterization of the Stieltjes transform of the limiting spectral density. We chose B to be rank-one since for the most commonly used neural network architectures the added bias is chosen equal for each sample. (2) studied the bias case for deterministic data and i.i.d. Gaussian random weights, and computed the exact training error of a singe-reeglarizzdd noisy autoenconder in the nigh-dimensional regime. Interestingly we find that in the case of additive bias it is impossible to choose an activation function f such that the eigenvalue distribution is preserved throughout multiple layers, unlike in the bias-free case where (f( = 0 yields the Marchenko- Passtur distribution in each layer. Finally, we remark that in the bias-free case our proof via the resolvent method has no significant advantage compared to the moment method, beyond requiring less combinatorics. The main advantage of the resolvent approach is that allows to include additive bias without much additional effort.
2 Model and main results
We consider a random data matrix X E R with i...d. random variables X.y with distribution 2/ and random weight matrix W X with i.i.d. weights Way with distribution 12, We assume that both distributions are centred with variance EX? of and EW? o². Morcover, assume
3

---

that the distributions V1, have finite moments of all orders Since for 1 i S n1 and S S m we have
WXX Vmo
1g
W Xkj
70
we note that in light of the central limit theorem the entries of the matrix WX are approximately yno
NN o normally distributed random variables. Therefore, for any t > O, we have the large deviation estimate
max 6
(WX)i /
>t
t S no
12 2
where we use the notation A S as shorthand for the incquality A S cB for some constant c. Let : R + R be O function with zero mean with respect the Gaussian density of standard deviation a i.e.
J [(Tuagaa)
e - dr 0.
(2)
22
We consider the random feature model generated by a gegge-hidden-layeee neural network,
_ YY E R with = m
WX
12O
where the activation function f is applied eise Let x: R - IR be a smooth cut-off function that is equal to one for |x| I 1 and zero for |x| 2 2. We then replace f by f())x ((no))) +). In particular, now have that is smooth with compact support. Moreover, for any > 0 and ng large
enough, with probability 1 no' the singular values of Y remain the same.
We are interested the eigenvalue density of the random matrix M in the infinite size limit. So, we assume that the dimensions of both the columns and the rows of each matrix are large and grow at the same speed, i.e. we introduce some positive constants di and & such that
ng
noo
and
no, R1, 00.
m
71
We denote by (A1, ........ Xn,) the eigenvalues of M and define its empirical spectral distribution by (&n, 1 E 8x,- Then, as n grows large, the empirical distribution of eigenvalues converges in distribution to some deterministic limiting density.
Theorem 2.1. There exists a deterministic measure = Po.v (01, such that almost surely weakly
Pn1 p as 11 00.
We notice that if m < then rank(M) = min min(n1, m) = m and M has zero eigenvalues. In this case, since Φ/vb > 1, there exists an atom 0 with mass pny (0) 1// O, and we have
81
Pm
80
8xc-
n1
Conversely, if n m, the matrix M has full rank and it is invertible. Since the nonzero eigenvalues of YY and y*y are the same, the limiting measure js of Theorem 2.1 turns out to be
1b
1
8n + i
where (.)+ = max(0, :). and jı is the limiting spectral measure m
LY
We will prove that the deterministic measure of Theorem 2. is characterized through a quartic self-consistent equation for the Stieltjes transform g(2) Tr G(z) of the empirical spectral measure Mans where
G(z) = (M - 2)- E CR.x xn1
'Thi assumption be relaxed by customary cut-off' argument, but refrain from doing for simplicity.

---

is the resolvent of the random matrix M and the spectral parameter 2 lies in the upper half plane
H= f= 32 0}. We set
(i( = [ ((owart) 12
-- -
V2T dx and 02(l) = dwwox ( ((
dx
(5)
V2T
Then, the following theorem characterizes 9 the solution quartic equation which depends only on the two parameters 01 (f) and (2()).
+e the measure p. is characterized
Theorem 2.2. For some ò, E > O and any HI with S2 > n1 through the following self-consistent equation
1 zg (01
02 21 -0 =0)) 9 (1 -1
02 02(01-2)) g² (1-28 21 (1+ zg) (1 29)) (1
2
S 11
v
(6)
almost surely.
Remark 2.3.
(i) We obtain analogous result for complex feature and weight matrices W. X, c.f. Remark E.
(ii) Note that the quartic self-consistent equation 6 may not have a unique solution such that Sg(z) > O. However, it has a unique solution which is analytic in the upper half-plane and satisfies g(z) -1/2 for large [z].
(iii) Since the resolvent itself satisfies TI G(z)/n1 -122 for large [z| and analytic in the upper half-plane, by continuity Theorem 2.2 implies that g(2) is approximately given by properly chosen solution of (6). Then, the limiting spectral measure itself can be recovered via the Stieltjes inversion formula,
m(A)=
-im 3g(A + ic),
-+0+ π
and Theorem 2.1 follows from Theorem 2.2
(iv) It follows from the self-consistens equation (6) that the limiting spectral measure ju is absolutely continuous w.r.t. the Lebesgue measure, and therefore so is μ away from zero. Moreover, for large equation (6) has real solutions and thus via Stieltjes inversion the limiting measure is compactly supported.
should be noted that Theorem 2.1 and Theorem 2.2 were proven in [22 6 under different assumptions and with a different method. The result [22] was obtained for i.i.d. Gaussian features and weights, whereas [6] extends the result to the case where both the inputs and the random weights have sub-Gaussian tails but are not necessarily Gaussian.
Remark 2.4.
Observing equation (6, note that if 02(ʃ) 0, then the limiting measure μ exactly the Marchenko-Pastu μMP distribution with parameter d/vb. Indeed, in this case, g(²) approximately satisfies the quadratic equation
1 (2+001) (--)) g(2) + (f) 2g(2)² - ~ 0.
which corresponds to the self-consistent equation satisfied by the Stieltjes transform of μMP (2)). As discussed in the introduction, this consideration is relevant when studying multilayer networks. Pennington and Worah [22 conjectured that the asymptotic spectral distribution preserved through multiple layers only by activation functions with (f( O and is given by the Marchenko-Pase distribution in each layer. Benigni and Péché (6) then proved this conjecture for bounded activation functions satisfying f2( 0. Moreover, if 0 (f) = (2(.). then equation (6) becomes cubic. In particular, the equality 0, (f) = Oz (f) holds and only if ʃ a linear function (for more details, we refer to the supplementary material in (22). In this case, M = // LYY" with Y = WX and thus the limiting measure I corresponds to the limiting spectral distribution of product Wishart matrix. The
spectral density for matrices of this type has been computed in 7 10].
5

---

f(z) tanh(x)
f(x) x
-10
Without bias
2
6
4-
2
0.2
0.4
0.6
0.8
1
1.2
0
20
40
60
80
-10
6
With bias
2
4
2
0
0
0.2
0.4
0.6
0.8
1
1.2
0
20
40
60
80
Figure We present the eigenvalue histogram of the covariance matrix YY for a single random realisation together with the theoretical limit from Theorems 2.1 and 2.5 for the functions f(x) tanh(x) and f(z) = x² with and without additive bias. We note that the presence an additive bias can both increase or decrease the largest singular value. The numerical experiments were conducted for the parameters n1 = 3000, Φ = og = ow 1, 1 = (left) or & = (right), and og 0 (top) or oB 0.25 (bottom).
2.1 Additive bias case
The previous model can be generalised by adding random biases. In neural networks, the bias an additional parameter that allows the model to better fit the given data. In this case, for each input data x IRne, a bias vector E IR= is added to the vector War E " We then apply non-linear function f: RR + R in element-wise fashion to its vector arguments Wx + b in order obtain random features f(Wx+ b) E R
We consider a random bias matrix B E R"1 of i.i.d. Gaussian random variables B, = B, with zero mean and variance EB? = of Note that the random matrix B has rank Let X E R xm and W E R xno be random matrices with i.i.d. entries, defined as before. Moreover, let f: IR IR be C function satisfying
--0²/2
1 (Vo203 + oz
dx
(8)
2
Just before, without loss of generality, upon replacing by (()x(log-- that f is smooth function with compact support. We then define the random matrix M by
may
M
1 1 with = m
WX noo
+ B).
(9)
where f is applied entry-wise. We introduce the parameter
w0 20
02 + o
and we define the following integral parameters:

---

c-0212
01(1) | ( of.
dx,
2T
1
19 + x2
of 02 22 + o
01,(1)
(12))
exp
dæ,
(10)
2nd o0 of
R22
26
/ (12)))((22))) exp
2 +
0i 72 ( + to
(2( = 2nō 2 og R22
dæ.
27
We can now state the analogue of Theorem 2.2 in the additive bias case. In particular, the following theorem shows that the normalized trace of the resolvent of M approximately satisfies the self- consistent equation (6) with parameters given by (T0).
Theorem 2.5. The Stieltjes transform g satisfies 4 with parameters given by (T)) where 01( is replaced by 01 (f) - (1, Moreover, there exists a single outlier eigenvalue Amax = 0101,6
O(T of M that is separated from the support of the rest the spectrum.
We remark that the parameters (1)(f), (f) can be alternatively expressed as infinite series, directly demonstrating that for σı # 0 and non-trivie f both coefficients strictly positive, (1,0(f), O. For notational implicitly, we introduce the Hermite inner product
(f.ghee = 1 J f()) x²12 dx. 2
Remark 2.6. We have
01.(f)
fxt f(
Vo of 120
0 + o
(1I)
o oF V Vo +
(
ooo + o ) (
02(f)
k20
and therefore (1)(f) 0, σb # 0 implies that f(ō-) is orthogonal to Hermite polynomials of any order, and consequently f = O. Similarly, 02(f) = 0, σb + 0 implies that f = const.
Multiple layers
In [2 it was observed empirically that in the bias- free case activation functions with (2( = have the remarkable property that for multiple layers
(1+1) ((w
Y = X
(12)
the singular value distributions (1), (7())yyr (2) .... all asymptotically agree (up to scaling) with the
probability distribution pa01,, 02) = u((1, 0) from Theorem 2.1 This observation is very natural
from point of view since find that (1) is approximately i..d. random matrix if (2( 0.
c.f. Proposition 3.2 below.
An interesting corollary of our Theorem 2.5 is that a similar isospectral property cannot be ensured for the case additive bias
(1+1)) = f( + BB),, ( X.
(13)
Indeed, in light of Remark 2.6, for ob * O we have (1,(f), > 0 for all activation functions f. and therefore already the random matrix (1) necessarily has leading order correlations, c.f. Proposition 3.3 below. Hence, convergence of the spectral density to the solution of (6) is not expected beyond the first layer. In Fig. we test this result experimentally and choose the activation function f()) = a |-| 62 with c1, C2 such that (2) is satisfied and (f( = We find that in the bias-free case (left), irrespective of the network depth, the eigenvalues of the covariance matrix (1) (1)) converge to their theoretical limit from Theorem 2.1 exactly as in [22] Fig. 1 In the case of additive bias (right), such convergence is observed, and this provides empirical evidence of our result.
In the notation of 22 fi.

---

1-layer
10a
10a
2-layer
5-layer
10-layer
104
104
dWass
n1
10
10-1
1-layer
2 2-layer
5 5-layer
10-3
10-3
10-layer
10
10a
103
10
10a
10x
Figure For randomly generated neural networks of varying depth and width, we compute the Wasserstein distance dwass between the empirical eigenvalue density of the covariance matrix
( (+) 1 to the distribution from Theorem 2.1 for the activation function f(x) Cj |z| C2. In the bias-free case (left), the Wasserstein distance decays the inverse of the network width, while in
the case of an additive bias (right) no convergence can be observed. The numerical experiments were
conducted for the parameters ox ow 1. 3 and o (left) or o 0.5 (right).
The spectrum of the covariance matrix (1) (Y)))* reflects the distortion of input data through the network and highly skewed distributions indicate poor conditioning which may impede learning performance [22]. Batch normalization seeks to remedy the distortion by normalising by the trace of the covariance matrix (y in each layer. [22] it was suggested that choosing activation functions with (f) = 0, i.e. functions which naturally preserve the singular value distribution, may serve as an alternative method of tuning networks for fast optimisation. Our result indicates that in the case of additive bias this alternative is not present. However, batch normalization seems help stabilising the singular value distribution also in the additive bias case, c.f. Fig.
Outline of proof of Theorems 2.2 and 2.5
The proof of both Theorem 2.2 and 2.5 can be broken into distinct parts. The first step is to show that Y f v E
W
IR can be viewed as a correlated random matrix with cycle correlations,
c.f. Propositions 3.2 and 3.3 below. The second step to prove the global law for the random matrix M YYY with the cycle correlations. In the following, we will sketch the derivation of the self-consistent equation. A more detailed proof is provided the supplementary material.
The key idea is to use multivariate cumulant expansion formula. Cumulants of a random vector X (X1.... Xn) can be defined combinatorial way by
k(X
X,) = (-1)) (I 1)!
II BET
II X NEEB
(14)
where the sum runs over all partitions и of the set [n] (1...., n), the product runs over the blocks B of the partition æ, and []] the number of blocks in the partition. The following expansion is commonly referred to as cumulant expansion and generalises the Gaussian integration by parts. In
the context of random matrix theory, the usefulness of this expansion was first observed in [15] and later revived in 12 13 A proof of the following lemma is provided in Appendix [Cr completeness.
Lemma 3.1 (Cumulant expansion). If X (X1...., X,) is a random vector with finite moments of all orders, then
R(X1, X X Ed,, ... .. (x)),
EX,
Σ
Σ
!!
12 f1....
where f: 1 + R is smooth.

---

Without batch normalisation
With batch normalisation
1-layer
Without bias
0.8
0.8
2-layer
0.6
0.6
5-layer
0.4
0.4
0.2
0.2
0
2
3
5
3
4
0.8
0.8
With bias
0.6
0.6
0.4
0.4
0.2
0.2
0
0
2
4
6
8
10
0
1
2
3
Figure 3: We present the eigenvalue distribution of neural networks of varying depth and in the presence/absence of both bias and batch normalization for the activation function f(x) = ci |a| C2- In the bias-free case, batch normalisation has no effect on the spectral stability, and throughout the network the theoretical distribution from Theorem 2.1 matches the actual eigenvalue distribution of the covariance matrix y (Y well. In the case of additive bias, the single-layer spectral density matches the theoretical limit from Theorem 2.5 to high accuracy. However, for multiple layers the spectral density diverges without additional batch normalization. Batch normalization alleviates the divergence, however the actual eigenvalue distribution deviates from the theoretical limit from Theorem 2.5] The numerical experiments were conducted for the parameters nı 3000, 2 and のg = 0 (top) or o = 0.5 (bottom). Here we used batch normalisation of the form (1) + (1) after each layer, choosing c to ensure unit empirical variance.
é = og = Te = 1, 1p
We start with the defining identity of the resolvent, 1a, 2G MG, where I,, denotes the nı n2 identity matrix, and we compute its average trace:
1 YY-G
YOG
+ zg
- Tr
Yos
(15)
71
E1j=1
where g(z) 1. Tr(M - =1 = 1 is the normalized trace of the resolvent of M. Since the random
variable (Y*G), can be seen function of Yıj. we can take the expectation on both sides of [15 and apply Lemma 3.1
<(Yigia Yigie Yig. .ize) Edy ... -.
YOG
1+ Eg
Σ f1....2hh
(16)
n1
(k (k- 1)!
m
The main goal now is to show that Y can be viewed as a random matrix with cycle correlations given as in the Propositions 3.2 and 3.3 below: Prop. refers to the bias-free case and Prop. 3.3 to the
additive bias case. We postpone the proof of both propositions to Subsections A.2 and B resp.
Proposition 3.2 (Correlation structure without bias). The random matrix Y defined by has joint cumulants given by
<(Yiriz) = O(m
<(Yi Y R 0f())
(17)
RoY Gagg Yegh..... y Yi 2 02(f))m1 1-k k>1
where in ii.... dos are all distinct, and we write X Rs Y as shorthand notation for X = Y(1 + 0(mo1/2)

---

Proposition 3.3 (Correlation structure with bias). The random matrix Y defined by 9 has joint cumulants given by
x((i)))) - O(m ((1
<( Yigi = (1()):
(18)
<(Y Yigh = 01,6(f)
111g: Vighs Yigior ... (iur)) m2
- k>
where ;15...,ū22 12x are all distinct.
Applying Propositions 3.2 and 3.3 to 16,, computing the partial derivatives and doing some book- keeping, we get the desired equation as no: n1. m - co. To complete the proofs of Theorems 2.2
and 2.5 one has to show the concentration of around Eg, stated in the following lemma.
Lemma 3.4. For the random matrix M m LYY and a complex number z such that +c
3z > n for some e > 0, it holds that Ew (g)==(()))))))))) Ew Ewg(2)1*
(19)
n (f
with high probability X, and analogously
Exlo(=) Exg(=)I* n²(S2)²
(20)
with high probability in W, where Ex (resp. Ew) is the expectation in the X-space (resp. W -psacee..
The proof of this lemma relies on a standard argument (e.g. see the proof of the concentration inequality [5, Subsection 3.3.2I) and given Appendix D
Conclusion
In this paper, we analysed the singular value distribution of fully random neural networks and found that in the case of additive biases is impossible achieve isospectrality by tuning the activation function. In addition, we showed that the resolvent method from random matrix theory also applies to the neural network analysis, despite the non-linearities and we expect that this robust method will prove to be useful in contexts where the conventionally used moment method becomes intractable.
Broader impact
Our result is purely theoretical one fully random features, weights and biases. Therefore, we do not expect our contribution to have ethical concerns or adverse future societal consequences.
Acknowledgments and Disclosure of Funding
D. Schröder would like to thank L. Benigni for illuminating discussions on the subject and both authors would like to thank him for his helpful comments on an early version of this preprint. Both authors thank the referees for their careful reading our manuscript. This work was carried out when the first author was research assistant at ETH Zurich in the group of W. Werner. The second author supported by Dr. Max Rössler, the Walter Hacfner Foundation and the ETH Zürich Foundation.
References
[1] A. Adhikari and 7. Che, Edge universality of correlated Gaussians, Electron. J. Probab. 24 Paper No. 44, 25 (2019) MR3949269
[2] B. Adlam, J. Levinson, and J. Pennington, A random matrix perspective on mixtures of nonlinearities for deep learning, preprint (2019), arXiv: 1912. 00827
[3] H. Ajanki, I.. Erdös, and Krüger, Stability of the matrix Dyson equation and random
matrices with correlations, Probab. Theory Related Fields 173, 293-373 (2019). MR3916109
10

---

[4] G. W. Anderson, A. Guionnet, and O. Zeitoumi, An introduction to random matrices, Vol. 1 118, Cambridge Studies in Advanced Mathematics (Cambridge University Press, Cambridge, 2010),
pp. xiv+492, MR2760897
[5] Z. Bai and J. W. Silverstein, Spectral analysis of large dimensional random matrices, Second, Springer Series in Statistics (Springer, New York, 2010), pp. xvi+551, MR2257175
[6] L. Benigni and S. Péché, Eigenvalue distribution of nonlinear models of random matrices, preprint (2019), arXiv: 1904. 03090
[7] T. Dupic and I. P. Castillo, Spectral density of products of wishart dilute random matrices. Part I: the dense case, preprint (2014). arXiv: 1401 7802
[8] L. Erdós, T. Krüger, and D. Schröder, Random matrices with slow correlation decay, Forum Math. Sigma 7, Paper No. e8, 89 (2019) MR3941370
[9] Z. Fan and Z. Wang, "Spectra of the conjugate kernel and neural tangent kernel for linear- width neural networks", Advances in neural information processing systems, Vol. 33. edited by
H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (2020), pp. 7710-7721.
[10] P. J. Forrester, Eigenvalue statistics for product complex Wishart matrices, J. Phys. A 47. 345202, 22 (2014), MR3251989
[11] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning, Adaptive Computation and Machine Learning (MIT Press, Cambridge, MA, 2016), pp. xxii+775, MR3617773
[12] He and A. Knowles, Mexoscopic eigenvalue statistics of Wigner matrices, Ann. Appl Probab. 27, 1510-1550 (2017) MR3678478
[13] Y. He, A. Knowles, and R. Rosenthal, Isotropic self-consistent equations for mean-field random matrices, Probab.. Theory Related Fields 171, 203-249 (2018), MR3800833
[14] A. Jacot, F. Gabriel, and C. Hongler, "Neural tangent kernel: convergence and generalization in neural networks", Proceedings of the 32nd international conference on neural information processing systems, NIPS' 18 (2018), pp. 8580-8589 8589.
[15] A. M. Khorunzhy, B. A. Khoruzhenko, and L. A. Pastur, Asymptotic properties of large random matrices with independent entries, J. Math. Phys. 37, 5033 1063-5060 (1996). MR1411619
[16] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning,
Nature $21, 436-44
(2015)
PMI PMID26017442
[17] 7. Liao and R. Couillet, "On the spectrum of random features maps of high dimensional data", Proceedings of the 35th international conference machine learning, Vol. 80, edited by J. Dy and A. Krause, Proceedings of Machine Learning Research (2018), pp. 3063-3071.
[18] C. Louart and R. Couillet, Concentration of measure and large random matrices with an application sample covariance matrices, preprint (2018), arXiv: 1805. 08295
[19] Louart, Z. Liao, and R. Couillet, random matrix approach to neural networks, Ann. Appl Probab. 28, 1190-110101111 1248 (2018) MR3784498)
[20] V. A. Marčenko and L. A. Pastur, Distribution of eigenvalues in certain sets of random matrices, Mat. Sb. (N.S.) 72 (114), 507-536 (1967), MR0208649
[21] A. Nica and R. Speicher, Lectures on the combinatorics of free probability, London Mathe- matical Society Lecture Note Series (Cambridge University Press, 2006).
[22] J. Pennington and Worah, "Nonlinear random matrix theory for deep learning", Advances in neural information processing systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (2017).
[23] J. Pennington and P. Worah, "The spectrum of the fisher information matrix a single- layer neural network", Advances in neural information processing systems, Vol. 31, edited by
S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Bisachi, and R. Garnett (2018).
[24] T. P. Speed, Cumulants and partition lattices, Austral. J. Statist. 25, 378-388 (1983),
MR725217


Reviews:
[{"review_id": "bGkCU_1mkWH", "reviewer": "Reviewer_HGSJ", "summary": "This paper studies the limiting eigenvalue-distribution of the gram matrix $M$ associated with a random feature model, and extends current theory (Pennington and Worah 2017; Benigni and Péché 2019, etc.) to the case of multiple layers with additive layer-wise additive bias. Understanding the distribution of eigenvalues of the gram matrix useful for the understanding of the training dynamics and generalization in neural networks.\n\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n---\n\n- A stark contrast between the bias-free case and the case with additive bias is shown: in the former case, the eigenvalue-distribution of $M$ converges almost-surely to a compactly supported deterministic probability distribution, whose Stieltjes transform satisfies a certain fourth order equation. In the later case, there is no such convergence (if there is more than 1 hidden layer), irrespective the chose activation function.\n\n- The main technique used to prove the theorems is the so-called \"resolvent method\". This is in contrast with the very combinatorial and cumbersome \"moment method\" used in prior works. \n\n- The paper is well-written and easy to follow. Related works are discussed in great detail, and the authors' contributions are well-situated.\n\nQuestions\n----\nL220 \"However, in the case of an additive bias no such convergence can be observed.\"  Is this a conjecture, an empirical observation, or is this rigorously proved somewhere in the paper ?\n\nMinor Issues\n-----\n- The super-script numbering of the references is non-standard (page 10).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, this has been taken care of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "SXN4QkKpdF", "reviewer": "Reviewer_o1S8", "summary": "This paper studies the limiting singular value distribution of non-linear random matrix of the form Y=f(WX), where W and X both consist of iid entries and f is applied entrywise, in the proportional growth asymptotics. The limiting law is characterized in terms of its Stieltjes transform as the solution to a self-consistency equation. The same result was obtained before by Pennington and Worah [21] in the Gaussian case and later extended by Benigni and Péché [6] to the general case. The difference is that the method in the current paper is resolvent-based. However, it is unclear what the advantage of this approach is. For example, is the result stronger than that of [6] in the sense of the required assumptions? The authors should clarify the main contribution of this work and what is new.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in general well-written, but I have some difficulty understanding the technical details and I felt a few steps are either missing or skipped. \n\n- The proof starts with the cumulant expansion in (16) and then applies Prop 3.2. But Prop 3.2 only deals with three special cases of cumulants. Why the contribution of other configurations are negligible? It seems it is assumed the only non-negligible contributions are those from even cycles. For example, it is claimed on p. 15 of the supplement \"Finally, we compute the cumulants of the entries of the random matrix Y. Since the activation function f is applied component-wise, it follows from the previous results that the only contribution comes from κ(Yi1, Yi2i3^*, ..., Yi2ki1^*) for k ≥ 1 and i 1 ,...,i 2k distinct, thus proving that Y has cycle correlations. \" This is not clear to me why this is the case and what the *previous results* being referred to are. For example, it is clear that odd cycles can be ignored due to the zero-mean assumption. But how about four cycles with double edges, something like \\kappa(Y_12,Y_12,Y_23,Y_23,Y_34,Y_34,Y_41,Y_41)?\n\n- Along the same line, in (23), I cannot understand the second line. Note that here the kappa in the first line has k arguments and kappa in the second line has 2k arguments. So this is not a matter of counting. I presume at this point we are removing all other contributions except for even cycles. So this needs to formally justified.\n\n- Also, from the writing it seems in the second line of (23) the indices i_1,...,i_{2k} are required to be distinct. What about the first line and also those summations in Lemma 3.1 and in (16)? None of the these are specified.\n\n- Another crucial step that is not justified is the last step of the proof of Prop 3.2, p16, which claims that the cumulant of the four tuple \\approx expectation of their product. Is this true for all other cumulants? In fact, \\approx is used throughout the paper without definition.\n\nOther comments:\n- line 106: The author should clarify \\kappa(Y_{i_1,i_2},Y_{i_2,i_3}^*,...,Y_{i_2k,i_1}^*). Do you mean \n\\kappa(Y_{i_1,i_2},Y_{i_2,i_3}^*,Y_{i_3,i_4}^*,Y_{i_4,i_5}^*,...,Y_{i_2k,i_1}^*) or \\kappa(Y_{i_1,i_2},Y_{i_2,i_3}^*,Y_{i_3,i_4},Y_{i_4,i_5}^*,...,Y_{i_2k,i_1}^*)? I presume the latter.\n\n- Existence and uniqueness of the solution to the self-consistency equation (1) is not discussed. Perhaps this is in the previous work [21,6]?\n- In equation (1) the dummy variable m conflicts with the sample size m which is also used everywhere. One should consider changing the notations.\n- It would be good to comment early the meaning of the integral parameters. For example, theta_1(f) is the variance of the entry of Y.\n- It is not very clear what \"invariance\" means as mentioned in the abstract and also in the paper. Perhaps you meant the limiting spectral distributions of Y=f(WX) and X coincide?\n- Lemma A.1. What is \\mathbb{H}? Is it only used for complex valued W/X?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "PYuY9xhQ5W", "reviewer": "Reviewer_Xqoe", "summary": "The authors study the asymptotic eigenvalue distribution of the post-activation Gram matrix in a one-hidden-layer neural network, in a regime where all matrix dimensions are growing proportionally. For input data with i.i.d. centered entries and a weight matrix with i.i.d. centered weights, the authors provide a new proof (under weaker distributional assumptions) of a result first due to Pennington and Worah that the asymptotic eigenvalue distribution is characterized by a quartic equation in its Stieltjes transform. The authors then provide an extension of this result to a model with i.i.d. Gaussian biases in its hidden layer. A qualitative difference in the setting with biases is that---in contrast to the bias-free setting, where special choices of activation function can allow the eigenvalue distribution to remain the same as that of the input Gram matrix, and hence remain constant across multiple layers---the authors show that the eigenvalue distribution with biases must necessarily evolve over layers (cf. Remark 2.6 and the discussion on page 7).\n\nThe proof is based on deriving the fixed-point equation for the expected Stieltjes transform using the cumulant-expansion generalization of Gaussian integration-by-parts, and explicitly analyzing the asymptotic mixed cumulants of the entries of the post-activation matrix. A technical challenge is to translate the more easily-obtained understanding of mixed cumulants of the pre-activation entries to the post-activations, and the authors accomplish this using a Fourier-analytic technique.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a nice contribution to this body of literature: To my knowledge, it is the first work to use this method of proof to study this type of non-linear matrix model. The results of Theorems 2.1 and 2.2 are perhaps not so new in the context of existing work, but the authors illustrate how this method can be used to obtain a new result in the form of Theorem 2.5. There is a high-level component of the proof that I don't understand---see below---but if this is a misunderstanding on my part and can be easily clarified or addressed, my opinion is that this work is above the bar for NeurIPS.\n\n----------------------------------------\n\nMajor comment: In Propositions 3.2, 3.3, and Lemma A.3, what exactly is the claim that is being made on which mixed cumulants are 0 and which are non-zero, and how is this used on lines 371 and 456 in the proofs of the main results?\n\nMy confusion lies in e.g. the following example: Consider the 4th-order cumulant 1/n_0^2 kappa( (WX)_{1,2}, (WX)_{1,2}, (WX)_{1,2}, (WX)_{1,2} ) where all four entries of WX coincide. Is this case treated in the discussion preceding Lemma A.3, and is Lemma A.3 claiming that this cumulant is 0? My understanding is that it's not 0, but since (WX)_{1,2}/sqrt(n_0) converges to a Gaussian, this is asymptotically 0. However now consider kappa( Y_{1,2}, Y_{1,2}, Y_{1,2}, Y_{1,2} ) for the post-activations: Is Proposition 3.2 claiming that this is 0? Since Y_{1,2} = f((WX)_{1,2}/sqrt(n_0)) converges to a non-Gaussian limit, I guess this is non-zero even in the asymptotic limit. But then perhaps I don't understand how this mixed cumulant (and analogous terms for larger k) is treated in the second equality of line 371. My confusion is related to the sentence \"Since the activation function f is applied component-wise, it follows from the previous results... that Y has cycle correlations\" on lines 420-423; I'm not sure what is meant by this sentence.\n\nA higher-level comment is that I feel a bit uneasy about the extent of casework that has been done---not only for Lemma A.3 as discussed above, but also in the \"Proof of (20)\" starting on line 495, and the proofs of Lemmas A.4 and D.1 in appendix D.\n\n------------------------\n\nOther minor comments:\n\n(1) I don't know if it's fair to characterize the moment method as \"more commonly used\" (in the abstract and conclusion). For example Refs. [19] and [9] also used the resolvent method, and I think an appendix of Ref. [9] derived also the quartic equation of Theorem 2.2 using a resolvent approach (although using a different method to analyze the resolvent).\n(2) Line 51, is it common to call gamma here the learning rate?\n(3) Can the authors state more precisely the moment conditions needed for nu_1 and nu_2 at the bottom of page 3, for their result to hold.\n(4) In Eqs. (18) and (20) of Propositions 3.2 and 3.3 the pattern of indices and conjugations in \"...\" are not clear, and it would be helpful to write out a few more terms.\n(5) Should the approximations of Proposition 3.2 and 3.3 be interpreted for each fixed k as n -> infinity? In the analysis of Eq. (16) one needs to consider the infinite series in k for each fixed n---can the authors elaborate on how the error terms are treated in this series?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "3pL0u18EAi", "reviewer": "Reviewer_5v5G", "summary": "This paper studies the spectral distribution of random matrix inspired by a one-hidden layer neural networks. In the first model, the matrix is obtained by applying a nonlinear function entrywise to the product of large rectangular matrices with zero mean IID entries. In the second model, a rank-one bias term is added before applying the non-linearity.  The main results  establish convergence of the emprical spectral distribution to a non-random limiting measure in the regime where the dimensions of the matrices scale linearly. This limiting measure is characterized in terms of the the second moments of the distributions on the matrices, the relative scaling of the dimensions, and certain properties of the non-linearity (described by Gaussian integrals). \n\nWhile the result for the first model has been derived previously (using moment based methods), the result for the second model is novel. Furthermore, in comparison to the prior work, this paper uses the resolvent method, which requires an evaluation of the cumulants instead of the moments -- a potentially benefit of this approach is that is can be applied more broadly. There is some discussion about the implications of the bias term in the context of multilayer networks. Specially, a non-zero bias preclude the certain invariance properties seen in the setting without bias. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The high level ideas of this paper are clearly explained. Also, the contribution of the paper is clear in the context of previous work. While the basic idea of the resolvent method itself it not new, this paper shows how it can be used to simplify and generalize the analysis of a problem of current interest. Hence, I think this paper will be of interest to researchers working in this area.\n\nOn the technical side, I have some concerns about the lack of higher order moment assumptions on the entries of the matrices. Unless I missed something, the author(s) require only the exists of finite second moments. But clearly some more assumptions are needed justify the CLT statement in lines 125-127. The nonexistence of moment conditions is also potentially problematic later (e.g., in the proof of approximate equality in Proposition 3.2) where higher order terms are said to be negligible. Overall, I would not be surprised if the all of the analysis goes through under some tail conditions (such as the ones given in citation [6]).\n\nThere is also room to improve the clarity of the technical results. I list some points below\n\n\t-What does the notation less-than-approximately-equal-to mean?\n\n\t- What is the definition of 1_m? (I assume identity matrix)\n\t\n\t- In line 146 - 147  there are some further statements about the limiting spectral measure (e.g., the nonzero part is continuous). These statements do not seem to follow from Theorem 1 which simply asserts the existence of a limit so I am not sure where they come from.\n\t\n\t- In line 148 , the function g(z) is the Stieltjes transform of the empirical measure, not the limit mu, and so the wording is a bit confusing. \n\t\n\t- Remark 2.4 part (ii) says that Theorem 2.1 follows from Theorem 2.2. However, doesn't one also need to establish uniqueness of the solution (in the proper limiting sense)? \n\t\n\t- It would be useful to provide full details of the batch normalization used in Figure 3.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the discussion is appropriate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the limited singular value distribution of a random feature model, which consists of an entrywise nonlinearity and a product of two i.i.d. matrices. In particular, the generalization performance of a single hidden layer neural network where the input data and hidden layer weights are i.i.d. random can be determined through this analysis. The authors show that the resolvent method can be used to simplify and generalize the existing techniques for this problem. The reviews all agreed that the paper contains strong theoretical results, only expressed minor concerns and suggestions, and recommended acceptance. As Reviewer o1S8 noted, adding a discussion of the advantage of the resolvent approach and clarifying the novelty of the results w.r.t. existing work (e.g., dealing with additive bias) will improve the paper. Please take into account the updated reviews when preparing the final version to accommodate the requested changes. Thank you for your submission to NeurIPS.
Decision: Accept (Poster)