Non-Gaussian Tensor Programs
Authors: Eugene Golikov
Greg Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning theory
infinitely wide networks
tensor programs

Abstract: Does it matter whether one randomly initializes a neural network (NN) from Gaussian, uniform, or other distributions? We show the answer is ”yes” in some parameter tensors (the so-called matrix-like parameters) but ”no” in others when the NN is wide. This is a specific instance of a more general universality principle for Tensor Programs (TP) that informs precisely when the limit of a program depends on the distribution of its initial matrices and vectors. To obtain this principle, we develop the theory of non-Gaussian Tensor Programs. As corollaries, we obtain all previous consequences of the TP framework (such as NNGP/NTK correspondence, Free Independence Principle, Dynamical Dichotomy Theorem, and μ-parametrization) for NNs with non-Gaussian weights.

Non-Gaussian Tensor Programs
Eugene Golikov
Greg Yang
École Polytechnique Fédérale de Lausanne Lausanne, Switzerland
Microsoft Research Seattle, USA
evgenii.golikoveee. soaeiiiigolkkoo@ppl...l
gregyangdmicrosoft.com
Abstract
Does it matter whether one randomly initializes a neural network (NN) from Gaussian, uniform, or other distributions? We show the answer "yes" in some parameter tensors (the so-called matrix-like parameters) but "no" in others when the NN is wide. This specific instance of for Tensor Programs (TP) that informs precisely when the limit of a program depends on the distribution of its initial matrices and vectors. To obtain this principle, we develop the theory of non- Gaussian Tensor Programs. As corollaries, we obtain all previous consequences of the TP framework (such as NNGP/NTK correspondence, Free Indepedence Principle, Dynamical Dichotomy Theorem, and u-parametrizzaionn for NNs with non-Gaussian weights.
general universality principle
Introduction
Universality in Neural Network Initialization Common initialization schemes of neural networks (e.g., 16, [ define specific ways of scaling initial layer weights with layer sizes, However, in general, they do express any preference on the initial weight distribution beyond iid sampling. use Gaussian distribution, other practitioners advocate for uniform one since it is possible to sample very large weights from a Gaussian, causing numerical instabilities.
While some works 13 12]
However, theorist, especially with a background on physics or probability theory, would suspect that the precise choice of distribution does matter the neural network's width tends to infinity. This is belief the general concept of universality large systems display a consistent behavior macroscopic scale regardless of the microscopic details
This consideration has led deep learning practitioners to treat the distribution of random initialization matter of personal choice. However, we shall see, the reality appears be more subtle: roughly speaking, distribution universality holds for hidden weights but does not hold for other parameters like input and output weights. We formulate the precise universality principle (Principle 2) for neural network random intialization in /Section 2] We do so in plain English, but deducing this principle requires some fundamental advancements in the theory of Tensor Programs. Below, briefly review this theory before describing our contributions to it.
Tensor Programs Just like autograd [25 empirically automates the calculation of chain rule of arbitrary computation graphs, Tensor Programs (TP) (2 has automated the theoretical calculation of aifinite-witth limits of the same (where width of computation graph corresponds to the size of matrices). What previously were difficult limits to calculate, now becomes routine via TP. For example, [77] in 1994 showed that randomly initialized wide shallow neural networks Gaussian Processes (which is called the Neural Network-Gaussian Process Correspondence, or
We refer the reader to the arXiv version for the latest version of the manuscript. "The simplest example of universality: taking the average of many iid copies of random variable always yield its mean, regardless of whether variable is Gaussian, uniform, Laplace, etc.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

NNGP Correspondence), but only recently this has been extended to deep perceptrons [13 16 and more advanced architectures such as convolutional neural networks [3 [8] and each such extension requires painstaking calculations and careful applications of Law of Large Numbers and Central Limit Theorem. But with Tensor Programs, one can show that NNGP Correspondence holds for any architecture all at once [29]. Similarly, in a certain parametrization, wide multi-layer perceptron (MLP) evolves like a linear model during training [14], but showing this for advanced architectures was very difficult. TP (30 34 again was able to prove this behavior for any architecture. Finally, TP gave rise to the Dynamical Dichotomy Theorem 33], a classification of all natural infinite-width limits of neural networks, and led to the discovery of Maximal Update Parametrization, These results underlie the hyperparameter transfer technology that for the first time enabled the hyperparameter tuning of enormous neural networks too expensive train more than once [33].
j.P.
Distributional Universality However, these results were only proven for computational graphs whose matrices and vectors are random Gaussians (which, in the special case of graphs pertaining to neural network training, means random Gaussian initialization of NN). What happens when they take more general distributions?
In this work, we show that any program with non-Gaussian matrices and vectors also have infinite- width limits under mild conditions, and thus recovering all of the aforementioned results for non- Gaussian objects automatically (see [Section 4). In fact, they often coincide with the limits of Gaussian samplings: we shall formulate a general distributional universality principle for Tensor Programs (Principle 3), from which the universality principle (Principle 2) for NN random intialization (mentioned in the beginning of this section) follows as special case.
For those familiar with the language of Tensor Programs, the principle can be summarized simply; all programs have the same limit if their Gaussian matrices are swapped out for non Non-Gaussial matrices with the same variances; however, this not true general for their initial vectors.
Applications to Random Matrix Theory The universality principle also makes the proof of the Semicircle and Marchenko- Laws in [T automatically valid for non-Gaussian random matrix ensembles. Likewise, it shows that the asymptotic singular value distribution of the input-output Jacobian of a random neural network does not depend on the distribution of random initialization beyond its variance.
Contributions In summary:
We clarify where initialization distribution matters and where it does not in wide neural networks by formulating precise universality principle (Principle 2).
More generally, develop the theory of non-Gaussiaa Tensor Programs, as well as stating the corresponding universality principle for Tensor Programs (Principle
We apply this general theory to obtain previous results of the Tensor Programs papers, such as NNGP and NTK correspondence, for non-Gaussian weight initializations (Section 4).
2 Distributional Universality in Words
In this section, we state, at level understandable to practitioners, where the sampling distribution in random initialization matters (beyond the first two moments). We begin with the simple example of MLPs
Motivating Example: MLP Let first consider the case of a simple biasless multilayer perceptron (MLP) ((E) with hidden layers and nonlinearity d:
(I) Here W WL E Raxn W1 E Roxd and WL+1 E Rodt xn, where din (resp. dont) is the input (resp. output) dimension, and we call the width of Then formulate the following universality principle:
f(E) w gyy++o(WW-o A(. ((W
Principle (Universality in MIP Initialization). As width tends to infinity, two different üid random initializations of biasless multilayer perceptron (MLP) induce identical training behavior as long as

---

1) they sample input and output weights the same way, and 2) they sample hidden weights with the same mean and variance
Here "identical training behavior" means that for any sequence of batches of data, performing SGD from either random initialization yields the same function after any number of training steps [
The key point in Principle 1 is that hidden weights are not sensitive to the exact distribution but the input and output weights are. We can demonstrate the sensitivity of the input and output weights on simple example. Consider the 1-hidden-layer (i.c. L 1) version of of. (I) with din = dout = 1 and with input and output weights tied, which we name U W W R" (where again is width):
((E) LUT
TTT(UE) ((UE).
Here, the additional factor compared to Eq. ( is just a normalization so that the network output will not blow up to infinity width becomes large; it's convenient but not essential for what we will discuss. Consider two alternatives (G) and (R) for sampling U:
n
(R) Ua 1 with prob. 1/2 or (G) Uo ~V 1).
Both methods have variance 1. Now suppose the nonlinearity o(x) equals to alI[ E -1/2, 1/2] Then
f(1) 0 with init (R) but f(1) + 22() > 0 with init (G)
a + 00, where -N 1). Thus (G) and (R) definitely do not induce identical training behaviors they are not even identical at initialization!
This example can be generalized to deep biasless MLPs to show that input and output weights are sensitive to the sampling distribution. Conversely, from our main theorem Theorem 3.7) below, will also be clear that hidden weights are insensitive to the sampling distribution.
What Principle 1 Gets Wrong in General Architectures Unfortunately, PPinciille is not correct if we go beyond biasless MLP. Indeed, if we just add bias to such an MLP, then one can easily generalize the example of Princiilee to show that biases are also sensitive to their exact sampling distribution. As the architecture becomes complex, it becomes difficult to figure out whether particular parameter tensor sensitive or not in an ad hoc fashion (e.g.. layernorm weights and biases? Self-attention weights? etc).
Principle for General Case Fortunately, there is simple rule tell which parameter tensors are sensitive based on the following.
Definition 2.1 ((33) ). Let P be a parameter tensor in a neural network of any architecture. As width becomes large, if P's size remains constant, then we say P is scalar-like; if exactly one dimension of P becomes large, we say vector-like; if exactly two dimensions P become large, we say P is matrix-like
Example 2.2. In Eq. (1) W ..... WL are all matrix-like while W and WL+1 are vector-like because din and dout are fixed even as n varies If we add biases,
=(E)= = bb+1 + WL+1 + o <(b + w
then b-+1 doou is scalar-like while b7 .....,." b I are vector-like.
Example 2.3. Some more advanced examples for practitioners, that may be skipped on first reading: If in Eq. (3) is convolutional (instead of dense) then the same categorization (of W b into scalar-, vector-, and matrix-like like) holds because the kernel size of convolutions is constant as width increases:
"ssame" means between the two initialization methods; the other hand, different weight entries have different distributions.
"This can be made even more general in the context of Tensor Programs; see 34 32
S 'This results in the so-called mean field parametrization 3 26 which a special case of the maximal
update parametrization [32].
'We further define tensor-like and so on, but practically speaking, all relevant large neural networks
(e.g.. BERT, GPT-3, etc [I[ will have most matrix like parameters due to storage and computation costs "tensor-like" parameters.
Think of dm and dout being tixed by the dataset, while you, model builder, freely vary
3

---

1) W. are matrix-like, 2) W W.+1, .b". .b'.... b" are vector-like, and 3) b is scalar-like. Layernorm weights and biases are vector-like the imput to that layernorm has exactly one hidden dimension (which is almost always the case in practice). Self-attention weights W W W are matrix-like if dhead is fixed as dmoded and Rhead increases if nhead is fixed as dmodel and dhead increases (one of which happens almost always in practice).
With this concept of scalar-, vector-, and matrix-like tensors in mind, we can formulate the general universality principle for neural network random initialization.
Principle 2 (Universality General Neural Network Initialization). As width becomes large, two different iid random initializations of a neural network of any architecture induce identical training behavio as long as 1) they sample scalar- and vector-like parameters the same way, and 2) they sample matrix-like parameters with the same mean and variance.
Remark 2.4. Note both Principle I and Principle 2 only give sufficient conditions for identical training behavior for all possible datasets and batches. But practically, when we only focus on specific datasets and specific training procedures at hand, it is possible that there could be weaker conditions that would ensure identical training behavior in such specific settings. For example, if the input dimension of a dataset is large and each example has entries that look iid then the input layer is not sensitive to the exact sampling distribution (aside from the first two moments) either. Nevertheless, Prinncppe I and Principle 2 yield guidelines that are generally applicable, upon which we may refine our reasoning,
wish, based on individual specifics.
3 Tensor Programs: Main Result
Colloquially, a Tensor Program is just a computation interleaving matrix multiplication and coor- dinatewise nonlincarities. In prior works, there have been several different formalizations of this concept. Here, we simplify and take the following definition.
Definition 3.1. Given matrices A A'.....A' A L E R initial vectors g' ........ Ma R and initial scalars c² Mo E R. consider the following iteration for i = Mo + 1, 1,..., M that generates new vectors g Mo+i .....9 E IR" and scalars Mo+1 cM E IR:
Wi W
c't 71
where x', = 6' .g.........
So 1: :'........ 1.
(4)
Here each Ø² is a chosen scalar function with (i- 1--1+++ +) 1) arguments and W is an n x n matrix. Each matrix W equals to either some matrix A' of the program or its transpose A The matrices W for different can possibly be the same. In this work, we shall call any computation of this form a Tensor Program (TP), or just a program for short. Thus each program is entirely determined by the Ma h=l U [ [ Mi=Mo+1 along with the correspondence between W and
(i-1)
data {Ai) Jj=1 U {g')) | A or A
Ma U Mc
This formulation of a Tensor Program is equivalent to NETSOR + in 31), as shown in Appendix E
As such, [Eq. (4) can express any computation expressible in a DL framework such as Py Torch [20] including gradient descent iterations of neural networks of any architecture, e.g. [32 34] This
expressivity allows one to treat a wide range of problems uniformly using just Theorem 3.4 below.
Example program For example, consider the first forward pass of a simple MIP with scalar input & and output (E):
(5) where & E R; U. V R W Ra and is activation function. We can express this in a TP as follows: g² + U, g2 + nV are the initial vectors and cº + E, 6² + 1 are initial scalars (where o² will just be ignored). A W is the sole matrix of the program. Then the program computes
f(E) V (Wo((U))
g
3 where 33 t- A² = =W and xt = = da sz c, ,)²) + (SU,)
1(5)=
4
where Nr. ②g Far ga 22 2,00 (nV)
SSee the discussion below Principle for the meaning of identical training behavior.

---

where c² and 9² are implicitly computed but ignored Extending this first step, the entire training process can further be written in TP, where the learned function outputs are expressed as scalars. for more examples.
See 29
30
34
32
Gaussian Tensor Programs The results achieved by the TP framework 28
31
34
so far most
commonly spring from the following version of the so-called Master Theorem:
Theorem 3.2 (Gaussian Master Theorem, original formulation of (T]. Consider Setup 3.3 below. Then, as n + co, for any pseudo-Lipschitz 1/,
1 Σ ④ Eg (ga......ge a=l
Ia M ,',...... ......,cMM m L
(6)
where V is a deterministic scalar given by certain recurrent formula.
Setup 3.3 (Gaussian Tensor Programs). Consider Tensor Program with M vectors .".....'"
g
R" and scalars c',..., .MM. Suppose 1) all initial vectors g g'....,g" Mo have standard Gaussian top entries 2) all initial scalars c², c.,..,, A. have almost sure limits as n - oo; 3) all matrices A have iid entries from NN, n "); 4) all the nonlinearities oi are pseudo-Lipschita! _
For example. Theorem 3.4 implies that Eq. (5)s function values f(E) after training converge to deterministic values in various infinite-width limits, in particular, the feature learning limit 33 33
We reformulate the above theorem in shorter form:
Theorem 3.4 (Gaussian Master Theorem, equivalent formulation), Consider Setup 3.3 Then, as + oc, every scalar ét converges almost surely deterministic limit è¹ which can be computed via a recurrent formula.
It is easy to see that the above statement is equivalent to the original Theorem 3.2 Indeed, any scalar in the program has the form + >., v'(g...... 9h ca..... ......, ca for some function 15, while any expression of the above form can be thought scalar in a new program. We are going to build upon the formulation of Theorem 3.4 since introduces fewer entities.
Non-Gaussian Tensor Programs We are going to generalize Theorem 3.4 non- Gaussian distribu- tions. But before we do so rigorously, we first formulate an easily statable principle that summarizes our results in an intuitive way.
Principle (Universality in Tensor Program Sampling). For simplicity, consider TP without initial scalars. As + oo, different iid random samplings of a TP's matrices and initial vectors result in identical limits of scalars as long as 1) they sample all initial vectors the same way, and 2) they sample all matrix entries with the same variance (and zero mean).
More generally, for programs with initial scalars, we just require the two samplings to have the same almost sure deterministic limits for them 12
Principle and Principle 2|ar are special cases of Principle 3 since any neural network can be expressed as TP. Principle Jitself will follow straightforwardly from Theorem 3.7
Now, let setup discussion of our main theorem, Theorem 3.7
Definition 3.5. We say f IR + R is polynomially smooth is C and its partial deriva- tives of any order are polynomially bounded, i.e. for any sequence (Pi,..., Pr) [k]", have or /l C(1 + |mn 1 + + I P)) for some C,p that may depend on (Pı,..., P,).
We intentionally simplified the formulation of TP to the form Eq. (4) in particular to simplify the proofs, but at the cost that expressing common computation encounters some redundancy as shown in this example.
"In the original formulation of (3]] initial vectors were assumed be sampled as (g...... Mo))) N(nn )nn) iid over a = 1,..., n for some j E RMM xin E Mo XMo However, one can always construct such vectors from ones with iid standard Gaussian entries using a linear elementwise map. This map be further absorhed into subsequent nonlinear maps in the program.
AA function f: I2* + R is called Lipschhi: if [[() f(y) s C| - yll + E |av/ l Iyal") for some C, d
We can, in fact, allow the initial scalars to have non- -on-dtterministic limits, in which case the non-initial scalars will have non con-eterministic limits as well.In this case, we want the two samplings to have identical limit distributions for the joint distributions of all initial scalars.
5

---

Setup 3.6. Consider Senui 3.3 but replace 3) and 4) with the following: 3*) there exists a sequence V3, ...... > such that all matrices A' have independent entries drawn from distributions with zero mean, variance n-1, and all higher kth moment bounded by ven vgn=k/2, 4*) all the nonlinearities of are polynomially smooth. We further require 5) all moments of initial scalars c¹ c',...;ch c.oo to exist.
Many widely-used nonlinearities, such as ReLU MaxPool, are not polynomially smooth, thus contradicting 4*). Nevertheless, if the original nonlinearity continuous and polymomially bounded (in particular, if it is pseudo-Lipschitz), one may apply Gaussian smoothing and get a polyno- mially smooth nonlinearity; see (ppendixx B The narrower the smoothing kernel, the better the approximation.
The most relevant specific scenario satisfying 3*) A ~ 1 D, where DD is a fixed distribution, vn
like uniform or truncated Gaussian with mean 0 and variance 1. In this case, Vi can just be taken to be the kth moment of D. But in general 3*) allows entries of A' to come from different distributions.
Our main result the following:
Theorem 3.7 (Non-Gaussian Master Theorem). Consider Setup 3.6 Then, as n + 0o, every scalar e' converges the same è as in Theorem 3.4 almost surely and in L for every E [1, oo):
& 12 Vp e (1,
In short, Theorem 3.7 relaxes matrix sampling be non- Gaussian and cun-identically-ditttee distributed in general, at the cost of requiring a) more smoothness in nonlinearities and b) all moments of initial scalars to exist. In past applications of TP, b) has always been satisfied but a) has not for relu networks. On the other hand, we also gain T.P convergence compared to Theorem 3.4 13 Note that /Sttup 3.6 still requires initial vectors to be Gaussian, but this not essential, as we discuss Section 4
On Tensor Programs with variable dimensions. While Setup 3.3 and 3.6 assume all hidden dimensions to be equal. Theorem h.4]s also for Tensor Programs with variable dimensions, see e.g. [31]. Since proof technique is based on interpolation between Gaussian and non-Gaussian weights, it can be straightforwardly extended to variable dimensions, as long as Theorem 3.4 holds in this setting.
Applications
As mentioned in the introduction, the Tensor Programs series of papers so far has proven a wide range of results, which typically have the characteristic of architectural universality, i.e. covering most existing neural architectures. But all of such results have assumed Gaussian weight initialization. Now, armed with Theorem 3.7, we show that the same results hold with non-Gaussian weight initialization as well under mild assumptions, thus extending them to other prevalent initializations such as uniform and truncated Gaussian. Thheoem 3.7 here acts like drop-in replacement for Theorem 3.4 in their proofs, except we need to add more smoothness assumptions on nonlinearities.
In general, coarse, qualitative statements, such as "wide neural networks at initialization are Gaussian processes," " still hold as stated. But note the exact quantitative statement, in this example regarding the kernel of the Gaussian process, can change as one changes the sampling distribution of vector-like and scalar-like like parameters, as indicated by Principle 2 and Principle 3
Definition 4.1. We shall call scalar random variable Gaussian image if it is image of standard Gaussian variable under a polynomially smooth function.
Below, we express rather abstract notion of a neural network that may seem bewildering to a reader who has not read previous papers in the Tensor Programs series. It helps to keep in mind specific examples, for example, the MLP with biases from Section 2. when reading the below statements. This notion generalizes all typical neural architectures (as shown in (29)) whose matrix like parameters randomly initialized with mean 0 and variance (11nn) (under mild moment conditions of SStup 3.6) and whose scalar- and vector-like parameters randomly initialized from Gaussian image.
We believe L convergence can be established without requiring so much smoothness in nonlinearities, but leave this to future work.

---

Setup 4.2. Consider a neural network f(E) = 1_ V with output layer weights V and embedding
vn (() on input &, and where n the dimension of the Tensor Program we describe next. Suppose 1) ((E) can be expressed as concatenation of a constant (wrt n) number of vectors x' E EI" (c.f.
Egq. (4) from a Tensor Program T in Setup 3.6, and 2) V's ss entries are initialized iid from a Gaussian image with mean 0 and variance I and are independent from the random objects of the program T.
NNGP Correspondence. The result below follows from Theorem 3.7
Corollary 4.3. Any neural network function described in Setup 4.2 14 converges in fintte-imensional distribution to a Gaussian process in the limit of n -
In Appendix D we prove similar result without assuming the activation functions to be polynomially smooth (we require I. Lipschitzness instead), see Corollary D.3 This result will follow from our non-
Gaussian Lipschitz Master Theorem, Theorem D.2 The proof of the above corollary follows the
same lines as the proof of Corollary D.3 given in Appendix D G
As discussed above, the kernel of this process in general will be affected by the distribution of vector-like parameters. For example, take f({) = V/ (UE) in /Sttpp 4.2 where & IR and is the indicator function of the interval [-1/2, 1/2]. Then sampling Ua as =1 with equal probability implies that f is identically 0 while sampling Ua from a standard Gaussian implies converges to a nontrivial Gaussian process.
NTK Correspondence. We can directly plug our Theorem 3.7 into the proof of [34] and obtain and Corollary 4.4. ConsiderSSeupp 4.2 and assume the loss function is continuously differentiable in the network output. Then under NTK parametrization and SGD weight update M in the limit of - oc, 1) the NTK of the network at any optimization step converges pointwise almost surely to finite deterministic limit & that does not depend on the timestep, and 2) moreover, the network function evolves according to kernel gradient descent with kernel O.
As in the NNGP case, the NTK's s infinite- limit can also be affected by the distribution of input and output weights. The same example there illustrates this: the network's 's limit NTK as e is equal to the kernel of the limit Gaussian process because e has 0 derivative almost everywhere.
We empirically validate/Corollane Corollaries 4.3 and 4.4 in Appendix N
While our non-Gaussian Lipschitz Master Theorem, Theorem D.2, mentioned above allows to generalize Corollary 4.3 to ReLU nets, it does not allow us to generalize Corollary 4,4 the same setup (because Tensor Program expressing the backward pass of a ReLU net has ReLU derivatives, which are not even continuous, as nonlincarities)
Random Matrix Theory. Our Theorem 3.7 implies the semi-circle law for non-Gaussian Wigner matrices, the Sarchenkk-Passtrr law for AA where A is Gaussaas,,,, and Free Independence Principle (FIP) for Tensor Programs with non-Gaussian initial weights, thus generalizing TP3 [31]; we discuss these results in Appendix C Moreover, since our Theorem 3.7 guarantees convergence in mean, we were able to state FIP without assuming linearly bounded nonlinearities as in 31.
14 As can notice from analyzing the proof of Corollary D.3 we do not need the output weights V to be Gaussian images. Instead, we could assume that they are iid with zero mean, unit variance, and higher moments existing. However, we are not able to prove Corollary 4.4 heloo without assuming the entries V to be Gaussian images. We assumed weaker setup since we wanted to have the same setup for both corollaries.
Notice we allow a weight matrix and its transpose be both involved in the forward pass of the network, in contrast to (29), but in line with more general theorem in [31]
''There is, however, small subtlety when applying the proof technique of Corollary D.3 to Corollary 4.3
prove Corollary D3. works for Lipschitz nonlinearities. therefore we could embed y into the underlying tensor program. However. Theorem 3.7 does not work for Lipschitz nonlinearities and it is not obvious if we can use polynomially smooth functions as test functions for weak convergence. Nevertheless, we can take bounded Lipschitz test function db and smoothen it with kernel of width 8 > 0 the same way we do in the proof Theorem D.2 Then it is casy see that |c e² < surely for any 8 > O, where c is the same as e in the proof of
There we use test function 1 and take he Lipschitz and bounded. Theorem D.2 used to prove
Corollary D.3 D.3 but with d-smoothed version of db. Since |E è s |Ec- Eed |E 20 + 12" é). we
get lim sup, [Ec- é S 8 + |2 - c). Taking infimum over > 0 gives E.C. +
17 For the exact meaning of this setup, see (34.
7

---

Classification of Infinite-Wiide Limits. Consider now an L-hidden-layy -laer biasless perceptron with width n trained using stochastic gradient descent (SGD). As in [32 Sec 3.2], we shall assume in this section that this perceptron's 's nonlinearities are either tanh or the so-called a-gelu for sufficiently
small a (c.f. 32 Assm 3.I 1B Note both tanh and o-gelu are polynomially smooth.
We generalize the notion of abbepaarametiization from [32 to non-Gaussian initializations:
Definition 4.5. Let DD be distribution with mean 0, variance 1, and all moments finite. An abc- parametrization for DD specified by a set of numbers {at, by b1)=11 U {c). parametrizes the MIP as follows: 1) each weight factors as W = n -- wf for the actual trainable parameter w', 2) the weights are sampled iid w n DD initialization, and 3) the SGD learning rate is taken rm for some constant 1/.
In their Dynamical Dichotomy Theorem, [32 classified all tbe-paramettrizations where D = NV 1) into the following categories: stable, nontrivial, kernel regime, and feature learning. Here, the stable and nontrivial categories overlap, and the kernel and feature learning regimes are mutually exclusive categories within their intersection. Each category characterized by some set of linear inequalities in {ar, by) / {c) (c.f. [32] Sec 3)). turns out that all but one abeeparametrization the so-called p--aaamtttriatiinn (abbreviated μP),, exhibit defects in the width limit (such as losing the ability to learn features). This is formalized in [32] Thm 5.6]. [33] then showed that μP gives rise to new technology called uTransffr that allows one to, for the first time, tune extremely large neural networks too expensive to train more than once, such as GPT-3 M.
Using Theorem 3.7, we see that all of the above notions, originally defined for Gaussian initialization, in fact are distributionally universal:
Theorem 4.6. Let D' be two distributions satisfying the moment conditions of Definition 4.5 Then an bbb--aaramerrizaioonn for D in feature learning (resp. kermel/stablehnnnrivvaaa regime iff is so for D'. It is the u-parametrization for 'D iff it is so for D'.
This suggests that the Transfer technique mentioned above works regardless of the initialization distribution of weights.
On potential societal impacts. Our work concerns generic behavior of neural nets in the limit of infinite width and therefore does not provide any foreseen societal impacts. The only direct practical application of our work we are aware of is theoretical justification for the hyperparameter tuning method of TP5 [33] for non-Gaussian weight initializations. However, what our work provides is merely justification for this method, while the method itself existed before our work (and was well-justified for Gaussian weight initializations).
Proof of Theorem 3.7
Limitations of the Proof Technique of Theorem 3.4, The proof of Theorem 3.4 as given in [T uses the Gaussianity of the matrices A' very essential way. leverages the property of multivariate Gaussians to remain Gaussian after conditioning on linear constraints.
In partieular, it lets one understand the distribution of g conditioned gi-l,..., .....'''..'.. g". such conditioning, W' is in general no longer iid but nevertheless constrained only by the following equalities (where now  and x should be considered deterministic under such conditioning):
Indeed, under
وو Wi for all j < and where W = W or W
This constraint is linear in W (even though it's generally nonlinear in g² g²..... nonlinear function of them). Therefore, when conditioned on gi. .g''. Gaussian, albeit with some conditional mean and covariance. Quickly glossing over the rest of the proof, this allows one to reason about the conditional distribution of g and eventually to reduce the almost sure convergence of the (i + 1 1)th scalar cf+1 to the almost sure convergence of the ith scalar in different program. Then an inductive argument over M for all programs of size M can prove Theorem 3.4. A more detailed proof sketch can be found in 31 Section 6).
"".....'"" g" the matrix W is still
since xd is
Now, when the matrices W are no longer Gaussian, this argument completely breaks down as we no longer have good control of the conditional distribution of W for general entry distributions. We
This means o-gelu smoothly approximates relu sufficiently well.
8

---

therefore apply different argument. The idea is to interpolate the weights from the Gaussian ones, for which convergence of ct is given by [Theorem 3.4 to the non- Gaussian ones, and show that c' does not change along this interpolation in the limit of large
Our Proof of Theorem 3.7
Definition 5.1 (Interpolated Program). Given a program Setup 3.6 Á' denote an iid Gaussian matrix with the same mean and variance as the non-Gaussian matrix A' of the program. Then define the interpolated program T(t) for t E [0, 1] as follows: T(t) is identical to T except that its matrices take the following values:
A (t) def A' cos
π (2t)
+ A sin
22t for all t E [0, [0 and i = Mo + 1.....,..... M,
Naturally, (t) inherits the same definition. The vectors and scalars in the program will change continuously as f varies, and consequently we write them as gt()) and ct(t) for = 1, ,,...,A6. M
In Section 5.1 we will discuss why this specific form of interpolation 20
is important in our case.
Here 0 corresponds to the Gaussian program, while 1 corresponds to the "target" non- Gaussian one. Eventually, we aim to show that all scalars c¹ (t). almost surely in the limit of n - oc, will remain constant as t varies from 0 to 1 (as stated by Theorem 5.2 shortly after setting some notations), thus proving that the Gaussian and non- Gaussian programs have the same limits.
Notations For any object (matrix, vector, or scalar) w()) of this interpolated program, we shorthand
(1) def d
(I).
dt
Big-O notation, e.g.. 0/2, always suppress multiplicative constants independent of n, but which may depend on everything else. The supremum sup, is always taken over E [0, 1]. Theorem 5.2. Consider a program in Setup and let e be a scalar in it. For any 2 1, we have 21
sup E |o(t)1 O(n -P2)), as n + co.
In other words, this result says é(t) ) is small in I norm uniformly over all t. With Theorem 5.2 some routine calculations (detailed below) then show that c(1) converges almost surely and in L to the same limit as c(0) yielding a proof of Theorem 3.7 as desired. Theorem 5.2 proven in Appendix L but we shall demonstrate it simple example in Appendix A
Routine Calculations Finishing the Proof of Theorem 3.7 from Theorem 5.2 is proven, we have, for any 2 1,
Once Theorem 5.2
S. ε(t)dt < {f ((((1) O(n n/2)
P
E =]c(1)-((0)) =
where the inequality follows from power mean or Hölder's inequality and the last equality follows from Theorem 5.2. Since the RHS goes O with n, this implies that c(1) - e(0) converges in L as n + oc. With a standard application of Borel-Cantelli lemma, this also implies almost sure convergence of c(1) c(0). Since c(0) converges è almost surely by the Gaussian theorem, Theorem 3.4, the same holds for c(1).
In the process of proving Theorem 5.2 we will have proved pth moment bound on c(1) for every finite p 2 (Lemma J.6) Then a standard truncation technique uses this to convert the almost sure convergence of c(1) I. convergence for all p E [1, oc); see Theorem M.3
remains to prove Theorem 5.2] which we demonstrate on simple example Appendix aatter discussing the key properties of our interpolation (Eq. (7)..
but note that they are invariant to t for any n when i S Mo because by construction, the initial vectors and scalars are the same between the Gaussian and non- non-Gaussian programs.
Thhs interpolation similar to that used in 2 in context of approximate message passing, which can be considered as an application of the TP framework to a specific kind of programs; see Apppeddix Flor comparison.
The hidden multiplicative constant here only independent of but can depend and other details of the program.
Note since c(0) (the scalar in the Gaussian program) is not guaranteed by Theorem 3.4 converge in L", the above L2 convergence of e(l) c(0) does not immediately imply the L convergence of c(1).

---

5.1 Key Properties of Our Matrix Interpolation
There are may ways to interpolate between (or, more generally, to couple) and A other than Eq. (7) Why did we Eic (7)² The following lemma proven in Appendix G Summmriiess the key properties of it, where especial attention should be paid to the fact that E aá = O:
Lemma 5.3 (Interpolation Properties). For any matrix entry a(I) (1) of a program in Setup 3.6 (I) (t(t) = E ((t))(()) = O for all t; (2) For any integers j. > 0 with sum I' = + k.
sup, E (a(tt S ven" 1t/2, 1/2, where Vg is the scaled moment bound in Setup 3.6
The fact Ea 0 is crucial and the main reason we picked the specific form Bq.. (7) of the interpolation. will allow us to kill leading terms in the Taylor expansions important to proving Theorem 5.2] We demonstrate this in example in Eq. (II) On the other hand, (2) is just very strong version of the obvious statement that "a and both have typical size 1/ //v,," but importantly, this is uniform over & [0, 1].
6 Related works
The Tensor Programs series discusses different applications of the (Gaussian) Master theorem proven
by (8)- They include: Gaussian process behavior initialization (TP1, 29 convergence
kernel method (TP2, (30., Free Independence Principle (TP3, (II. dynamical dichotomy and
u-parameterization (TP4, 32 and finally application of pP to hyperparameter tuning (TP5, (33).
Neural networks converge to Gaussian processes as their width goes to infinity, as was proven by [13 16 for fully-connected nets, and by [8 3 for convolutional nets; see also [9]. Using the Master theorem, 229 showed that this behavior holds for a very wide class of architectures, including not only convolutional, but also graph convolutional and recurrent neural nets, ResNets, networks with batch normalization, and networks with attention. The seminal work of [14] demonstrated that under certain parameterization, the learning dynamics neural net converges that of kemel method. The corresponding kernel was called Neural Tangent Kernel, or NTK, and drawn a lot of attention in recent years. While the result of [14] was proven only for fully-connected nets with smooth activations, the Master theorem allows to generalize this result for a wider class of architectures (the same mentioned above), see 30 22 21 27 and others studied trainability of very deep and wide neural networks using random matrix theory. Their analysis crucially relied on the assumption that hidden representations of neural network at initialization were freely independent from the weights. TP3 [31] was among the first works to validate this assumption rigorously; see also [19]. Infinite-width behavior of neural net depends on scaling of its hyperparameters (like initial weights variance and learning rate) with width. Dynamical dichotomy proposed in TP4 (2) is classification of scalings that are meaningful in a certain sense. Another classification of scalings with a different notion of meaningfulness was proposed earlier by 7 g but only for two-layered networks.
A distribution universality property similar to our Theorem 3.7
was shown by 2 for approximate mes- saging passing. However, their model does not cover most of possible neural network computations;
see Appendix F for discussion.
7 Limitations of our results
First, our Theorem 3.7 applicable only to Tensor Programs with smooth nonlinearities, which rules out several popular activation functions like ReLU or MaxPool. Our Theorem D.2 (see Appendix D does not really solve the issue since a Tensor Program expressing the backward pass involves derivatives of the activation functions, which are even continuous for ReLU. As a workaround, we could consider their smoothed versions (e.g. Softplus instead of ReLU) with a controllable smoothness parameter, and put this parameter very close to zero, thus getting "almost ReLU"*.
8 Conclusions
We present generalization of the Master theorem of 28 to non-Gaussian weight initializations. Our generalization allows for the same applications as the original Master theorem, thus broadening the scope of applicability of the Tensor Programs machinery.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901 2020.
[2] Wei-Kuo Chen and Wai-Kit Kit Lam. Universality of approximate message passing algorithms. Electronic Journal of Probability, 26:1-44, 2021.
[3] Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. arXiv preprint arXiv:1805.09545 2018.
[4] Jacob Devlin, Ming-Wor Weig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018.
[5] Adrià Garriga-Alona Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. arXiv preprint 1808.00580 2018
[6] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
[7] Eugene Golikov. Towards a general theory width limits neural classifiers. In International Conference on Machine Learning, pages 3617-3626. PMLR, 2020.
[8] Eugene A Golikov. Dynamically stable infinite-width limits of neural classifiers. arXiv preprint arXiv:2006.06574 06574, 2020.
[9] Boris Hanin. Random neural networks in the infinite width limit as gaussian processes. arXiv preprint arX:v:2/07..0166, 2021.
[10] Michael Hardy. Combinatorics of partial derivatives. arXiv preprint math/0601149, 2006.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034 2015.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Geoffrey E Hinton, Alex Krizhevsky, and Ilya Sutskever. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25(1 106- 11 11411, 2012.
[14] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571-8580, 2018
[15] Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sock-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:
2017.
[16] Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahra- mani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
[17] Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis, University of Toronto, 1995.
[18] Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolañia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv:1810.05144 2018.
11

---

[19] Leonid Pastur. On random matrices arising in deep neural networks. gaussian case. arXiv preprint arXiv:2001.005 001108880....8, 2020.
[20]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[21] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in neural information processing systems, pages 4785-47955 2017.
2017.
[22] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[23] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances neural information processing systems, pages 2360-3368 2016.
[24] Grant M Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of neural networks: An interacting particle system approach. arXiv preprint arXiv: 00915, 2018.
[25] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- proccpppggggiing errors. nature, 323(6088) 1986.
[26] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130o0onn,,333331)1 130(3):1820-1855 2020.
[27] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pen- nington. Dynamical isometry and vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393-5402. PMLR, 2018.
field theory of cnns: How train 10,000-layer
[28] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian pro- cess behavior, gradient independence, and neural tangent kernel derivation. arXiv preprins 1rXiv:190..04760. 2019.
[29] Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architec- ture are gaussian processes. arXiv preprint vrxiv:/9100 /2478, 2019.
[30] Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. 14548, 2020.
[31] Greg Yang. Tensor programs ii:: Neural matrix laws. arXiv preprint arXiv:2009.10633 2020.
[32] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in anfnite-width neural networks. In International Conference on Machine Learning, pages 1 11727-11737. PMLR, 2021.
[33] Greg Yang. Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint S.Vi::2203.03466 2022.
[34] Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In International Conference on Machine Learning, pages 11762- 11772. PMLR, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 3 for scope and [Sectinn and for contributions.
(b) Did you describe the limitations of your work? [Yes] See SceeSection? 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the last paragraph of Seecion 4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sectiinn Setup 3.6 particular.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 5 and
Ap
pendices D
G and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See/Appendix N.2
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See plots in Ampppeddxxx N]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See[Appendix N.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "cTRSILK1in", "reviewer": "Reviewer_irWk", "summary": "The authors generalize the Master Theorem of Tensor Programs (TP) by demonstrating that non-Gaussian initializations result in many of same consequences of TP including (perhaps most interestingly) the NNGP correspondence (infinitely wide neural networks behave as Gaussian processes). Differences between this result and the original Gaussian initialized TP include the following: requirement of smooth nonlinearities, and weaker convergence (in probability versus almost surely). Much of the manuscript is dedicated towards proof. ", "questions": "1. Why do the authors consider it useful to note that the semi-circle distribution and Marcenko-Pastur laws for non-Gaussian matrices fall out of Theorem 2? Going through tensor programs is not necessarily easier than existing proofs. What perspective/insights might deriving these results from TP have to offer? ", "limitations": "Limitations and societal impacts are discussed. ", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The formalism introduced by TP was (and still is) of both theoretical and practical use (say, for hyperparameter tuning). Generalizing it even further is beneficial for the DL community. \n-Proof of the non-Gaussian Master Theorem is thoroughly detailed. \n-Nearly all of the consequences of the Gaussian Master Theorem are obtained. \n\nWeakness:\n-Notation is cumbersome. This is a challenging paper, indeed. Tensor Programs I (G. Yang) expends serious effort towards the development of NETSOR to aid the architectural universality result, but this paper is at times a pure exercise in analysis. Any type of diagram or visual to accompany the proof of Theorem 2 would have been nice. \n-Requirement of smooth nonlinearities (although this is addressed, several times over). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5f4YWKe24tt", "reviewer": "Reviewer_88YS", "summary": "The paper extends the Tensor Program to the non-Gaussian case, which justifies a previous hyperparameter tuning method for non-Gaussian weight initialization.", "questions": "The authors may consider moving the proofs to the appendix and using the space to explain more about Tensor Program and its application.\n\nFor example, in Section 2, readers may want to know how to convert a practical convolutional networks (with weight sharing convolutions and batch normalization) to Tensor Program. Since the authors emphasize on universality, it is also informative to discuss whether the scheme is applicable in graph neural networks, where the central limit theorem may not hold given sparsity.\n\nMoreover, in Section 3, it is useful to discuss how to use the theoretical result for hyperparameter tuning, instead of citing pror works. Lastly, since the result only hold for infinite-width case, it is helpful to include some empirical study to show the gap between theory and practice.", "limitations": "Not applicable.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Since I don't have a background in Tensor Programs, I can only comment on writing.\n\nThe proposed theory helps to explain some previous practices in learning neural networks; however, it does not suggest new techniques nor has empirical verification. Moreover, the paper is generally dense and dry, therefore challenging for non-specialists to follow. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5Aes-X7Rm5N", "reviewer": "Reviewer_eCbt", "summary": "This work continues in the spirit of Tensor Program series of papers. Starting from a theoretical\nresult showing that nearly any neural architecture with Gaussian weight initialisation in the limit\nof infinite width exhibits Gaussian Process behaviour, this paper extends this theorem to apply where\nweights have non-Gaussian initialisation. This requires assuming that all matrices (A) used have iid\nentries from a Gaussian distribution, all bias vectors also come from an iid Gaussian distribution, and\nthat all non-linearities be polynomially-smooth.\n  ", "questions": "  The paper offers to me seems a fairly modest contribution. With the restrictions needed to not require Gaussian\n  weights it's unclear why simpler extensions could not have been done. For example, Appendix section A. states that\n  images of smooth polynomial maps can be used to get around the Gaussian initial bias and layer weights.\n  Why could the same trick not been used in original formulation?\n", "limitations": "I believe the authors have adequately addressed the limitations and potential negative societal impacts of the work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "  Originality:\n\n  This work is very original and I am not aware of any other theoretical contributions of this sort\n  in the literature. \n\n  Technical Quality:\n\n  The work is technically sound and although I did not carefully inspect the proofs I don't have\n  significant reasons to doubt the results. Although there are not experiments in the paper, given\n  the theoretical nature of the contribution I don't think it's necessary.\n  \n  Clarity:\n\n  The work relies heavily on familiarity with previous Tensor Programs papers for background and motivation\n  which makes the paper and its contribution less self-contained. But given some familiarity, the paper was\n  straightforward to read and follow.\n  \n  Significance:\n\n  The paper offers to me seems a fairly modest contribution. With the restrictions needed to not require Gaussian\n  weights it's unclear why simpler extensions could not have been done. For example, Appendix section A. states that\n  images of smooth polynomial maps can be used to get around the Gaussian initial bias and layer weights.\n  Why could the same trick not been used in original formulation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3dhIdK91_tR", "reviewer": "Reviewer_3QrM", "summary": "The paper generalizes the Tensor Programs framework, which computes infinite-width limits of deep networks / differentiable computation graphs under the assumption that weights are randomly initialized from independent Gaussians with variance 1/n, to the case of iid non-Gaussian initializations. ", "questions": "nits:\n- line 24: 'applications of Law of Large Numbers' doesn't parse, I would say 'applications of *the* Law ...'\n- line 41: typo 'identially' -> identically\n", "limitations": "The technical conditions and limitations of the result are clearly articulated and discussed.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The result in this work is an incremental extension of the Tensor Program framework to non-Gaussian weight initializations, and is clearly not intended as a standalone exposition of the framework. Unfortunately I have only a vague familiarity with the existing Tensor Program line of work and so can only provide superficial comments, not critique the result in any detail.\n\nSignificance: this result contributes to the theory of infinite-width limits of neural nets, which is certainly of general interest and has motivated interesting new methods that may be practical in some settings, e.g., for hyperparameter search. The specific contribution of this paper is incremental, and the result (although apparently nontrivial) does not seem particularly surprising, so its broader significance seems limited at best. Does this extension 'deserve' to be its own top-conference paper, rather than (say) an appendix to the general framework? I don't feel qualified to judge, but given the community's recent interest in this line of work I'm willing to give it some benefit of the doubt.\n\nQuality and clarity: the paper is generally well-written. Technical conditions and results are clearly stated, as are the limitations and the relationship to previous results. I did not attempt to check the proof (section 4) but it appears well-motivated, structured and carefully reasoned; I believe it is likely correct. \n\nOriginality: the proof technique appears to follow a similar approach used in other recent work (Chen and Lam 2021), but its application and extension within the Tensor Programs framework is novel and, I think, nontrivial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept