Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes
Authors: Thiziri Nait Saada
Alireza Naderi
Jared Tanner
Venue: ICLR 2024 poster (2024)
Keywords: Deep Neural Networks
Gaussian processes
Neural Networks initialisation
Edge of chaos
Large width limit
Mean-Field
Primary area: general machine learning (i.e., none of the above)

Abstract: The infinitely wide neural network has been proven a useful and manageable mathematical model that enables the understanding of many phenomena appearing in deep learning. One example is the convergence of random deep networks to Gaussian processes that enables a rigorous analysis of the way the choice of activation function and network weights impacts the training dynamics. In this paper, we extend the seminal proof of Matthews et al., 2018 to a larger class of initial weight distributions (which we call pseudo-iid), including the established cases of iid and orthogonal weights, as well as the emerging low-rank and structured sparse settings celebrated for their computational speed-up benefits. We show that fully-connected and convolutional networks initialised with pseudo-iid distributions are all effectively equivalent up to their variance. Using our results, one can identify the Edge of Chaos for a broader class of neural networks and tune them at criticality in order to enhance their training. Moreover, they enable the posterior distribution of Bayesian Neural Networks to be tractable across these various initialization schemes.

Published as a conference paper at ICLR 2024
BEYOND IID WEIGHTS: SPARSE AND LOW-RANK DEEP NEURAL NETWORKS ARE ALSO GAUSSIAN PROCESSES
Thiziri Nait Saada, Alireza Naderi & Jared Tanner Mathematical Institute University of Oxford (nai Hsaitsaadat, naderi, tanner) @maths. ox. ac. uk
ABSTRACT
The infinitely wide neural network has proven useful and manageable math- ematical model that enables the understanding of many phenomena appearing in deep learning. One example is the convergence of random deep networks Gaussian Processes that allows a rigorous analysis of the way the choice of acti- vation function and network weights impacts the training dynamics. In this paper, we extend the seminal proof of Matthews et al. (2018) to a larger class of initial weight distributions (which we call PSEUDO -IIDD) including the established cases of IID and orthogonal weights, as well as the emerging low-rank and structured sparse settings celebrated for their computational speed-up benefits. We show that fully connected and convolutional networks initialized with PSEUDO-IID, distri- butions are all effectively equivalent up to their variance. Using our results, one can identify the odge-offChhas for broader class of neural networks and tune them at criticality in order to enhance their training. Moreover, they enable the posterior distribution of Bayesian Neural Networks to be tractable across these various initialization schemes.
INTRODUCTION
Deep neural networks often studied at random initialization, where in the limit of infinite width, they have been shown to generate intermediate entries which approach Gaussian Processes. Seemingly this was first studied for one-layer networks in Neal (1996) when the weight matrices have identically and independently distributed (IID) Gaussian entries, and became a popular model for deep networks following the seminal results for deep fully connected networks in Lee al. (017)) and Matthews et al. (2018). Specifically, the latter formulated a proof strategy that quickly became cornerstone in the field, paving the way for various extensions of the Gaussian Process limit (see Section I.I). The resulting Gaussian Process has proven to be of practical interest for at least two reasons. First, emerges as a mathematically motivated choice prior in the context of Bayesian Neural Networks. Secondly, it helps the analysis of exploding and vanishing gradient phenomena done in Schooenhllz jet al. (207)) and [Peniigtonn et al. (2018). amongst other network properties.
In this paper, extend the proof of the Gaussian Process from Matthews al. (0118) to larger class of initial weight distributions (which we call PSEUDO-IID). PSEUDO- distributions include the already well-known cases of ITD and orthogonal weights. Moreover, explore other important settings that conform to our more general conditions (e.g. structured sparse and low-rank), yet for which the Gaussian Process limit could not be derived previously due their violation of the assumption.
Why studying low-rank and structured sparse networks at initialization? In recent years, deep learning models have significantly increased size, while concurrently smaller AI chips have been preferred. The most widely studied approaches for bridging the gap between these two aims are to reduce the number of parameters by either pruming the network to be sparse or using low-rank factors. The lottery ticket hypothesis Frankle & Carbin (2019) has given empirical evidence of the existence of pruned net-networks (i.e. winning tickets) achieving test accuracy that is comparable and even superior their original network. Unfortunately most pruning methods produce sparsity patterns

---

Published as a conference paper at ICLR 2024
that are unstructured. thereby limiting potential hardware accelerations (see Hoefler et al. (2021) Remarkably, [Chen et al. (2022) revealed some winning tickets with structured sparsity, that can be exploited by efficient hardware accelerators GPUs and FPGAs to reduce greatly their in-memory access, storage, and computational burden (Zhu et al. ). Likewise, the emergence of low-rank structures within deep networks during training is observed where the low-rank factors are aligned with learned classes Han et al. (2022). These factors are shown to improve the efficiency and accuracy of networks, similarly the lottery ticket hypothesis Osawa al. (2017); Yang al. (2020a b); Tai
et al. (2022); Price & Tanner 2023); Karimi Mahabadi et al. (222) Berlyand et al. (2023). If such computationally efficient winning tickets exist, why would one want to train dense (or full-rank) network at high computational price only to later reduce it to be sparse (or low-rank)? This question has given rise to single-shot approaches, often referred to as Pruning at Initialization (Pal) such as SNIP Lee et (2019), SynFlow Tanaka et al. (2020), GraSP Wang et al. (2020a), NTT & Zenke (2020). Consequently, the present work delves into the properties such parsimonious networks at initialization, within the larger framework of PSEUDO-IID regimes.
1.1 RELATED WORK
To the best of our knowledge, the Gaussian Process behaviour in the infinite width regime was first established by Neal (1996) in the case of one-layer fully connected networks when the weights sampled from standard distributions. The result has then been extended in Mattheww al. (2018) to deep fully connected networks, where the depth is fixed, the weights are distributed as IID Gaussians and the widths of the hidden layers grow jointly infinity. Jointly scaling the network width substantially distinguishes their method of proof from the approach taken in Lee et al. (017). where the authors considered sequential limit analysis through layers. Indeed, analyzing the limiting distribution at one layer when the previous ones have already converged Lee et al. (2017) significantly differs from examining it when the preceding layers are jointly converging to their limits Matthews et al. 2018). The latter approach now serves a foundational machinery in the infinite width analysis from which numerous subsequent works followed. This paper is one of them.
Since this Gaussian Process behaviour has been established, two main themes of research have further been developed. The first consists of the extension of such results for more general and complex architectures such as convolutional networks with many channels Novak et al. (2020), Ciarriga-Alonso eet al. (2019) or any modern architectures composed of fully connected, convolutional or residual connections, as summarized in Yang) (2021), using the Tensor Program terminology. The second line of work concerns the generalization of this Gaussian Process behaviour to other possible weight distributions, such as orthogonal weights in Huang et al. (2021) or, alternatively, any IID weights with finite moments as derived in Hanin (2021). Note that the orthogonal case does not fit into the latter as entries are exchangeable but not independent. Our contribution fits into this line of research, where we relax the independence requirement of the weight matrix entries and instead consider PSEUDO-IID distributions made of uncorrelated and exchangeable random variables. This broader class of distributions, Definition 2 enables us to present unified proof that strictly generalizes the approaches taken so far and encompasses all of them, for two types of architectures, namely, fully connected and convolutional networks.
1.2 ORGANIZATION OF THE PAPER
In Section 2 focus on fully connected neural networks, formally stating the PSEUDO-IID regime and its associated Gaussian Process (GP) in Theorem[] whose rigorous proof is provided in Appendix A Our PSEUDO-IID regime unifies the previously studied settings of IID and orthogonal weights, while also allowing for novel initialization schemes conform to the Gaussian Process limit, such as low-rank and structured sparse weights. We expand the definition of PSEUDO- IID distributions to convolutional kernels and extend the resulting Gaussian Process limit to convolutional neural networks (CNNs) in Section 2.2 In Section we provide examples of PSEUDO-IID distributions in practice, supporting our theoretical results with numerical simulations. Moreover, we allude to two important consequences of the GP limit in 3.3. namely the analysis of stable initialization of deep networks the so-called Edge-of-Chaos (EoC), well as tractability for the posterior distribution of Bayesian Neural Networks, in our more expansive PSEUDO-IID regime. Lastly, in Section 4 we review our main contributions and put forward some further research directions.
2

---

Published as a conference paper at ICLR 2024
2 GAUSSIAN PROCESS BEHAVIOUR IN THE PSEUDO-IID REGIME
We consider an untrained fully connected neural network with width N2 at layer & E {1, L+ ++11, 1}. Its weights W(() E IR Nex Ne and biases b(0) E R' at layer é are sampled from centered probability distributions, respectively MWv and Ps (4) Starting with such a network, with nonlinear activation R + R. the propagation of any input data vector 20) = X S No through the network is given by the following equations,
NL-1 (x) - Σ w
( (z)
(x) bb)
-
where (x)) E R.V: is referred to as the preactivation vector at layer É. or feature maps.
In this work, we consider all biases to be drawn IID from ((, of)) in both fully-connected and convolutional cases. Gaussian biases are in fact needed to ensure the gaussianity of the feature maps. Our focus will thus be restricted to the weight matrices and tensors that result Gaussian Process. PSEUDO-IID distributions (formally introduced in Def. ( only need to meet some moment conditions along with an exchangeability requirement that we define below.
Definition 1 (Exchangeability)) Let XI.... .x, scalar, vector or matrix-valued random variables. We say (X:)== are exchangeable if their joint distribution is invariunt under permulations, i.e. (X1...., Xx)) & (Xa(1).... .Xa(n)) ( for all permutations : [n] - [m]. A random matrix is called row- (column-) exchangeable if its rows (columns) are exchangeuble random vectors, respectively.
cow-exchnngeable and column-exchii exchangggcaee weight matrix W E Rxx" wise exchangeable, which means its distribution is not typically invariant under arbitrary permutations of its entries; particularly, out of (mn)) possible permutations of the entries, W only needs to be invariant under m!n! of them an exponentially smaller number. We can now define the family of PSEUDO-IID distributions.
x is not general entry-
Definition 2 (PSECDO-IID). Let m, n be two integers and a E IK be any fixed vector. We will say that the random matrix W = (W,) E gmxx is in the PSEUDO-IID distribution with parameter o² if
(i) the matrix is row-exchangeable and column column-exchangeabble
(ii) its entries are centered, uncorrelated, with variance E(W?) a²
(iii) E Σラ Wir K|lallnn for some constant K,
(iv) and lim क. B(W Wny W Wea.))) di &igin for all j # j'.
When W() has IID Gaussian entries and the other weight matrices W(0, 2 < l < L 1, of neural network (see equation [] are drawn from a PSEUDO-IID distribution, we will say that the network under the PSEUDO-ID I1D regime.
The PSEUDO-IID conditions (i)-ivv are included to serve the following purposes: Conditions (i) and (ii) allow for non-lID initializations such as orthogonal weights while being sufficient for a CLT-type argument to hold. The variance scaling prevents the preactivations in equation frrm blowing up as the network's 's size grows. Condition (iii) requires the moments of the possibly dependent weights to behave like independent ones, while condition (iv) constrains their cross-correlations. to vanish fast enough. Together they maintain that the limit, these variables interact as they were independent, and this is all our proof needs. The importance of condition (iii) is expanded upon in Appendix G It worth noting that because the first layer's ' weight matrix has only one dimension scaling up with n, the tension between the entries' dependencies and the network size cannot be resolved, thus take these weights to be IID Whether some of the conditions are redundant still remains an open question.
In fact, it is enough for (()) to have IID rows. We assume Gaussian IID entries for simplicity, yet this can he further relaxed the cost of a more complicated formula for the first covariance kernel (see equation 2T)
3

---

Published as a conference paper at ICLR 2024
Throughout this paper, consider specific set of activation functions that satisfy the so-called linear envelope property, Definition 3] satisfied by most activation functions used in practice (ReLu, Softmax, Tanh, HTanh, etc.).
Definition 3. (Linear envelope property) A function Φi RR - R is said to satisfy the linear envelope property if there exist c, M 2 O such that, for any x E IR,
[0(x)| S M
2.1 THE PSEUDO-IID REGIME FOR FULLY CONNECTED NETWORKS
Our proofs of PSEUDO-IID networks converging to Gaussian Processes are done in the more so- phisticated simultaneous width growth limit as pioneered by Matthews al. (2018). For a review of the literature deep networks with sequential vs. simultaneous scaling, see Section [.] One way of characterizing such a simultaneous convergence over all layers is to consider that all widths Nt are increasing functions of one parameter, let us say n, such that,
grows, all layers' widths increase: vl E (1.... L}. N₂- = Nz[n]]. We emphasize this dependence on 72 by appending an index X [r] to the random variables X when is finite and denote by X [[+] its limiting random variable, corresponding to n - oo. The input dimension Ng and the final output layer dimension NL+1 are finite thus they do not scale with Moreover, the input data are assumed to come from countably infinite input space X' (see A.I]). Equation can thus be rewritten, for any x E X, as
Ne-i(mi Σ
K [z)[n]]
w/y 2/ (x) + 692.
[x)( o(h (z))
(2)
and the associated Gaussian Process limit is given in Theorem I which we can now state.
Theorem (GP limit for fully connected PSEUDO-IID networks). Suppose a fully connected neural network as in equation is under the PSEUDO-IID regime with parameter ow and the activation satisfies the linear envelope property Def. Let X be a countably-infinite set of inputs. Then, for
every layer 2 < & L+ I, the sequence of random fields (i,z) E [Ne] x X + (x)(n) E R converges in distribution to a centered Gaussian Process (i, [ X whose covariance function is given by
h (x)(+) ERN,
E A (x)[+]
1 (<')++] 81, (x,,'',
(3)
where
oî 1 oī
82 2, { t= 1.
(4)
die No (I, x'),
2.2 THE PSEUDO-IID REGIME FOR CONVOLUTION NEURAL NETWORKS
We consider a CNN with C₁ number of channels layer & E (I,.... L + 1} and two-dimensional convolutional filters U.() E & mapping the input channel j E {1, Ct-13 to the output channel i E {1, C.). The input signal X (also two-dimensional) has Co channels "I, XC.
and its propagation through the network is given by
Co
(
* *Xj
l 8=1, 1,
(x)(n)
1
1=1
X [n]n]] = =oh,
((h(((())
(X))
6 1
(0)
+
(+) i.j
xz
(8-1) (x),
8 2,
(5)
In [ the symbol 1 should be understood as having the same size as the convolution output, the non-linearity ((-) is applied entry-wise, and emphasize on the simultaneous scaling with by appending [n] to the feature maps h. (F) (x) We denote spatial (multi-) indices by boldface Greek letters μr, etc., that ordered pairs of integers taking values in the range of the size of the array.
4

---

Published as a conference paper at ICLR 2024
For example, if X is RGB (Co 3) image of H D pixels, i = 2. and jr (a, then Xi, returns the green intensity of the (a. 3) pixel. Moreover, we define [|4] to be the patch centered at the pixel le covered by the filter, e.g. if μ (a, 3) and the filter covers k x k (2kg + 1) x (2k, + 1) pixels, then [ = { _ a ko & a' Σ a + ko: B ko S 3 < B + ko), with the usual convention of zero- pero-padding for the out-of-range indices. Sufficient conditions for PSEUDO-IID CNNs to converge to a Gaussian Process in the simultaneous scaling limit are given in Definition 4
Definition (PSEUDO-II for CNNS), Consider a CNN with random filters and biases {UL) and (b()) as in equation 5 is said to be the PSEUDO-IID regime with parameter o² if U()) has IID (0, ) entries and, for 2 s e < L + 1, such that for any fixed vector == (aj,)j...
(i) the convolutional kernel U E IBCoxt-1xkxk is row-exchangeable and column- exchangeable, that is its distribution is invariant under permutations of the first and second indices,
(ii) the filter' entries are centered, uncorrelated, with variance E[(U!)) (3. o² /C2--1
(iil) E Eラ En G T2 K ((ee) for some constant K,
(iv) and lit Ce-1lul" E !(U!! U U tadi iusi me mnd Siciha st Mpottta
for all j¹.
Similar the fully connected case, the filters' entries can exhibit some interdependencies as long as their cross-correlation. vanish at a fast enough rate dictated by conditions (iii)-(iv)
Theorem 2 (GP limit for CNN PSEUDO-IID networks). Suppose a CNN as in equation i under the PSEUDO-IID regime with parameter By and the activation satisfies the linear envelope property Def.
3 Let X be a caantably-infiitz set of inputs and E I denote a spatial (multi-) ) index. Then, for
every layer & < T. 1, the sequence of random fields (i, X, p) E [Ce)
x X x I + h
(X [m]
converges in distribution to centered Gaussian Process (i, X, 1) E [Ct] X x I whose covariance function is given by
(X)
E 1 [+]
(9) (X')=]] = 6..5( oo
Σ
Kt <X)
(6)
vEInInINl]
where
E ((nn)~ 2 ((xx')) [o(u))(1), (2 2,
K X, X')
Co Ni
f= 1.
These equations resemble those derived in the fully connected case, apart from the introduction of additional terms accounting for pixels within patch μ.. These terms vanish when the filter kernel size reduced 1. Our proof of Theorem can be found in Appendix B
PSEUDO-IID IN PRACTICE
In this section, we illustrate some important PSEUDO-IID distributions through examples. We further show that the Gaussian behaviour serve, among others, two purposes of practical significance (see section 3.3). First, we elucidate the analysis of signal propagation in neural networks. Secondly, this Gaussian Process limit simplifies the initially complicated task of choosing prior over large set of distributions a Bayesian Neural Network could realize at initialization.
3.1 EXAMPLES OF PSEUDO-IID DISTRIBUTIONS
We elaborate here on some typical initialization schemes and their belonging to the PSEUDO- class. To the best of our knowledge, rigorous proofs of the Gaussian Process convergence in the literature are restricted to the IID cases only (see Section I.I.. An important non-lID case that also results in
Gaussian Process is the random orthogonal initialization, partially derived in Huang et al. (021)
5

---

Published as a conference paper at ICLR 2024
for fully connected networks. Nonetheless, we regret the authors remained elusive on the treatment of the first layer, an aspect which we have since then enhanced. Our proposed PSEUDO-IID regime encompasses both IID and orthogonal cases (see Appendix D) but also allows for broader class of weight distributions such as random low-rank or structured sparse matrices, for which the Gaussian Process limit has not been established before, despite their practical significance (see Section I
Low-rank weights. Low-rank structures are widely recognized for speeding up matrix multiplica- tions and can be used to reduce memory requirements, see Whilst such structures inevitably impose dependencies between the weight matrix entries A Iu thus breaking the IID assumption, Nait Saada & Tanner (2023) introduced a low-rank framework that falls within our PSEUDO regime. Let C == [C,," C.] E IR mxx be uniformly drawn orthonormal basis for random
r-dimensional subspace. Suppose P (P.)) Rx has IID entries Ps D. Setting A == CP. the columns of are spanned by those C. The row and column exchangeability of follows immediately from that of C' and P. and the moment conditions (iii) and (iv) are controlled by the choice of distribution D. Direct computation of the four-cross product that appears in condition (iv) gives
Ai,
Σ ISRKK'S
CiCin.kCin Cia.k
where s B(PP.)) Using the expression in Lemma 3 of Huang al. (2021) we can caleulate the above expectation and deduce condition (iv) when r is linearly proportional to m.
Structured sparse weights. Block-wise pruned networks have recently been under extensive study for their efficient hardware implementation Dao et al. (2022b a). Once the sparsifying mask is fixed, we may apply random row and column permutations on the weight matrices without compromising
the accuracy the computational benefit. Let A (Atg) Rmxx have entries Au iid D and E _m be the binary block-sparse mask. Let Á = (A Pn. where Pm and Pa are random permutation matrices of size m and multiplication. Then, by construction, Á is row- and column-exchangeabbe and, for suitable choices of underlying distribution D, it satisfies the moment conditions of Definition 2 Appendix H contains some illustrations for reference.
respectively, and represents entry-wise
Orthogonal CNN filters. Unlike the fully connected case, it is not obvious how to define the orthogonality of a convolutional layer and, once defined, how to randomly generate such layers for initialization. Xiao et al. (2018) defines an orthogonal convolutional kernel U E RSut X xkxk made of Cour filters of size le through the energy preserving property || * X = |X| 2. for any signal X with Can imput channels. Waan et al. (2200)) requires the matricized version of the kernel to be orthogonal, while Qi et al. (2020) gives more stringent definition imposing isometry, i.c.
Cou
6,
j= otherwise,
U U., + U,
0
where is at (0, 0). and 0 elsewhere. Another definition in Huang et al. (202)) calls for orthogonality of "spatial" slices U, E Sost Xei,, for all positions μ.
We take a different approach than Wang et al. (2020b) for matricizing the tensor convolution operator, setting the stride to and padding to O: reshape the kernel U into a matrix E Caut x and fffeexxd
unfold the signal X into X E R** f x where d is the number of patches depending the sizes of the signal and the filter. This allows Ü to be an arbitrary unstructured matrix rather than the doubly Block-Toeplic matrix considered in Wang et al. (2020b), as shown in Figure [ Matricizing the tensor convolution operator imposes the structure on the signal X rather than the filter Ú. Orthogonal (i.e. energy peergy-prssrvving kernels Ü can then be drawn uniformly random with orthogonal columns, such that
1 R22
and then reshaped into the original tensor kernel U. Note that this construction is only possible when Ü is tall matrix with trivial null space, that is when Cout 2 k2 otherwise the transpose might be

---

Published as a conference paper at ICLR 2024
Filter
Signal
Filter
Signal
B
U
x
U
X
U X
1
A
B
D
B
Ú
X
X
Figure 1: There various ways to compute convolutions U X between tensor filter U and a 2D signal X (in the middle) from matrix multiplications. We illustrate the approach taken in Garriga-Alonso al. (2019) on the left, where the reshaping procedure is applied the filter, whilst the method we followed, shown on the right, consists of reshaping the signal instead in order to define special structures on the CNN filters such as orthogonality, sparsity and low-rank.
considered. We emphasize that equation is sufficient (and not necessary) condition for U to be energy-preserving, since X. by construction, belongs a very specific structure set T C R and, therefore, U needs preserve norm only T (so not everywhere). Therefore, we do claim the generated orthogonal convolutional kernel U is "uniformly distributed" over the set of all such kernels.
Now let check that what we defined to be orthogonal filters in the previous section verify the conditions of Definition Each filter U,g is flattened as U E lx and forms part of row of
U shown below:
Ú1,1
U1
Ü Úlet
Ú
2,1
2,
U2.cm
Cower
Coue.2
Cane.Cin.
Applying permutations on the indices and j translates permuting rows and "column blocks" of the orthogonal matrix U, which does not affect the joint distribution its entries. Hence, the kernel's dis- tribution is unaffected, and therefore U is row- and cellumn-exchangaabbee The moment conditions are both straightforward to check U.jiæ Ü Ü, 1) μ {1, k2). where is the counting
number of the pixel To check condition (iv), note that ((U(") U U U( 2ng) =
R(U,,.. Üio. Ute Ul.. that is four-cross product of the entries of an orthogonal
matrix, whose expectation is explicitly known to be (CI-1)C((C+2)) Onaiis &ic. 0pe &pec (Huagg et al. 2021 Lemma 3).
Ce+1
3.2 SIMULATIONS OF THE GAUSSIAN PROCESSES THEOREM ① FOR FULLY CONNECTED NETWORKS WITH PSEUDO-IID WEIGHTS
Theorem T establishes that random fully connected PSEUDO-IID networks converge Gaussian Processes in the infinite width limit. Here we conduct numerical simulations which validate this for modest dimensions of width N₂ n for 3. 30, and 300. Figure 2 shows histograms of the empirical distribution of the preactivations when the weights are drawn from various PSEUDO-IID distributions/ Even n 30 there is excellent agreement of the histograms with the limiting
EExpeiiments conducted for Figures 2 used a fully connected network with activation o(x) = tanh(z), weight variance aw 2 and without bias. Dropout used probability 1/2 of setting entry to zero, low-rank used rank [n/2]], and block-sparsity used randomly permuted block-diagonal matrices with block-size [n/5]. The code reproduce all these figures can be found athttps://shottue go/
7

---

Published as a conference paper at ICLR 2024
width
width 30
width 300
IID Uniform
II Gaussian
with dropout
PSEUDO-IID
Gaussian low-
rank
PSEUDO-IID
Gaussian struc-
tured sparse
Figure 2: For different instances of the PSEUDO-IID regime, in the limit, the preactivation given in the first neuron at the fifth layer tends to Gausssian whose moments are given by Theorem The experiments were conducted 10000 times on random 7-layer deep fully connected network with input data sampled from 8
variance predicted by our Theorem plots in Appendix
I These histograms in Figure 2 are further quantified with Q-Q
Figure B] illustrates the rate with which two independent inputs Fa and 24 generate feature maps corresponding to uncorrelated Gaussian Processes, when passing through a network initialized with PSEUDO-IID weights. The experiment is done by examining the joint empirical distribution
of h1 (z.))r)) and h! (t (xx) at the same neuron index and layer & through the same network. The level curves depiet the theoretical joint Gaussian distribution given by the covariance kernel in Theorem [ Interestingly, we observe the fastest convergence to the Gaussian Process in the orthogonal initialization. The other PSEUDO-IID distributions considered exhibit good agreement at n = 30 which improves at n = 300. Some more experiments for fully connected networks and CNNs initialized with orthogonal filters be found in respectively, Appendix E and
3.3 IMPLICATIONS OF THE GAUSSIAN PROCESS LIMIT
Bayesian Neural Network and Gaussian Process. As opposed to frequentist approaches, Bayesian inference aims at finding a predictive distribution that serves to quantify the model's predictions uncertainty and prevent overfitting. Starting from prior, this posterior distribution is updated once the data observed. Considering Bayesian neural networks, these priors are to be defined on the weights and biases if adopting parametric perspective, or, alternatively, directly the input-output function represented by the model. Note how one can jump from one view to the other given that random initialization of the parameters of neural network induces distribution the input-output function (2(+1)(x)) (xr) of the network. Nonetheless, finding meaningful prior over the space of functions is in general delicate task.
In the large width limit, when the weights are drawn PSEUDO-IID in L-layer deep network. Theorem I secures that the function represented by the network is exactly a Gaussian Process with known covariance kernel K (L++)) which only depends on the variances σb, aw and the activation function. Therefore, the exact posterior distribution can be computed, yielding the Neural Network Gaussian Process (NNGP) equivalence described in Novak et al. (2020). Since the covariance kernel obtained in our less restrictive setting recovers the one in Novak et al. (2020), Mattheww et al.) (2018) and Garriga-Aloma et al, (2019). their experiments still hold in our case and we refer to these works for practical considerations how compute the posterior distribution. Specifically, NNGP yields
8

---

Published as a conference paper at ICLR 2024
width 3
width 30
width 300
II Uniform
with dropout
PSEUDO-IID
Orthogonal
PSEUDO-IID
Gaussian low- rank
PSEUDO-IID
Gaussian struc-
tured sparse
Figure 3: The empirical joint distribution of the preactivations generated by two distinct inputs flowing through the network. The large width limiting distribution as defined in Theorem [ is included as level curves. The input data Ta. F3 are drawn IID from 8 and 10000 experiments were conducted on a 7-layer fully connected network. The horizontal and vertical axes each subplot are respectively h (Ix)) and h (5) (xs).
better accuracy compared to finite point-estimate neural networks [ee et al. (2017) and approximates finite Bayesian networks very well (Matthews al. 2018 Section 5).
Edge of Chaos (EoC). Random initialization of deep neural networks plays an important role in their trainability by regulating how quantities like the variance of preactivations and the pairwise correlation between input signals propagate through layers. An inappropriate initialization causes the network to collapse to a constant function or become chaotic mapping oversensitive input perturbations. The Edge of Chaos (EoC) initialization strategy rectifies these issues, additionally causing the network have gradients consistent magnitude across layers. Calculation of the FoC requires integration with respect to the distribution of preactivations in intermediary layers, which, in general, is intractable. However, the Gaussian Process limit simplifies the EoC analysis carried out in Poole et al. (2016) and Xiao al. (2018). Moreover, under the same distributional assumption, [Schoenholz et al. (201)) Penningtto et al. (018)) show that initialization on the EoC achieves dynamical isometry, making backpropagation of errors stable. Our main contributions [ and 2 allow similar calculations to be made for PSEUDO-IID networks such as the
of Theorems examples of low-rank, structured sparse, and orthogonal CNN shown in Section 3.1 The BoC for low-rank networks has been calculated in Nait Saada & Tanner (2023) under the assumption of Theorems
at that time unproven. Interestingly, structured sparse or low-rank initializations are equivalent to their dense and full-rank counterparts up rescaling of the variance by a fractional factor of sparsity rank.
T and 2 which
terms of signal propagation,
CONCLUSION
We proved in this paper new Gaussian Process limit for deep neural networks initialized with possibly inter-dependent entries. Examples include orthogonal, low-rank and structured sparse networks which are particularly of interest due to their efficient implementation and their empirically observed enhanced accuracy. Our result makes possible exact Bayesian inference as well as tractable Edge of Chaos analysis, for broader class of either fully connected convolutional networks. We expect the present work paves the way for a better understanding of the training dynamics of the emerging deep neural networks with parsimonious representations.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Juba Nait Saada for insightful discussion and feedback on the manuscript, as well as Alex Cliffe for valuable comments and help in drawing Figure 1. Thiziri Nait Saada is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) through the grant EP/W523781/1, Jared Tanner supported by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA).
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Leonid Berlyand, Etienne Sandier, Yitzchak Shmalo, and Lei Zhang. Enhancing accuracy in deep learning using random matrix theory, 2023.
Patrick Billingsley. Convergence of probability measures; 2nd ed. Wiley series in probability and statistics. Wiley, Hoboken, NJ, 1999. URL http:: http////d.//e/ cern wttp://cc.e.ccecec/////2/222222
Julius R. Blum and Murray Rosenblatt. A class of stationary processes and a central limit theorem. Proceedings of the National Academy of Sciences of the United States of America, 42 7:412-3, 1956.
Tianlong Chen, Xuxi Chen, Xiaolong Ma. Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets, 2022.
Benoit Collins, Sho Matsumoto, and Jonathan Novak. The weingarten calculus, 2021.
H. Cramér and H. Wold. Some theorems on distribution functions. Journal of the Lon- don Mathematical Society, 299- 1936. doi: wtt.://wwww.....o//....ooo. cww/doi hatmm//oi/bb////// /abs/10
4.290. URL https: www.ooothhheseccc inw.eeeeooiineeiibrrrrrr
ley.
1112/ mms 11 4.290
Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang. Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models, 2022a.
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training, 2022b.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks shallow gaussian processes, 2019.
X.Y. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics the central path. In International Conference Learning Representations, 2022. URL httphtttss///opnneeeeeee ew. net fot///orum? id=w UbdvWH_R R3
Boris Hanin. Random neural networks in the infinite width limit gaussian processes, 2021.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1): 10822-88811100 11005, 2021.
Wei Huang, Weitao Du and Richard Yi Da Xu. On the neural tangent kernel of deep networks with orthogonal initialization, 2021.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1022- 1035. Curran Associates, Inc 2021. URL hwtphttcpp./prroeeli..ggo http://wweeeaaioe.eeereeeeeeeeee pdf fi Filesh Teess//a bass/paper/202111 wttpl//flll//// 08 weee8Bbbeee /tte///0ee0aaaaoo0o0000000030000
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz. Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes, 2017. URL
oxc/abs/1711.0015
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity, 2019.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent transfer, 2020.
11

---

Published as a conference paper at ICLR 2024
Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks, 2018. URL [tllattpp://ariiv.oom
abs A.ps/1804.10 11271
Thiziri Nait Saada and Jared Tanner. On the initialisation of wide low-rank feedforward neural networks, 2023. BtLhttps: /ttp:///axvvvvvoiii httgg//b//// /ttp:///31..311111
R. M. Neal. Bayesian Learning for Neural Networks, Vol. 118 of Lecture Notes in Statistics. Springer- Verlag, 1996.
Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl- Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes, 2020.
Kazuki Osawa, Akira Sekiya, Hiroki Naganuma, and Rio Yokota. Accelerating matrix multiplication in deep learning by using low-rank approximation. In 2017 International Conference on High Performance Computing & Simulation (HPCS), pp. 186-192, 2017. doi: 10. 10.11099/PCS
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 1924-1932. PMLR, 2018.
Ben Poole,
Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos, 2016. URL battps://arxil.
org/abs 6 05340
Ilan Price and Jared Tanner. Improved projection learning for lower dimensional feature maps. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. International conference on machine learning, pp. 7824-7833 7835. PMLR, 2020.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017. URL https
//openreview new. fttts/ofrrmm1ddHii UNgggg
Yu-Shan Tai, Chieh-Fang Teng, Cheng- Yang Chang, and An- Andy Wu. Compression-aware projection with greedy dimension reduction for convolutional neural network activations. In ICASSP 2022 2022 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 66-60,20 2022. doi: 10.1 (BP/ICCAASP449222222..776666
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, 2020.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, 2020a.
Jiayun Wang. Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11505-11515, 2020b.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla con- volutional neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 5393-5402. PMLR, 10-15 Jul 2018. URL htttt::::ppoceeedeitite press/ 1//00/xiaaa hatas//80xxiia 8aa. html
Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architecture gaussian processes, 2021.
12

---

Published as a conference paper at ICLR 2024
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, June 2020a.
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification, 2020b.
Chaoyang Zhu, Kejie Huang. Shuyuan Yang, Ziqi Zhu, Hejia Zhang, and Haibin Shen. An efficient hardware accelerator for structured sparse convolutional neural networks on fpgas, 2020.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM GAUSSIAN PROCESS BEHAVIOUR IN FULLY CONNECTED NETWORKS IN THE PSEUDO-IID REGIME
A.1 STEP 1: REDUCTION OF THE. PROBLEM FROM COUNTABLY-INFIITE TO FINITE DIMENSIONAL.
Firstl, we must clarify in what sense a sequence of stochastic Processes h; [n]n i E N converges in distribution its limit h4 For sequence of real-valued random variables, we can define convergence in distribution _ X by the following condition: (x(x,, - (x for all continuous functions f R RR Similarly, we can define weak convergence for random objects taking values in RR (countably-indexed stochastic Processes), provided that we equip I& with "good" topology E that maintains sufficiently rich class of continuous functions. The metric p on the space of real sequences @ defined by
p(h, h2) = Σ 2 min 1h, 1(1)
(10)
one example. Thus, we can speak of the weak convergence h;(1) h,[n]__, [,;(]] [h, + [x]] for all f R + R continuous with respect to the metric p.
[]] in the sense that
Fortunately, to prove the weak convergence of infinite-dimensional distributions, it is sufficient to show the convergence of their finie--imensional marginals Billingsley (1999).
A.2 STEP 2: REDUCTION OF THE PROBLEM FROM MULTIDIMENSIONAL TO ONE-DIMENSIONAL
Let C ((),, the vector (h!2) (xp) (nl) E RR" converges in distribution to (=p)[+)) (x,))()) e R.. By the Cramér-Woi
(ip,ær)) be a finite subset of the index [Ne]] X. We need to show that
theorem (Cramér & Wold (936)), we may equivalently show the weak convergence of an arbitrary
linear projection
Σ
(1.0 (x) (/0)
(11)
(1
Ne-1(nl
Σ
Σ a(r,))
W 2/ (x)")(
(12)
(NZ)ER
Ne-1(n)
(2) (a, [m).
(13)
VNe-ilm
where
y (a, (),mm = ow Σ ((t (0 (0)((11)
(14)
(LxEEC
(8) and c. is centered and normalized, i.c. Ee = 0, (t) 1. We will be using suitable version
of Central Limit Theorem (CLT) to prove the weak convergence of the series in equation 13 to Gaussian random variable.
A.3 STEP 3: USE OF AN EXCHANGEABLE CENTRAL LIMIT THEOREM
The classical Central Limit Theorem (CLT) establishes that for a sequence of IID random variables. the properly scaled sample mean converges to Gaussian random variable in distribution. Here we recall an extension of the Central Limit Theorem introduced Blum Rosenblatt (1956), where the independence assumption on the summands is relaxed and replaced by an exchangeability condition. The following statement is an adapted version derived in Matthews et al, (2018), which is more suited to our case.
In fact, needs be Polish space, i.e. complete separable metric space.
14

---

Published as a conference paper at ICLR 2024
Theorem 3 Matthews al. (2018) Lemma 10). For each positive integer n, let (Xn.it N be an infinttll exchangeable Process with mean zero, finite variance o and finite absolute third moment, Suppose also that the variance has a limit Tim o o². Define
Ni
1
S
Xn.j
(15)
where N N is a strictly increasing function. If
(a) E I = 0,
(b) lim 1 o
(c) E 1 on (VN())).
then Sn converges in distribution to N(0,00)
Comparing equation 13 with equation 15 we need to check if the summands Xn. == -0 [m] satisfy the conditions of Theorem Wewil carefully verify each condition in the following sections. At each step. make note in italics whenever loosening the assumption the first layer's 's weights from being IID Gaussian to row-IID introduces a slight variation in the proof.
A.3.1 EXCHANGEABILITY THE SUMMANDS
To apply Theorem we must first prove that the random variables 7/ Let us expand the expression for 7 (a,, [n] further and write it in terms of the preactivations of layer - 2, i.e.
(99 (a, [n] are exchangeable.
(a, ow Σ
(3 (h)
(8) (() (2)[nl)
(L,))L
Ni-zln]
=ow Σ
(0) Φ
1) 20 bb)
(1,))C
we apply a random permutation on the index j. let us say o (1....,,Ne-1 + (1....,,N. then the exchanggeeaabil and ballum-exccangeabbee fj and that (t-1) has same distribution as (f-11) for any w-(0) has same distribution as w(( (z(l))
of the PSEUDO-IID weights ensure that
jk
i E (1,...,,N)), & E (I,..., N Note that this extends easily to the normalized versions of the weights 2(0) Additionally, the biases are set to be IID Gaussians, they are a fortiori exchangeable and their distributions remain unchanged when considering random permutations of (E) indices. Therefore, ?i (a, [n] equal (2)
Ta() (a, [m] in distribution, hence exchangeability.
For = 2, when the weights w:( jk are row-lID only, they are also row-exchangeable and the exchangeability of 3
3 (a, (n) follows.
A.3.2 MOMENT CONDITIONS
We mean by moment conditions the existence of a limiting variance of, well the conditions (a)-(c) in Theorem We will prove the moment conditions by induction on the layer number L.
Existence of the limiting variance of the summands. To show the existence of the limiting variance the summands, let us first write down such a variance at a finite width. Since y exchangeable (see A.3.T), their distribution is identical and may simply calculate the variance of
(0) are
15

---

Published as a conference paper at ICLR 2024
(6) 71 follows.
(ol0))(a, = =B( [ml)) = =klow Σ
10,
(1-1) (x)(m))
(i,x)EC
ow
(iaita (
(a,l 96.1
(1)
(e-1) (x (xa) 21 (xs)(()))
(ka aa JEC (ELL
ow
(((iai ((s S (
(1-1)(2a)(1))) 21 (za)lll) (16)
where we first considered the independence between the normalized weights at layer & and the activations at layer I; then used the fact that the normalized weights are uncorrelated in PSEUDO-IID HID regime.
The convergence of the second moment of the summands is thus dictated by the convergence of the covariance of the activations of the last layer. By the induction hypothesis, the feature maps 1 (((11) [x)[ converge distribution, so the continuous mapping theorem guarantees the existence
of a limiting distribution for the activations 2| 2(f-1) (x.a)rr [m]] and =1 2((-1) (xx) [n] as n tends to infinity. Note that this result holds even if the activation function has a set of discontinuity points of Lebesgue measure zero, e.g. the step function. Thus, the product inside the expectation equation [6 converges
in distribution to limiting random variable žj (8-1) (x.) [+]] (f-11 (xr)) (+).
From Billingsley (1999). one knows that a sequence weakly converges to limiting distribution and is uniformly integrable, then we can swap the order of taking the limit and the expectation. Thus, by Proposition we have
(), = lim (o (0))) [m)]l]
= oo Σ
((te))) [(2 ((-1)) 21 (ma))x] (17)
For 2, the activation 21 (xx) does not scale with n, iee. 21 (x)) = 20 (m6)(+) thus swapping expectation and limit trivial.
Condition (a). At given layer f, we need to show that Xn,1 = r1 ((a, and Xn.2 = 1/ (a, [n] are uncorrelated. We have
E(X,, X X,,2) = E (aw
Σ
(e-1)))1 (1)1))))))))) (ow Σ (9 (x)
(0)))))
(K,)eC
((,))EC
2
Σ
f0, 20,2 E (8-1)
(e) (0)
(f=111)) 2/2 (c))(_)
ow
(Ha JEL (BL
since 2 and
(8)
(
uncorrelated by the PSEUDO-IID assumption, for é 2.
2
16

---

Published as a conference paper at ICLR 2024
Condition (b).
x n,
x E
(ow
E
((ia))
(0) (ow E 2
1
(aa)e
(i,r)ER
ow Σ
((t.))))
(GER)EC tEfa,b,dd)
(E(a,bedd)
E
1,,11 (1))((-1) (L- (1))())))0))(m))
(
II
1 1
LE (a,b)
tE(c,))
= Ow
E
II LE(a,b,c,d)
((h))) Fn(la: ib
(E. tefa,b,c,d)
(18)
IE
$1
(t-1) (x)((i)
II (e,d)
2/ (o.))))
(a.b)
where
F2 (1a, lo, les id) E Ele
(0) i. 2..2
(19)
We justify the convergence in distribution of the random variable inside the expectation in the exact same way as in the previous section, referring to the continuous mapping theorem and the induction hypothesis. By Proposition [1 as n oo, the above expectation converges the expectation of the limiting Gaussian Process, i.c.
lim E -0c
II IE(a.))
of ((-1) (xa)(rl)
(
II sE(c,d)
2/ 16"" (x))lii)
= =E
II AE (a.b)
=1 (x))())
(I II tE(c,))
>/t (m)(()
Moreover, condition (iv) of Definition 2 implies that
lim (aalas db des id) = Si Sinzia- 70000
Substituting the two limits back in the equation 18 and using the independence of the activations layer - 1 given by the induction hypothesis, we get
lim B. x X =0000
=o Σ
II rEla.dic,))
((t))))) Sio
(EC tE(a,b.c,d)
E
II (E(a.))
21
(6- ((-1)) (wx))el)
III
2f (2r)
8 (c.dd)
=o 1
ow
Σ
II a(tizt))
Sin E
2( (e-1) (20))+)
(EL, te (a,b)
tE(a,b)
LE(a,b)
Σ
II
((())))
II
H (x))++)
(L.Z))EC tE(c,))
(E(c,))
(E(c,d)
= (o
For (o 2, the activations 21 (1) (xx) do not scale with i.e. 2 (1) = 20 (xx)(+), thus swapping expectation and limit is trivial once again. The independence between the activations at different neurons boils down the independence between the rows of (
17

---

Published as a conference paper at ICLR 2024
Condition (c). To show that the third absolute moment of the 71 (a, [x] grows slower than / VNe-1(m) n as n - oc, is sufficient to bound it by constant. Applying Hölder's inequality on (8) X ni (a, [n]] y== = 4/3.9 4, obtain
(a, (a, [m]
[m] x1
Therefore, condition (c) boils down to showing that lim E E (a, [n]] is finite, for which we will once again make use of the uniform integrability of the feature maps derived in C Define
(E)
Gn (Ra ib, des ia) = ((c Co.
(20)
and observe that E h/1 19 (a, [n]] =
=)m]|]===((00 =E((ow
Σ
(0) (8-1) (1-111111111011111111) (z) I ((1x) (1
(Fxx)E
WO ow
E
((ts,a)) Gn(bas les
II
=1 (t- 2f- =
rEfa,b,cd))
Le (a.b,c.d)))
Using Cauchy-Schwarzzinequue we can bound G,, by the fourth moment of the normalized weights, i...
Gn V Var(el() 1c 1..1 1 (e)())) ba.1) Var(e (0) (s.1) = = (0 1..1 S Var(e f..1) =
B(( to.1l -1
Then, we use condition (iii) of the Definition inequality for p = bound the fourth moment:
2 for a = (1,0, ,0) combined with Hölder's
n²
E
[[w4 1..1]
1..1
0
n Ki
K, 04 (1).
Killall
-2
Furthermore, the induction hypothesis gives the convergence in distribution of the feature maps from the last layer, and combined with the continuous mapping theorem we get the convergence in distribution of the above product inside expectation. Using Lemma] [] the uniform integrability of the activations follows, and Billingsley's 's theorem (Lemma enables us to swap the limit and the expectation. Thus,
lim 2 100
II
=1 (xe)())
E
of (e-1) (c))(+))
(E(a,b.c,d)
(a,
(1) For = 2, the activations 2 (xx) do not scale with n, iee. (2) (2x) == 2 (xx) thus swapping expectation and limit is trivial once again.
(1)
To bound the product of four different random variables on I, is sufficient to bound the fourth order moment of each (see Lemma B). We can do using the linear envelope property (Definition satisfied by the activation function to get, for any (i, x) E L.
E 2
(x)
< 2 M (.)
The induction hypothesis indicates that 1 moment is bounded. Using the fact that we chose the set L. be finite, we can take the supremum over all æ. Therefore, we have
(x)(+) follows Gaussian distribution, whose fourth (t- (x))
(8- 1) (xx)
2-1 sup E E (rrln(-""(())) on (1).
E
(ix))L
18

---

Published as a conference paper at ICLR 2024
Combining the above bounds, then have
lim 7 [)[m]] <c. 10000
For i = 2, the preactivation (x) Ee= Wila bi" have finite fourth moment provided
that W does, which we assume in the row-IID case.
A.3.3 CONCLUSION FROM THE EXCHANGEABLE CLT
In the above sections, we showed by induction that at any depth, if the feature maps from previous layers converge in distribution to IID Gaussian Processes, then the assumptions of Theorem 3hold and the one-dimensional projection of the feature maps at the current layer also converges distribution to Gaussian random variable with specified variance. More precisely, finite set L and projection vector any linear one-dimensional projection of the feature maps at the current layer. ( (a, [n], converges in distribution to a Gaussian N (0, (a) [+]) as n grows. This gives the convergence of the feature maps at layer to Gaussian Processes.
showed that for any
Considering the unbiased quantity
(a,
E ((.))) (x)++) b(9), (R.)ER
we can compute its variance:
E (a,
Σ (e. (o.F))E
al.. (()
E 11 (12(((2()))) (xa)[+] -n(]
n/2 (20)(t] of
As we saw in the previous sections,
(a) ow
Σ (L. (6..2)EC
(ilanna) (lsr))))) B(1 (e2 (xa)(x) (1 (m))
Thus, by identification, and using the inductive hypothesis, recovers the recursion formula for the variance as described in Theorem
[
For any da, in E N, Fa, Th E EX
E [ 1 (za) (+0)(+] diavio (o 2
(e-1) (x.) 21 (e-1) (o))(*] og [á(u)((v)] + oō)
δi (oir (un))
For f. = 2, we denote by DD the distribution of the weights the first layer, obtain instead (xx) (xx)) + +o?)
E h (1 (mw)lt]] Eiavio (ok En (x) (xx)z
(21)
A.4 IDENTICAL DISTRIBUTION AND INDEPENDENCE OVER NEURONS
As saw, for any the feature maps h (a, [m)] are exchangeable, and, in particular, identically distributed. This still holds after taking the limit, that is n [+] and (22 have the same distribution for any j. k E N.
still remains to show the independence between ([((+t) and h [x] for As we now know the limiting distribution is Gaussian, it suffices to analyze their covariance to conclude about their inde-
pendence. As derived in the previous section, for any x, a' E X. i + j. ( (z)[ (<')(+] = hence the independence.
19

---

Published as a conference paper at ICLR 2024
B PROOF OF THEOREM 2 GAUSSIAN PROCESS BEHAVIOUR IN CONVOLUTIONAL NEURAL NETWORKS IN THE PSEUDO-IID REGIME
We apply the same machinery to show the Gaussian Process behaviour in CNNs under the PSEUDO- IID regime, closely following the steps detailed in the fully connected case. To reduce the problem to a simpler one, one can proceed as previously by considering finite subset of the feature maps at layer E, [ {(i1, X1, (1), ... (ip, Xp, μP)) S [C2] X' x I, where I consists of all the spatial multi-indices. We will follow the same strategy outlined Sections Scctions|A.I A.1 Given finite set L
and the projection vector a [RC,,, we may form the unbiased one-dimensional projection as
(a,
Σ
((i.X.1))
[x] - b.
(22)
(L.
which can be rewritten, using equation the sum
Cz-1 [x]
(a, VCe-1(1) [r]]
Y (a, (a, [m]]
16
(23)
j=l
where the summands are
7/ [o, = ow
Σ ((....))
Σ
E 2 (X)
(24)
(i.X.))E
VELN
As before, we introduced the renormalized version E of the filter U such that E(() Pij.
We will proceed once again by induction forward through the network's layer verifying the assump- tions of the exchangeable CLT (Theorem 3) and using on (*) (a, [n] to conclude its convergence in distribution to Gaussian random variable.
The exchangeability the summands. Similar to the fully connected case, we employ the row- and column- columm-exchangeaablity of the PSEUDO-IID convolutional kernel to show the exchangeability of o (f)ss Let us expand equation 24 and write
(a, [m] = ow Σ (L.X
Σ
E Li
(1-1) j.v
(x)
METuI
Cenenn
ow Σ
(e-1) (t-2) j.k KK.E
(X)(m)
(E
vElu]
SElPl
The joint distributions of (u!9 Cr-1 and (U
Cz-1 j=1
(3
are invariant under any permutation +
a(),, therefore (0's exchangeable.
Moment conditions.
Condition (a) is straightforward the filters' entries are uncorrelated by the <ix
PSEUDO-IID assumption: (( (t) U()
t. P ] -. 8e
The moment conditions are shown to be satisfied by induction through the network and the proofs boil down to showing the uniform integrability of the activation vectors to be able to swap limit and expectation. This uniform integrability in the CNN case under PSEUDO-IID weights is rigorously demonstrated Proposition 2 One can compute the variance of one representative of the summands, let us say the first one, follows:
20

---

Published as a conference paper at ICLR 2024
(a = E(1 19 (a, [)(])) E
Blow Σ a(.X..))
Σ
E(0) ((-1)
1 (x)
(i.X.
MEIN
ow
Σ
ai Xu (3..
(1 Ko) JEC No
Σ
1.
E.
E
(ff-11) (f (x,][m]]
M. E [Ma]] U& [16]
Ow
Σ
(aiaaaaltll Xo.
(1. No.XX.Na)EE Mo)E2
E
B
B
E
B
(e-1) 2
(Xa)la) =//"""" [m]
Ma
V.ElNa] UyE [16b]
E
OW
(a. (1h.
(1a M. JC (a)E
Σ
8i Ou
(t-1) (Xa][n) =1,00 (X.)(()))
E [A..
[A6]
Given the uniform integrability of the product the activations in CNN, we may swap the order of taking the limit and the expectation to have
(0 (a, L)[+] = lim (a (a, [) n-000
=ow
Σ
((th
Σ [LE[Ma]
8eg Den E
( (Xa)(+) =1--") (x.)(+))
As in the fully connected case, we conclude that the feature maps at the next layer converge to Gaussians Processes, whose covariance function given by
E (e) (xa) 110 (Xb][t] = 6i (0) + o
Σ
k((0) (Xar Xb))
we [etn]Imm)
where
K10(Xa (xa
(1. (x. Co Lijil (Xa)i,(Xa))) (Xb)a
82 2 f=
C LEMMAS USED THE PROOF OF THEOREMS I 2
We will present in this section the lemmas used the derivation of our results. The proofs are omitted in cases where they can easily be found in the literature.
Lemma 1 (Hölder's inequality). For probability space (Q,F,)) let X,Y Y be two random variables on and p. q > 1 such that p + q Then,
X
(E|XP) (
21

---

Published as a conference paper at ICLR 2024
Lemma 2 (Bellingsley's 's theorem). Let X(n) be sequence of random variables, for E N and X another random variable. If [ is uniformly integrable and the sequence [(n) converges in distribution x{+) = X, then X is integrable and one can swap the limit and the expectation, i.e.
lim [x] = EX[+].
n-000
We adapt Lemma 18 from Matthews al. 2018) from the proof derived in the cited article.
setting and the proof can trivially be obtained
Lemma (Sufficient condition to uniformly bound the expectation of four- crors product). Let X1. Xg, X3, and X1 be random variables on R with the usual Borel o-algebra. Assume that [| 4 < oo for all E (1, 2, 3, 4). Then, for any choice of Pr (0, 1, 2) (where (1, 4)), the expectations [T [X, are uniformly bounded by a polynomial the eighth moments X, 8c.
This Lemma can be easily derived from standard Pearson correlation bounds.
We now have the tools to show that all four-cross products of the activations are uniformly integrable in the PSEUDO-IID setting, which enables us to swap the limit and the expectation in the previous sections of the appendix.
Proposition 1 (Uniform integrability in the PSEUDO-IID regime - fully connected networks). Con- sider fully connected neural network in the PSEUDO-IID regime. Consider the random activations (za)))) 2) (xb)))) 2 (xx) (0" (za)[n] with any j,j,k, k, b,l E N, zas Tb: Tes Id X, neither necessarily distinct, as in equation 2 Then, the family of random variables
(2) (xrw) (m)))( (2) (x=)[m) (xa)(n)
(xs)[m)=
indexed by n is uniformly integrable for any l = 1.... I. L
Proof. The proof is adapted from Matthews et al. (2018) to the PSEUDO-IID regime in fully connected neural networks described in equation
a collection of random variables uniformly L L'-boudded for p > 1, then it is uniformly integrable. So, will show that our family of random variables is uniformly ++ bounded for some >> ie. there exists K < oo independent of n such that,
E
( (Ia)
(0) (ax) [n]= (m]=) (xa)(m) (m)))
S K.
which is equivalent
E
| (9) (xx)
(x [1
(2a)
< K.
To do so, Lemma gives us a sufficient condition: bounding the moment of order 4(1 + e) of each term in the product by constant independent of n. For any I; E and i N, this moment can be rewritten in terms of the feature maps using the linear envelope property 3 and the convexity of the
map x + x 4(1+e) as
| (0
[m] [[+c]]
(B(L+))-B[(AAU + M41++)) (z. [m]])]]]0+
E
Thus it is sufficient to show that the absolute feature maps | ( [md] have finite moment of order 4 + independent of IG, i, and n, for some e. For the sake of simplicity, we will show this by induction on l for 1.
Base case. For = 1, the feature maps 1:" = En w.y x, + 6." are identically distributed for all i E {1..... N.. [n]} from the row-exchangean wwweexchangcability of the weights. Morcover, from the moment condition of Definition there exists p 8 such that
No
E|A (x)
E
+ +EP
K The RHS of the above equality independent of Ni (and n). Therefore, for all n E N and i , we have found constant bound for the feature map's moment of order 8.
F
22

---

Published as a conference paper at ICLR 2024
Inductive step. Let assume that for any (21)i=1 and E N, the eigth-order moment of [nf-"" (x)] is bounded by a constant independent from n.
We will show that this implies
E | ( (m=)(m]] <0. oo.
Considering the vector of activations ( (1-5-110p (n)) (a:))() we have, from the moment condition (iii) of the PSEUDO-IID regime the following conditional expectation,
Ne-1 [n]
E
(h (x:))
I
Kllō(ho
(x))) 1
(n]]
Thus using the recursion formulae for the data propagation in such architecture given by equation 2 the convexity of + R R+ and taking conditional expectations,
Ne- [m
E h 99
S 2-
W o(1
1)
(xx)
[m]]
(25)
No [m]
= 22
|
+ E E
wl (nh (L-1) [m]]
[l]
2 E
(0 8
+ KN
[m]
E (1
( (xx) (xx)) [m11
(26)
As the biases are IID Gaussians, the first expectation involving the bias is trivially finite and does not depend on i as they are identically distributed. For the second expectation, we compute,
[n]
(l (6- (1)
(x:)
(m]]
E
(0(h)e-) bm)))
(27)
N [m]
(c+
$
N im]
+ (1 2cM (x)))
This last expression can written as a linear combination of N quantities of the form (f- (xx) [m) (h (f-1) (f0
E (n (x:)) [m] (c- (h (ax)) "2 [m]
(c:)) [m]]
and we bound each of them making use of Lemma 3 The factor [n] in equation 26 thus cancels out with the number of terms the sum equation 27 As the preactivations are exchangeable, they are identically distributed the dependence on the neuron index can be ignored, and taking the supremum over the finite set of input data {r²} does not affect the uniformity the bound; which concludes the proof.
Proposition (Uniform integrability in the PSEUDO-IID regime- CNN.). Consider a convolu-
tional neural network in the PSEUDO-IID regime. Consider collection of random variables
23

---

Published as a conference paper at ICLR 2024
(da)(l)) 23 (m)) (0) (s)) 2i (ma))n) with any j. E,,,NN,, ther necessarily distinct, obtained by the recursion equation Then, the family of random variables
#k
N, Xs, X.. Xa E X, nei-
(2) (m)) (m)) (0) (X.)(m)= (xa)
indexed by n. is uniformly integrable for any e {1, .. L +1.
Proof. As previously, this proposition holds some novelty of this paper, extending already known proofs in the standard Gaussian ITD setting to the PSEUDO-IID regime in CNNs. Note that this directly implies the universality of the Gaussian Process behaviour for CNNs in the IID regime, which has been established so far only by Yang 202T) to the best of our knowledge. Our result goes beyond. We recall that the data propagation is described by Equation equation 5
Once again, we observe it is sufficient to show such that the moment of order 8 of the feature maps is uniformly bounded. We do this again by induction.
Base case. Since independent, for some p 8 we can write
[n]
61") + ES= EvElul Uli and weights and biases
Co
+ 1
E
Fj
ME
[1]]
K
X
|س]
11 2/2 +
where X [e] is the part of signal around the pixel μ and we have used the condition (iii) of Definition in the last line. Observe that the RHS independent
and n.
Inductive step. Let us assume that for any {X1)==1 μ e I and N, there exists co (0, such that the eighth moment of the preactivations from the previous layer | (X.)[]] is bounded by a constant independent from j E N, v E I, and n, for all X, E X
Mirroring our proof in the fully connected case, will show that this propagates the next layer,
[x)] < <0.
From the third condition of the PSEUDO-IID regime, we can compute the expectation conditioned on (4-1)
the vector of activations (h o.o [n).
Ce-
C [n]
E
U
o (x)))
|] =
U
(e-1)
(x)
E [P]
(1 (x,)) [m]
= [C+,((m]
(1 (x.)) [r|]
where the norm of the activations be computed using the linear envelope property,
Ce
[n]
(l E
(X.)
Σ (0(h(e-)) (X.) (6(h(t"))
(X.))
PEIN
[n]
Σ
2 1=1 (x.+ 2cM
1=1
VEINJ
24

---

Published as a conference paper at ICLR 2024
This last quantity turns out to be the weighted sum of (A Ce-1(nl)* terms (recall & being the filter size, which is finite) the form
h (e- (X, )P [n]h 12.1 (X, (X,) E
(f-
(x)]) (X, (n] (X, (X,)P*[n]],
that can bounded using Lemma combined with our inductive hypothesis. Observe how the factors [n]1(n]* cancel out and lead bound independent from
Using the recursion formulae for the data propagation in the CNN architecture recalled in equation and the convexity of the map + x8 on R we have
Cz-1ln) Σ
E
[n.) 2 10
(1 (h (X.))
mn]
[E]
The first expectation finite and uniformly bounded as the biases are IID Gaussians in the PSEUDO- IID regime and we have just shown above the boundedness of the second expectation. Therefore,
E| (x < 00,
(28)
and taking the supremum over the finite input data set does not change the uniformly bounded property, which was needed be shown.
D IID AND ORTHOGONAL WEIGHTS ARE PSEUDO-IID
IID weights. I (Atz) E R has IID entries Aiy iid D, then it is automatically row- and column-exchangeable and the entries are uncorrelated. Therefore, as long as the distribution of the weights 'D satisfies the moment conditions of Definition 2 the network is the PSEUDO-IID regime. A sufficient condition satisfy (iii) is for Ay to be sub-Gaussian with parameter 1/2)). Then, given the independence of entries, the random variable Σ}= Ay-Asy would be nab-Gausiian with parameter ||a|||n~1/ 1/2 and its th moment is known to be O[I a |/n-:/);; see Vershynin 2018). Appropriately scaled Gaussian and uniform IID weights, for example, meet the sub- GausSiastyy criterion and therefore fall within the PSEUDO-IID class. Condition (iv) is trivial in this case.
Orthogonal weights. Let A= (AA) E Fuxx be drawn from the uniform (Haar) measure on the group of orthogonal matrices O(n). While the entries are not independent, they are uncorrelated, and the rows and columns are exchangeable. To verify condition (iii), we may employ the concentration of the Lipschitz functions on the sphere, since one individual row of A, let say the 8th one (A Ai,)), drawn uniformly from g Let f(Ar,,,... Ai, = Σラ AgA then f is Lipschitz with constant ||a||2. By Theorem 5.1.4 of Vershynin 2018). the random variable (A..1, Ai..) = Σn Ej=1 Aij is sub-Gaussian with parameter | I| = ||a||2n 1/2 which also implies E Σラ a,A,l" 12 O[Iaa|En -p/2). Finally, the exact expressions of the th moments known for orthogonal matrices (see Collins al. (202T) for introduction to the calculus of Weingarten functions) and satisfy condition (iv). Note how we simplify the analysis here by considering square weight matrices from the second layer and onward.
E FURTHER NUMERICAL SIMULATIONS VALIDATING THEOREM I
Fig. shows Q Q plots for the histograms in Fig. compared their infinite with Gaussian limit. These Q - Q plots show how the PSEUDO-IID 1ID networks approach the Gaussian Process at somewhat different rates, with IID uniform approaching the fastest.
25

---

Published as a conference paper at ICLR 2024
width 3
width 30
width 300
II Uniform
IID Gaussian
with dropout
PSEUDO-IID
Gaussian low- rank
PSEUDO-IID
Gaussian struc- tured sparse
Figure Q-Q plots of the preactivations values Fig. as an alternative way of showing the convergence of the preactivation of fully connected network to a Gaussian as fully characterized in Theorem The settings of the experiment are the same as those Fig.
Fig. 5 eexplores the growing independence of entries in h [n) for different by showing their joint distributions for two distinct choices of i; moreover, convergence the limiting isotropic Gaussian distribution (h, (z), h (x)) i # j. is overlayed the same plots. Uniform IID converges the quickest, while PSEUDÓ-IID Gaussian low-rank and structured sparse converge towards an isotropic distribution somewhat slower, albeit already showing good agreement at n 30. The horizontal and
vertical axes in each subplot of Fig. are h (5) (x) for 1 and 2 respectively.
F NUMERICAL SIMULATIONS VALIDATING THEOREM 2 FOR ORTHOGONAL CNN FILTERS
Fig. explores the growing independence of entries in ( (X)[n] for different by showing their joint distributions for two distinct choices of i; moreover, convergence to the limiting isotropic
Gaussian distribution (he [+) (9 [+]), + j. is overlayed in the same plots.
Fig. explores the asympiotic gaussian nature of entries in (9 (X) [n] at the same channel index i by showing their joint distributions for two distinct input data. For the same reasons advocated by
Garriga-Alonso al. (2019); Novak et al. (2020), the computation of the covariance kernel comes a high computational cost. To overcome this, we decided to focus on showing the gaussian nature of the preactivations in CNNs initialized with orthogonal filters algorithm to display the level curves the hest gaussian fit the point clouds. This gaussian turns out to have non zero correlation, in agreement with 2
defined them in 3.1 To do
so, we used the Aopeecation-Maxiiiaaae
Moreover, convergence to the limiting Gaussian distribution (h [*], (x) (X) (X)[+)) i + j, seems
to be slower than in the fully connected case, if comparing this figure with Fig.
26

---

Published as a conference paper at ICLR 2024
width = 3
width 30
width 300
Uniform with dropout
Non PSEUDO- IID Cauchy
PSEUDO-IID Gaussian low- rank
PSEUDO-IID
Gaussian struc- tured sparse
Figure 5: For fully connected networks in the PSEUDO-IID regime, it is shown in Theorem that in the large width limit, at any layer, two neurons fed with the same input data become independent. We compare the joint distribution of the preactivations given in the first and second neurons at the fifth layer with an isotropic Gaussian probability density function. Initializing the weight matrices with ITD Cauchy realisations falls outside of our defined framework, resulting poor match. The inputs were sampled from S and 10000 experiments conducted on 7-layer network.
27

---

Published as a conference paper at ICLR 2024
3 channels
30 channels
300 channels
PSEUDO-IID Orthogonal filters
Figure 6: The empirical joint distribution of the preactivations generated by one same input flowing through 7-layer CNN with 3 x 3 kemels, at two different channel indexes. An isotropic 2D gaussian is included as level curves, corroborating with the predicted asymptotic independence between neurons as given in Theorem The input data X, are drawn from the MNIST dataset and 1000 experiments were conducted. The horizontal and vertical axes in each subplot are respectively (4) (X.) and 24 (Xe), where μ is the pixel located at (5,5).
3 channels
30 channels
300 channels
PSEUDO-IID Orthogonal filters
Figure The empirical joint distribution of the preactivations generated by two distinct inputs flowing through 7-layer CNN with 3 x kernels. The large channel limiting distribution as defined in Theorem involves the computation of a kernel covariance at high cost. We use instead the Decccation-Maximizztte
algorithm for each subplot and include level curves the best gaussian fit. The imput data X., X₁ are drawn from the MNIST dataset and 1000 experiments were conducted. The (correlated) gaussian nature of the preactivations is thus empirically verified as the gaussian fit goes better and better. The horizontal and vertical axes in each subplot are respectively (4) 11. (Xa)
and (4) (Xb), where μa is the pixel located at (5,5).
G EXAMPLES CONCERNING THE BOUNDED MOMENT CONDITION (III) OF THE PSEUDO-IID DISTRIBUTION
The bounded moment condition (iii) of the PSEUDO-IID distribution in Definition in the distinct proof of IID matrices taken in Hanin 2021 (Lemma 2.9). We show some examples of distributions in Figure X which verify or violate the conditions we identified as sufficient to rigorously prove the convergence of random neural networks Gaussian Processes in the large width limit.
2 is a key condition
H STRUCTURED SPARSE WEIGHT MATRICES IN THE FULLY CONNECTED SETTING
Fig. shows examples of permuted block-sparse weight matrices used to initialize a fully connected network in order to produce the plots given in Fig. 2
28

---

Published as a conference paper at ICLR 2024
leetica cosri nases ran samudo ID case
1045
Autoregressivve process non preudo ID case Cauchy case
korma
10
Orthograa aseado
10ah
a Aupt coo
1024
1040
10a
1 aa R Yea Crep Sega
0
200
400
600 800 1000
dimension N
Figure 8: Plots of [E] Σك Xj| against dimension N for different cases where conditions (ii) and (iii) of the pseudo IID regime are either satisfied or violated. Condition (iii) is considered with K, where X (X1, XN) is regarded
(1,...,)) 1) such that it becomes E Σー=XX, as one row of the weight matrix. The chosen distribution for the vector X impacts whether the network is in the PSEUDO-IID regime. In the identical coordinates case, X = (X1,... ... .x, the concatenation of the same realisation X1 sampled from a standard normal. Not only the traditional IID assumption is broken as the coordinates are obviously dependent but also condition (iii) is violated, thus resulting in unbounded expectation when growing the dimension N. The autoregressive Process (X,, -...... XN) shown is obtained by X; e, X where e; are IID multivariate Gaussian noises of dimension N. The correlation between the coordinates does decrease fast enough with the dimension to get a bound on the computed expectation and condition (iii) is once again violated. not obvious
On the contrary, from the plot produced by sampling IID Cauchy distributions, whether condition (iii) holds. Nonetheless, condition (ii) which ensures the finiteness of the variance not, thus a random network initialized with IID Cauchy weights falls outside the scope of our identified broad class of distributions to ensure convergence towards Gaussian Process. The last cases of samples taken either from scaled multivariate normals in red or uniformly sampled from the unit sphere in orange (with appropriate scaling such that condition (ūi) holds) show that there exists bound independent from the dimension on the expectation of interest. The empirical expectations are taken considering averages over 100000 samples.
width
width 30
width 300
Figure 9: Example a permuted block-sparse weight matrix at initialization fully connected network with increasing width N. The matrix is initialized with identically and independently sampled diagonal blocks from scaled Gaussian. Its rows and columns are then randomly permuted in order to satisfy the PSEUDO-ID conditions. The block size is set to be [0.2 N7. Entries in yellow are zero and entries in black are nonzero.
29


Reviews:
[{"review_id": "RHuVz1xwiq", "reviewer": "Reviewer_uyEL", "summary": "The paper extends previous results that showed randomly initialized multi-layer Bayesian neural networks with i.i.d. parameters are equivalent to Gaussian processes at the infinite width limit. The paper improves upon existing results by showing a similar result for a broader class of random parameters, namely pseudo-iid parameters. This class of random variables not only subsume classes of random variables that were included in previous literature (iid, orthogonal), but include new ones such as low-rank and block-sparse random variables. The authors results apply to fully connect networks as well as convolutional neural networks.", "questions": "- Pg. 1, Definition 1: $\\mathbf{a}$ is not defined.\n- Pg. 2, Section 1.1: It might make sense to mention the focus of some previous work on Bayesian neural networks, and how/why this is not the case in the current paper.\n- Pg. 3, Section 2.1: Referring to $N_0$ as the width of the first layer might be confusing, given that the input layer is different than the layer $l=1$, which might also be considered the first layer.\n- Pg. 3: What is the significance of the input space being assumed countably infinite?\n- Pg. 6, Section 3.1: It would make sense to be more explicit about how each of the following example distributions are not covered by previous research's findings.\n\n===========\n\nPost-rebuttal note: I thank the authors for their feedback. Although the arguments for the paper's contributions being limited have some merit, I still believe the contributions are sufficient to warrant acceptance, and retain my recommendation for doing so.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The connections to previous literature and the contributions of the present work in light thereof are clearly stated.\n- The writing and exposition is clear and easy-to-follow.\n- The new class of random variables for which the authors extend existing results is significantly larger than those investigated previously, and can facilitate future research in multiple directions.", "weaknesses": "- The paper expands upon existing results in a fairly specific strand of research, and it is not immediately clear why the novel families of random variables that they confirm to constitute Gaussian processes would be practically interesting. Though they provide some justifications in Section 3.3, these ideas are left for future research.\n- Their focus on randomly initialized deep neural networks in general, rather than Bayesian neural networks in Matthews et al. 2018 also limits the practical implications of their work. Though the latter were able to examine the implications of BNN-GP's in posterior inference, this is out-of-scope for the current work, limiting its practical implications to initialization schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m0SSuNands", "reviewer": "Reviewer_SWNp", "summary": "Generalises previous results regarding neural networks with IID initialization becoming Gaussian Processes (GP) in the infinite-width limit to other initialization schemes, such as orthogonal, normalised, low-rank and sparse initialization schemes. Prove the results for both fully connected and CNN networks and provide simulation results for the different initialization schemes.", "questions": "* Is it correct that the weights of the first layer must be Gaussian IID in all the relevant cases? So the proof work only for PSEUDO-IID weights in all other layers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The presented PSEUDO-IID framework replaces the IID requirement with centered and uncorrelated  entries, in a row-exchangable and column-exchangable matrix, under some technical conditions to prevent dependencies from changing the results. This is a general alternative to IID, which may be a useful generalisation for other cases as well.\n * Many different initialization schemes are PSEUDO-IID: IID, orthogonal, low-rank, and permuted block-sparse.", "weaknesses": "* The weights of the first layer must be Gaussian IID in all the relevant cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Av4HIPe3Kz", "reviewer": "Reviewer_EeDC", "summary": "The authors show that when the weights of neural networks are initialized in a \"pseudo\" i.i.d. manner, as defined by them, the random fields generated by each layer converge to Gaussian processes as the width of all layers goes to infinity simultaneously. They show this for fully connected neural networks and CNNs. \n\nTheir definition of pseudo i.i.d. involves row and column exchangeability, a variance given as a parameter, and bounded higher order moments. They illustrate the generality of their proposed assumption by showing several non IID examples used in practice that fall under their proposed definition. \n\nFinally, they present numerical simulations for the convergence to Gaussian processes for fully connected networks with widths up to 300, where they examine the variance and joint distributions over a single neuron.", "questions": "Maybe it is referenced in the citations, but how will Gaussian processes in the limit help in developing new network regularisers?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Their major contribution is showing convergence of pseudo i.i.d. initializations and simultaneous scaling. This appears to be a challenging problem compared to sequential scaling, and their assumption seems quite general because it includes some non i.i.d. ways of initialization, including orthogonal and low rank weights, some of which are also faster in practice. \n* Their contribution also seems like a first step to identify conditions for edge of chaos. \n\nIn general this paper is written very clearly, and the authors have given a good summary that shows exactly where their contribution fits in the existing literature. Moreover, their ideas and the proof technique for fully connected layers is easily understandable.", "weaknesses": "For CNNs, only one existing definition related to orthogonality, by Wang et al. in 2020, fits their pseudo i.i.d regime. Moreover I am not sure about feasibility, but including a numerical simulation for CNNs would make the study complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hIhRD1wmsC", "reviewer": "Reviewer_rGFF", "summary": "This paper extends the results of Matthews et al. (2018) about the convergence of random deep networks to Gaussian processes, to a wider class of distribution called ``Pseudo-IID\".", "questions": "Above I discussed my main concern with this paper. Here are some other comments:\n\n1. Definition 2 should come before Definition 1.\n\n2. Each of the elements in Definition 1 must be discussed right away after the definition is given. To the reader, it is almost impossible to understand why these assumptions are either needed or interesting. If it is ``difficult\" to explain these at this point in the flow of the paper, then probably those definitions should come after, and keep the introduction clean from technical mathematical definitions. \n\n3. While I read and understood the proof, it might be a good idea if the authors will explain the novelty and mathematical difficulties as compared to current literature.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The strength of this paper is yet another result/evidance about the convergence of random deep networks to a Gaussian process, for broader class of initial weight distributions.", "weaknesses": "The main weakness of this paper is that it seems as though that while the class of initial weight distributions is indeed broader than the Gaussian one Matthews et al. (2018), the techniques used to prove the result are pretty much the same as in Matthews et al. (2018), and the differences are just technicalities (e.g., CLT for exchangeable RVs). Therefore, since the main contribution of this paper is supposed to be for the theoretical study of NN, I believe that the paper is not novel in that sense. Also, there are many grammatical mistakes and in general it feels that the authors did not make an all out effort when writing their paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)