Precise characterization of the prior predictive distribution of deep ReLU networks
Authors: Lorenzo Noci
Gregor Bachmann
Kevin Roth
Sebastian Nowozin
Thomas Hofmann
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Bayesian Neural Networks
Priors
Meijer-G
NNGP
finite width corrections

Abstract: Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. 
Similar in spirit to the kind of analysis that has been developed to devise better initialization schemes for neural networks (cf. He- or Xavier initialization), we derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights.
While theoretical results have been obtained for their heavy-tailedness,
the full characterization of the prior predictive distribution (i.e. its density, CDF and moments), remained unknown prior to this work. Our analysis, based on the Meijer-G function, allows us to quantify the influence of architectural choices such as the width or depth of the network on the resulting shape of the prior predictive distribution. 
We also formally connect our results to previous work in the infinite width setting, demonstrating that the moments of the distribution converge to those of a normal log-normal mixture in the infinite depth limit. 
Finally, our results provide valuable guidance on prior design: 
for instance, controlling the predictive variance with depth- and width-informed priors on the weights of the network.

Precise characterization of the prior predictive distribution of deep ReLU networks
Lorenzo Noci'
Gregor Bachmann"
Kevin Roth"
Dept of Computer Science
Dept of Computer Science ETH Zürich gregor bachmann ethz.ch
Dept of Computer Science ETH Zürich kevin.roth@inf ethz. ch
ETH Zürich
lorenzo. noceioo.nooiino @inf ethz.ch ch
Sebastian Nowozin Microsoft Research Cambridge, UK
Thomas Hofmann
Dept of Computer Science ETH Zürich
Sebastian. Manaasiiia..ooooziimmm.arroooae
thomas. www.amn@inff....mmmmom.mmmmmo..m ewteeiiff.ehahhee ethz ch
Abstract
Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. Similar in spirit to the kind of analysis that has been developed to devise better initialization schemes for neural networks (cf. He- or Xavier initialization), we derive a precise characterization of the prior predictive distribution finite-width ReLU networks with Gaussian weights. While theoretical results have been obtained for their heavy-tailedness. the full characterization of the prior predictive distribution (i.e. its density, CDF and moments), remained unknown prior to this work. Our analysis, based on the Meijer-G function, allows us quantify the influence of architectural choices such the width or depth of the network on the resulting shape of the prior predictive distribution. We also formally connect our results to previous work in the infinite width setting, demonstrating that the moments of the distribution converge to those normal log-normal mixture in the infinite depth limit. Finally, our results provide valuable guidance on prior design: for instance, controlling the predictive variance with depth- and width informed priors on the weights of the network.
Introduction
is well known that standard neural networks initialized with Gaussian weights tend Gaussian processes (Rasmussen, 2003) in the infinite width limit (Neal, 1996; Lee et al., 2018; de G. Matthews al., 2018), coined neural network Gaussian process (NNGP) in the literature. Although the NNGP has been derived for number of architectures, such convolutional (Novak et al., 2019; Garriga- Alonso et al., 2019), recurrent (Yang, 2019) and attention mechanisms (Hron al., 2020), little known about the finite width case.
The reason why the infinite width limit is relatively tractable to study that uncorrelated but dependent units of intermediate layers become normally distributed due to the central limit theorem (CLT) and result, independent. In the finite width case, zero correlation does not imply independence, rendering the analysis far more involved as we will outline in this paper.
One of the main motivations for work is to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. As argued by Wilson and Izmailov (2020); Wilson (2020), the prior parameters does not carry meaningful
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

interpretation; the prior that ultimately matters is the prior predictive distribution that is induced when a prior over parameters is combined with a neural architecture (Wilson and Izmailov, 2020; Wilson, 2020).
Studying the properties of this prior predictive distribution is not an easy task, the main reason being the compositional structure of neural network, which ultimately boils down to products of random matrices with a given (non-linear) activation function. The main tools study such products are the Mellin transform and the Meijer-G function (Meijer, 1936; Springer and Thompson, 1970; Mathai, 1993; Stojanac et al., 2017). both of which will be leveraged in this work to gain theoretical insights into the inner workings of BNN priors.
Contributions Our results provide important step towards understanding the interplay between architectural choices and the distributional properties of the prior predictive distribution, in particular:
We characterize the prior predictive density of finite-width ReLU networks of any depth through the framework of Meijer-G functions.
We draw analytic insights about the shape of the distribution by studying its moments and the resulting teavy-tailedness. We disentangle the roles of width and depth, demonstrating how deeper networks become more and more heavy-tailed, while wider networks induce more like distributions.
We connect our work to the infinite width setting by recovering and extending prior results (Lee et al., 2018; Matthews al., 2018) the infinite depth limit. We describe the resulting distribution in terms of its moments and match it to normal log-normal mixture (Yang, 2008), empirically providing an excellent even in the non-asymptotic regime.
Finally, introduce generalized He priors, where desired variance be directly specified in the function space. This allows the practitioner to make an interpretable choice for the variance, instead of implicitly tuning it through the specification of each layer variance.
The rest of the paper is organized as follows: in Section 3.1, introduce the relevant notation for the neural network that will be analyzed. We describe the prior works of Lee et al. (2018); Matthews al. (2018) in more detail in Section 3.2. Then, in Section 3.3, we introduce the Meijer-G function and the necessary mathematical tools. In Section 4 we derive the probability density function for a linear network of any depth and extend these results to ReLU networks, which represents the key contribution of our work. In Section 5, we present several consequences of our analysis, including an extension of the infinite width setting to infinite depth as well as precise characterizations of the -eill-esisss in the finite regime. Finally, in Section 6, we show how one can design an architecture and a prior over the weights achieve a desired prior predictive variance.
2 Related Work
Although Bayesian inference in deep learning has recently risen in popularity, surprisingly little work has been devoted to investigating standard priors and their implied implicit biases in neural architectures. Only through the lens of infinite width, progress has been made (Neal, 1996), estab- lishing a Gaussian process behaviour at the output of the network. More recently. Lee et al. (2018) 1) and Matthews et al. (2018) extended this result to arbitrary depth. We give brief introduction in Section 3.2. Due to their appealing Gaussian process formulation, infinite width networks have been extensively studied theoretically, leading to novel insights into the training dynamics under gradient descent (Jacot et al., 2018) and gencralization (Arora et al., 2019b).
Although theoretically very attractive, the usage of infinite width has been severely limited by its inferior empirical performance (Arora et al., 2019a). While first insights into this gap have been obtained (Aitchison, 2020; Aitchison et al., 2021), the picture is far from complete and a better understanding of finite width networks is still highly relevant for practical applications. In the finite regime, however, such precise characterizations in function space have been elusive so far and largely limited to empirical insights -Seaphr- et al., 2017, 2018) and investigations of the heavy-tailedness of layers (Vladimirova et al., 2019; Fortuin et al., 2021). The field of finite-width corrections has recently gained lot of attraction. Hanin and Nica (2020) studies the simultaneous limit of width and depth of the Jacobian of ReLU net. More recently, number of concurrent works

---

appearing shortly before or after ours, such as Zavatone- Veth and Pehlevan (2021); Roberts et al. (2021); Li al. (2021) study properties of large-but- but--iitte neural nets. Notably, Zavatone- Veth and Pehlevan (2021) concurrently derived similar results on the characterization of the prior predictive distribution of finite width networks, however, we did not discover them until after we had completed this work. Note also that our novel limiting behaviours (cf. Sec. 5.1) and our analytical insights into heavy-tailedness (cf. Sec. 5.2) clearly distinguish our work from theirs. offer valuable guidance a similar limiting result in the infinite-width and depth setting in the case of Resnets (ours for conlecconnected, cf. Sec. 5.1), albeit with completely different proof technique. Moreover, also give precise insights into the prior predictive distribution for finite width (cf. Sec. 4), whereas Li et al. (2021) only work with the limits.
addition, our results also prior design for ML practitioners (cf. Sec. 6). Li et al. (2021) derive
Finally, this work is also related to the studies signal propagation into finite-width random networks (Poole et al., 2016; Schoenholz et al., 2017) and initialization (He et al., 2015; Hanin and Rolnick, 2018). In particular, He al. (2015) uses a second moment analysis to specify the variance of the weights. In this sense, our approach extends it by deriving all the moments of the distribution.
3 Background
3.1 Fully Connected Neural Network
Given an input æ E Rd we define 0-parameterized L layer fully-connected neural network fo( as the composition of layer-wise affine transformations W()) E 1Rd4-1xd; and element-wise non-linearities R R,
fo(x) W(L) (W. (w(() ())
where 0 = (W ..... W(L) denotes the collection of all weights. Throughout this work, we assume standard initialization, i.e. w variance of Often, it will be more convenient to work with the corresponding unit-level formulation, expressed through the recursive equations:
f) ~O, of), where weights in each layer can have a different
di-
1 (x)
Σ 3=1
w W 9j (x)
(x) =
When propagating input through the network refer to f E Rd as the pre-activations and to g (z)) Rd: as the anstvactivasions. When it is clear from the context, enhance readability, we will occasionally abuse notation denoting (() (z) and g( g() gd) (z). We will also use the short- hand [m] = {1...., m}. Imposing a probability distribution on the weights induces a distribution on the pre-activations f() (z). Understanding the properties of this distribution is the main goal of this work.
3.2 Prior Predictive Distribution and Infinite Width
precise characterization of the prior predictive distribution of neural network has been established in the so-called infinite width setting (Neal, 1995; Lee et al., 2018; Matthews et al.. 2018). By considering variances that scale inversely with the width, i.e. of t and fixed depth be shown that the implied prior predictive distribution converges law a Gaussian process:
N, it can
fo 4 GP(O,
(3)
where 2))) : RR x Rdd æ, ((L) x') is the NNGP kernel (Lee et al., 2018), available in closed-form through the recursion
(x) x') x vTa
(1, x²) = E E (0000000000(((0(022))))) (0(2)0((22)
(4)
(x.z)) {x,
for I 1,...,I. I. and where 5
E Rx22 The proof relies
(0('' wx) (0
on the multivariate central limit theorem in conjunction with an inductive technique, letting hidden
3

---

0.2
0.0
(a) Plots of Meijer-G functions
Visualization of integration path
Figure (a) Plot of the Meijer-G functions Gp (-16) for I 1, and 5), (0, 5, 5). (b) An example path C in the complex plane. Notice how the orange singularities are always the left of C and the red ones always to the right.
layer widths go infinity in sequence. Due to the Gaussian nature of the limit, exact Bayesian inference becomes tractable and techniques from the Gaussian process literature can be readily applied (Rasmussen, 2003). Although theoretically very appealing, from a practical point of view, infinite width networks are not as relevant due to their inferior performance (Novak et al., 2019). Gaining better understanding this performance gap hence of utmost importance.
3.3 Meijer-G function
The Meijer-G function is the central tool to our analysis of the predictive prior distribution in the finite width regime. The Meijer-G G function is ubiquitous tool, appearing in variety of scientific fields ranging from mathematical physics (Pishkoo and Darus, 2013) to symbolic integration software (Adamchik and Marichev, 1990) to electrical engineering (Ansari et al., 201 1). Despite its high popularity in many technical fields, there have only been a handful of works MI leveraging this elegant and convenient theoretical framework (Alaa and van der Schaar, 2019; Crabbe et al., 2020). In the following, we will introduce the Meijer-G function along with the relevant mathematical tools develop our theory.
For E C with real-part R(s) > 0, denote by ((s) the Gamma function defined as
T(s)
ee-le-=das
We then consider the analytic continuation of I, which extends it to the entire complex plane, as frequently done in complex analysis. Now fix m. P. N such that 0 < m S and 0 S S and consider a E Rº, R2 such that a; - by & Z Vi 1.... P and j ......q. The Meijer-G function (Meijer, 1936; Mathai, 1993; Mathai and Saxena, 2006). is defined as:
2 /
Gm (=1
x(s)= dss,
(6)
where:
((s 11 [bb + 8) [(1 aj s)
IT ['(1 - 5) TT= I' +
and the integration path L defines suitable complex curve, described in the following. Recall that the function r(s) has poles at 0, -1 -2....... ... all the way to -0. Hence, T(1 a, s) has poles -a, + 1, + ......ar the way to oc, and l(b, + s) has poles -bj, - 1...., all the way to -c. The path L is defined such that the poles of r(b; + +s) are to the left of L and the ones of T(1 a, 8) are to the right of it. The condition a; by E Z,o makes sure we can find such a separation as this implies that the poles do not overlap. We illustrate Figure lb. red we display the poles T - a, + 8) and in orange the poles of N(b + s). For interested readers refer to Beals and Szmigiclski (2013) for more extensive introduction.
example of such path in

---

The defining property of Meijer-G functions is their closure under integration, i.c. the convolution of two Meijer-G G functions is again a Meijer-G function. Combined with the fact that most elementary functions can be written as Meijer-G function, this property becomes extremely powerful at expressing complicated integrals neatly. Our proofs leverage this result extensively by expressing the integrands encountered in the prior predictive function as Meijer-G functions.
Throughout this text, we will only encounter Meijer-G functions of simpler signature G (-1b). For completeness, we show its functional form here:
G (=lb)
T(b; + s)ds.
2Ti J
Small values for m E N correspond to familiar functions such as the exponential G (z = e
and the modified Bessel function of second kind 6 1 1 ) (z). We visualize several Meijer-G functions of this form Figure Ia. For illustrative purposes, we normalize the functions to have a maximum value of 1.
4 Predictive Priors for Neural Networks
In this section detail our theoretical results the predictive prior distribution implied by fully-connected neural network with Gaussian weights, both with and without ReLU non-linearities.
Linear Networks: First, we consider linear networks, i.c. fully-connected networks where the post-activations coincide with the pre-activations. They can be characterized as the product of Gaussian random matrices, for which the result in terms of Meijer-G functions is known (consider for instance Ipsen (2015)). To highlight the differences between the linear and the non-linear approach, we re-prove the linear case leveraging our proof technique and notation. For simplicity, we assume w.l.o.g. that the input is normalized, i.e ||=||| 1. We now present the resulting distribution of the predictive prior:
Theorem 4.1. Suppose l 2 and the input has dimension dg Then, the joint marginal density of the random vector f() (i.e. the density of the 1-th layer activatiia))) is proportional to:
p(f(0) 6/i00
(12
0.
1 1 (d1 di) ...... (di-1
di))
(9)
02
where o² = -
Our proof is based on inductive technique by conditioning on the pre-activations f() of the previous layer while analyzing the pre-activations the next layer. Due to space constraints, we defer the full proof of Thm 4.1 to the Appendix B.2. In the following, to give flavor of our technique and highlight the great utility of the Meijer-G function, we present the base case as well as the key technical result (Lemma 4.2) to perform the inductive step obtain the final statement.
Base case (L = 1): Here we restrict our attention to the first layer pre-activations which, given an input æ. are defined as:
= do
W jk
(10)
Conditioned the input, this is sum of dg i.i.d. Gaussian random variables w-ll)
N0 of).
which is again Gaussian with mean zero and variance ofl [|||||]] 1| i.e. (() NO | @ (0, of). where the last equality follows from ||x|| 1. As we are conditioning on the input, the joint distribution of the first layer units is composed of di independent Gaussians and hence
(0) ~ N(0,
(11)
5

---

Indeed, as anticipated from Thm. 4.1, the corresponding Meijer-G function encodes the Gaussian density: 1011
[1 20 ( exp
11 20
(12)
Induction Step: Conditioning on the previous pre-activations f(--1) brings us into a similar setting as in the base case because the resulting conditional distribution is again a Gaussian:
pl (f
(N (
(13)
In contrast to the fixed input a, now need to apply the law of total probability to integrate out the dependence on f((=1), leveraging the induction hypothesis for p(f((-1) obtain the marginal distribution (f). This is where the Meijer-G function comes in handy as it can easily express such
an integral. In combination with the closedness of the family under integration, this enables us to
perform an inductive proof. We summarize this in the following Lemma:
Lemma 4.2. Let f and fi be di d--dimensional and di-1 -imeeninnsi vectors, respectively, where >> E N. Let of > O, x² > 0 and b1.... b E R. Then the following integral:
|1
R p(f()) (())))(()))))
1-1.0 0,1-1
b1...., bl ....,,,bi))) df
(14)
21-152
can be expressed as:
11 22
1
5 di) + be.1):
I C G.
0,
(d) dj) + bl.......
(15)
2
where 0² = ofo?, and E R is constant available in closed-form
The proof largely relies on well-known integral identities involving the Meijer-G function. Again, due its technical nature, refer for the full proof and for the exact constant C to the Appendix B.I and B.6, respectively.
ReLU Networks: In the previous paragraph have computed the prior predictive distribution for a linear network with Gaussian weights. Now, we extend these results to ReI.U networks. The proof technique used is very similar its linear counterpart, the main difference stems from the need to decompose the distribution over active and inactive ReLU cells. As consequence, the resulting density is superposition of different Meijer-G functions, each associated with a different active (linear) subnetwork. This is presented in the following:
Theorem 4.3. Suppose Z 2, and the input has dimension dg- Define the multi-index set R = [d X [d.1]] and introduce the vector u" E c through its components u. = (r, d,).
(IIFRaOl 20
p(fROLD) = -
0, (,,"))
+ 1000 #00(fRduu)
(16)
PER
where o² = I1 of and the individual weights are given by
gr = π - 42 2d: (02)
1-1
d
1
(17)
ri
2d.
i=1
and
1-1
2d
40
(18)
24
i=1
We refer to Appendix C.2 for the detailed proof. Observe that sub-networks with the same number of active units per layer induce the same density. The theorem reflects this symmetry by not summing over all possible sub-networks but only over the representatives of each equivalence class, absorbing

---

the contribution of the other equivalent networks into 9r. Indeed, notice that [R] - exactly the number of equivalent sub-networks contained the original network. Notice also that Ir governs how much weight is assigned to the corresponding Meijer-G function. The normalization constant is incorporated into the weights Ir since the Meijer-G function does not integrate to (see Appendix B.4). As result, the weights add up to the sum of the respective integration constants, and not to 1.
d, which
Analytic Insights into the Prior Predictive Distribution
Here we highlight how one can use the mathematical machinery of Meijer-G functions to derive interesting insights, relying on numerous mathematical results provided in the literature (Gradshteyn and Ryzik, 2013; Brychkov, 2008; Andrews, 2011). With this rich line of work at our disposal, we can easily move from the rather abstract but mathematically convenient world of Meijer-G functions to very concrete results. We demonstrate this by recovering and extending the NNGP results provided Lee al. (2018); Matthews et al. (2018). In particular, our analysis allows for simultaneous width and depth limits, showing how different limiting distributions emerge as a consequence of the growth with respect to L. Finally, we characterize the heavy-tailedness of the prior predictid for any width, providing further evidence that deeper models induce distributions with heavier tails, observed in Vladimirova et al. (2019). We hope that this work paves the way for further progress understanding priors, leveraging this novel connection through the rich literature Meijer-G functions.
5.1 Infinite Width: Recovering and Extending the NNGP for a Single Datapoint
We will start by giving an alternative proof in the linear case for the Gaussian behaviour emerging as the width of the network tends to infinity, recovering the results of Lee et al. (2018); Matthews al. (2018) in the restricted setting of having just one fixed input æ. We extend their results in the following ways:
1. We provide convergence proof that is independent of the ordering of limits.
2. We characterize the distributions arising from simultaneous infinite width and depth limit, considering different growth rates for depth
For ease of exposition, we focus on the equal width case, i.e. where dj =... = di m with one output dr. 1. To have well-defined limits, one has resort the so-called NTK parametrization (Jacot et al., 2018), which is achieved by setting the variances as of = 1 and of = 2 for = ........ We summarize the result in the following:
Theorem 5.1. Consider the distribution of the output p (FReU). as defined in Thm. 4.3. Denote
X ~ N)0, Y C 27) for X 1 Y and the resulting normal log nog-normal mixture by Z = XY, for > 0. Let the depth grow as L c ym where B 2 0 and E N fixed. Then it
holds that
E 2k (2k 1)!!
if
E
(L) (Re.U))
2k
m 000
E[ (3 1)1!
i8 =
(19)
if
where (2k - 1)!! (2k - 1). .)...3... denotes the double factorial (by symmetry, odd moments are zero). Moreover, for 3 < it holds that
(L) 4 X for oC,
(20)
for any k > 1.
The proof leverages the fact that we can easily calculate all the moments fkeiu at finite width using well- integral identities involving the Meijer-G function. For the limit, care has to be taken as the moments of /eeu involve the moments of the Binomial distribution Bin(m, 1) for which recursive but no analytic formulas available. By carefully studying the leading coefficients of the
7

---

resulting polynomial, we can establish the result in Thm. 5.1.
This result is in stark contrast to Lee al. (2018); Matthews al. (2018) which cannot deal with simultancous depth limit due to the inductive nature of their proof. but can only describe the special case of fixed depth (7 1, 3 = 0). Notice that for depth growing asymptotically than width (L ym for B < 1), we obtain the convergence in distribution since the Gaussian distribution identified by its moments (See Theorem 30.2 in Billingsley (1986)). For 3 we can also prove convergence of the moments and identify them as arising from the normal log- nognnormal mixture (Yang, 2008). Unfortunately. this does not suffice conclude convergence in distribution as there are no known results on the identifiability of Z. Empirical evidence however, presented in Figure and Figure 3, suggests that the normal log-normal mixture captures the distribution to an excellent degree of fidelity. We demonstrate this through the following experiment. Note first that 7 + (0, 1) in law as 7 - O, which expected, as for 7 - 0, we assume fixed depth 7= c. We can hence use this result the con-ssymtootic regime by setting 7 m expecting to interpolate between the two distributions as we vary depth and width. We illustrate the empirical findings in Figure and Figure 3. We find an astonishing match to the true distribution both in terms of CDF and PDF. Moreover, expected, we recover the Gaussian distribution for small values of 7.
slower rate
4.
The decoupling of the moments into two separate factors gives insights into the role of width and depth on the shape of the distribution. The emergence of the log-normal factor in this infinite depth limit highlights how deeper networks encourage more heavy-tailed distribution, if not countered by a sufficient amount of width. This becomes even more drastic once depth outgrows width (3 > 1), leading to divergence of all even moments. In the following section we will show that also in the finite width regime, heavy-tailedness becomes more pronounced we increase the depth.
1017
20a
(a) 7 0.01
(b) 0.1
(c) 0.5
Figure 2: Convergence to normal log-normal mixture: We display the CDFs of a ReLU network, standard Gaussian CDF and the normal log-normal mixture. The width is fixed to 100 while depth is increased from 1 to 50 using the parameter 7. The neural network CDF is constructed empirically by drawing 1 samples from the prior. Note that as depth increases, the CDF departs from the Gaussian but still follows the normal log-normal mixture.
0.1
0.4
033
0.1
00
0.0
(a) 0.01
(b) 7 0.1
(c) 7 = 0.5
Figure 3: Convergence to normal log-normal mixture: We show the density plots for the same setting as in Figure 2. Note that as depth increases, the density departs from the Gaussian but still follows the normal log-normal mixture.
5.2 Heavy-tailedness Increases with Depth
From the moments analysis, it simple to recover a known fact about the prior distribution of neural networks, namely that deeper layers are increasingly heavy-tailed (Vladimirova et al., 2019). To see this, we can derive the kurtosis, a standard measure of tailedness of the distribution (Westfall, 2014), defined as:
IE (X-
E[X])*
(21)
VXF
8

---

where X is univariate random variable with finite fourth moment. We can calculate the kurtosis of a ReLU network analytically at any width, relying on closed-form results for the lower order moments of the Binomial distribution. We outline the exact calculation in the Appendix C and state the resulting expression here:
(m, = m + 5
L-1
(22)
Note how the kurtosis increases with depth and decreases with the width m, highlighting once again the opposite roles those two parameters take regarding the shape of the prior. As expected, for fixed depth L, the kurtosis converges to 3, i.e. limm-> KReLu(m (m. L) = 3, which is the kurtosis of a standard Gaussian variable (0, 1). For the simultaneous limit L rym, find value of 3 which exactly matches with the expression derived in Thm. 5.1:
E|
e?>2!!!
lim KRELU (m,ym) E| m 700
37
(23)
eO1!!
6 Prior Design
In Section 5.2, we have already outlined how the choice of architecture influences the heavy-tailedness of the distribution. If a Gaussian-like output is desired, the architecture should be designed in such a way that the width m significantly exceeds the depth L (small 7). while heavy-tailes predictive priors can be achieved by considering regimes where L exceeds m (big 7). As will shortly after in this section, another consequence of our analysis is that we can now directly work with the variance in function space, instead of implicitly tuning by changing the variances each layer. In this way, the variance over the weights has a clear interpretation in terms of predictive variance, making it easier for the deep learning practitioner to take an informed decision when designing prior for BNNs. This type of variance analysis has previously been used to devise better initialization schemes for neural networks coined "Heeinitialization" (He al., 2015) in the Gaussian case and "Xavier initialization" (Glorot and Bengio, 2010) for uniform weight initializations. Therefore, we coin the resulting prior Generalized He-prior.
We consider the parametrization of = 1 and of = 21, m which, as we show in Appendix C, leads the following predictive variance:
Predictive variance:
2
2
(24)
Suppose we want a desired output variance a². This can be achieved as follows: let as with
i = 2,....,, ., be l - coefficients such that Di-1 a, = =1-1 1. Then choose f = (a²)= that the layer variances of are given as
S implying
2 (0²) +
of
(25)
He-priors correspond to the special case where 0² = and a; = = 1, 1,..... 1 1- 1, while a standard
Gaussian N(0, 1) prior results in の² (m) t-1 of 1, and o; 1, i 1, ,,.... - 1. Note how "far" He-priors can be from standard Gaussian for relatively deep and wide neural networks. While rather innocent-looking, using standard Gaussian (V, 1) prior can lead to extremely high output variance.
Combining with the previous insights, practitioners can now choose desired output variance along with a desired level of heavy-tailedness in a controlled manner by specifying the architccture, i.c. the width m and the depth L.
7 Discussion
Our work sheds light on the shape of the prior predictive distribution arising from imposing Gaussian distribution on the weights. Leveraging the machinery of Meijer-G functions, characterized the

---

density of the output in the finite width regime and derived analytic insights into its properties such as moments and heavy-tailedness. An extension the stochastic process setting in the spirit of Lee et al. (201 8); Matthews et al. (2018) as well as to convolutional architectures (Novak et al., 2019) could bring theory even closer to practice and we expect similar results to also hold in those cases. This is however beyond the scope of our work and we leave it as future work.
Our technique enabled us to extend the NNGP framework to infinite depth, discovering how in the more general case, the resulting distribution shares the same moments as a normal log-normal distribution. This allowed us to disentangle the roles of width and depth, where the former induces a Gaussian- like distribution while the latter encourages heavier tails. Empirically, we found that the normal log- normal mixture provides an excellent fit to the true distribution even in the non- asymptotic setting, capturing both the cumulative and probability density function to a very high degree of accuracy. This surprising observation begs further theoretical and empirical investigations. In particular, discovering a suitable stochastic process incorporating the normal log-normal mixture, could lend further insights into the inner workings of neural networks. Moreover, the role of heavy-tailedness regarding generalization is very intriguing, potentially being an important reason underlying the gap between infinite width networks and their finite counterparts. This is also in-line with recent empirical works on priors, suggesting that heavy-tailed distributions can increase the performance significantly (Fortuin et al., 2021).
Using these insights, we described how one can choose fixed prior variance directly in the output space along with desired level of heavy-tailedness resulting from the choice of the architecture. We leave it as future work to also consider higher moments to give more nuanced control over the resulting prior in the function space. Finally, we hope that the introduction of the Meijer-G function sparks more theoretical research on BNN priors and their implied inductive biases in function space.
References
Adamchik, S. and Marichev, O. I. (1990). The algorithm for calculating integrals of hypergeometric type functions and its realization in reduce system. In Proceedings of the International Symposium on Symbolic and Algebraic Computation, ISSAC '90, page 212-224, New York, NY, USA. Association for Computing Machinery.
Aitchison, L. (2020). Why bigger is not always better: on finite and infinite neural networks. Proceedings of the 37th International Conference on Machine Learning.
Aitchison, L., Yang, A., and Ober, S. W. (2021). Deep kernel processes. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 130-140. PMLR.
Alaa, A. M. and van der Schaar, M. (2019). Demystifying black-box models with symbolic meta- models. In NeurlPS, pages 11301-11311.
Andrews, L. C. (2011). Field guide to special functions for engineers / Larry Andrews. Field guide series; 18. SPIE, Bellingham, Wash.
Ansari, I. S., Al-Ahmadi, S., Yilmaz, F., Alouini, M.-S., and Yanikomeroglu. H. (2011). A new formula for the ber of binary modulations with dual- selection over generalized-k composite fading channels. IEEE Transactions on Communications, 99(10):2544-2558
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R. (2019a). On exact computation with an infinitely wide neural net. 33rd Conference on Neural Information Processing Systems (NeurIPS).
Arora, S., Du, S. S., Hu, W., Li, Z., and Wang, R. (2019b). Fine grainerained analysis of optimization and generalization for overparameterized two-layer neural networks. Proceedings of the 36th International Conference Machine Learning (ICML.
Beals, R. and Szmigielski, J. (2013). Meijer g- g-fucttions: a gentle introduction. Notices of the American Mathmattiaa Society, 60:866-873.
Benyi, A. (2005). A recursive formula for moments of a binomial distribution. The college mathe- matics journal.
10

---

Billingsley, P. (1986). Probability and Measure. John Wiley and Sons, second edition.
Boros, G. and Moll, V. (2004). Irresistible Integrals: Symbolics, Analysis and Experiments in the Evaluation of Integrals. Cambridge University Press.
Brychkov, (2008). Handbook of special functions: Derivatives, integrals, series and other formulas. Chapman and Hal//RCC.
Crabbe, J., Zhang. Y., Zame, W., and van der Schaar, M. (2020). Learning outside the black-box: The pursuit of interpretable models. I Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 17838-17849. Curran Associates, Inc.
de G. Matthews, A. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. In ICLR.
-lam-Shepherd, D., Requeima, J., and Duvenaud, D. (2017). Mapping gaussian process priors to bayesian neural networks. In Second workshop on Bayesian Deep Learning (NeurIPS).
Flam-Shepherd, D., Requeima, J., and Duvenaud, D. (2018). Characterizing and warping the function space of bayesian neural networks. Third workshop on Bayesian Deep Learning (NeurIPS).
Fortuin, V., Garriga-Alonso, A., Wenzel, F., Rätsch, G., Turner, R., van der Wilk, M., and Aitchison,
L. (2021). Bayesian neural network priors revisited. arXiv preprint arXiv: 100.06671.
Carriga-Alonso, A., Rasmussen, C. E., and Aitchison, L. (2019). Deep convolutional networks as shallow gaussian processes. In ICLR.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. Teh, W. and Titterington, M., editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy. PMLR.
Gradshteyn, and Ryzik, (2013). Table integrals, series, and products. Elsevier/Academic Press.
Hanin, B. and Nica, M. (2020). Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, 376(1):287-322
Hanin, B. and Rolnick, D. (2018). How start training: The effect of initialization and architecture. arXiv preprint Xrviv: 803.0/719.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. (2020). Infinite attention: Nngp and ntk for deep attention networks. International Conference Machine Learning, pages 4376 4386. PMLR.
Ipsen, J. R. (2015), Products of independent gaussian random matrices.
Jacot, A., Gabriel, F., and Hongler. C. (2018). Neural tangent kernel:
Convergence and generalization in neural networks. 32rd Conference on Neural Information Processing Systems (NeurIPS).
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2018). Deep neural networks gaussian processes. In ICLR.
Li, M. B., Nica, M., and Roy, D. M. (2021). The future is log-gaussian: Resnets and their infinite- width-and-width limit at initialization.
Mathai, A. M. (1993). A handbook of generalized special functions for statistical and physical sciences. Oxford University Press, USA.
Mathai, M. and Saxena, R. K. (2006). Generalized hypergeometric functions with applications in statistics and physical sciences, volume 348. Springer.

---

Matthews, A. G. d. G. Rowland, M.. Hron, J., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:
Meijer, C. (1936). Uber whittakersche bezw. besselsche funktionen und deren produkte. Nieuw Archiefvoor voor Wiskunde, 18(2):00-29.
Neal, R. M. (1995). Bayesian learning for neural networks. PhD thesis, University of Toronto.
Neal, R. M. (1996). Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer.
Novak, R., Xiao, L., Bahri, Y., Lee, J., Yang. G., Hron, J., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. (2019). Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR.
Pishkoo, A. and Darus, M. (2013). Some applications of meijer g-functions as solutions of differential equations in physical models. Journal of Mathematical Physics, Analysis, Geometry.
Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2016). Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29:3360-3366 -366.
Rasmussen, C. E. (2003). Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer.
Roberts, D. A.. Yaida, S., and Hanin, B. (2021). The principles of deep learning theory. arXiv preprint arXiv:2 106, 10165.
Rudin, W. (1976). Principles of Mathematical Analysis. McGraw-Hill.
Schoenbolz, S. S., Gilmer, J., Ganguli, S., and Sohl-Dickstein, J. (2017). Deep information propaga- tion. In ICLR.
Springer, M. D. and Thompson, W. R. (1970). The distribution of products of beta, gamma and gaussian random variables. SIAM Journal on Applied Mathematics, 18(4):721-77.
Stojanac. Z., Suess, D., and Kliesch, M. (2017). On products of gaussian random variables. arXiv preprint arXiv:1711.10511
Vladimirova, M., Verbeek, J., Mesejo, P., and Arbel, J. (2019). Understanding priors in bayesian neural networks the unit level. International Conference on Muchine Learning, pages 6458-6467. PMLR.
Westfall, P. H. (2014). Kurtosis as peakedness, 1905-2014. rip. The American Statistician, 68(3):191- 195.
Wilson, A. G. (2020). The case for bayesian deep learning, arXiv preprint arXiv:2001.10993
Wilson, A. G. and Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv.2002.08791.
Yang, (2019). Wide feedforward or recurrent neural networks of any architecture are gaussian processes.
Yang, M. (2008). Normal log-normal mixture, leptokurtosis and skewness. Applied Economics Letters, 15(9)7377-742
Zavatone-Vent Veth, A. and Pehlevan, C. (2021). Exact priors of finite neural networks. arXiv preprint 114734.
12


Reviews:
[{"review_id": "wD-egiNgiIl", "reviewer": "Reviewer_E89g", "summary": "Uses the Meijer-G function to analyse the properties of finite-width Bayesian neural networks.  Provides additional proofs of the infinite-width limit, in addition to results for the infinite depth limit.  Shows that finite networks are heavy tailed and that this increases with depth.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the paper, the while the maths are a bit heavy and will be unfamiliar to most researchers, they do a fantastic job giving a clear high-level description of    what the Meijer-G function is and why it is useful to understand Bayesian neural networks.\n\nMy key issue surrounds references to past work.  The authors appear to be unfamiliar with a line of work that began with Aitchison (2019) \"Why bigger is not always better:  on finite and infinite neural networks\".  This work explicitly address the question of why infinite networks perform worse than finite networks (which is mentioned here in  the related work).  It concludes that finite networks differ from infinite networks principally in the existence of flexibility in the Gaussian process kernel, with deeper, narrower networks having more flexibility.  Flexibility/variability in the kernel naturally implies heavy-tailed-ness of the output distribution.  But more importantly, the view in terms of representations gives us much more insight into what the flexibility in finite nets is actually used for.  This work was continued by Aitchison el al. (2020) \"Deep kernel processes\", which uses these insights to develop a new family of flexible, deep kernel methods that are in some cases equivalent to deep neural networks, and give considerable  insights into representation learning in practically relevant deep networks.\n\nThere are a number of most likely contemporaneous works (potentially submitted to this conference).  While the authors may not have been aware of many of these (I realise that some may have appeared after the submission date to the conference), it is important to discuss the interrelationships between these works in the \"Related Work\" section.  These papers include:\n* https://arxiv.org/abs/2104.11734\n* https://arxiv.org/abs/2106.06529\n* https://arxiv.org/abs/2106.00651\n* https://arxiv.org/abs/2106.04013\n\nVery happy to increase the score if these issues are resolved!\n\nThe \"Predictive variance\" section is a bit confusing.  It just seems to tell us how to set the variance of the weights of an infinite width in the network such that the output variance is a prescribed value e.g. tells us that a standard Gaussian prior gives extremely high output variance.  Unless I'm missing something, this has been obvious for quite a long time, and was the motivation behind the introduction of this the He normalisation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ckb4hlr9nVP", "reviewer": "Reviewer_55jz", "summary": "The main result of the paper is a derivation of the exact characterization of priors in function space through Meijer G-functions. The results are obtained for Gaussian priors, ReLU and linear activation functions. The article also provides a solid analysis of the obtained characterization. \n\n**Advantages**: \n* It is the first accurate description of hidden units for standard feed-forward neural networks among the concurrent work mentioned in the paper. \n* The results are in line with other works on the heavy-tailed nature of hidden units and the Gaussian process limit. \n* There is a study on the role of width and depth demonstrating how wider networks lead to more Gaussian-like distributions. \n* Authors also introduced generalized He priors that enable to tune the variance induced in the function space. \n\n\n**Limitations**: \n* Only linear and ReLU functions \n* Only Gaussian priors \n* Without the bias term \n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Suggestions**: \n* While there is a study on the depth limit, I think it is important to talk about the deep propagation papers and Edge of Chaos: Schoenholz et al. “Deep information propagation”. In addition,  Hayou et al “On the impact of the activation function on deep neural networks training” argue that smooth activation functions lead to better information propagation. Do you think it is possible to achieve similar results based on the obtained depth limit?\n* I would like to see a discussion on the limitations of the setting and the Meijer-G functions approach such as if it is possible to add a bias term or to work with other activation functions (e.g. smooth function, see the previous paragraph). \n* Lee et al. “Finite Versus Infinite Neural Networks: an Empirical Study” might be a relevant citation about the performance of finite-width networks over infinite-width counterparts \n* Aitchison. (2020) “Why bigger is not always better: on finite and infinite neural networks.” might be added to the discussion\n\n**Clarity**:\n\nThe paper is well-written and enjoyable to read despite the technicality of the results.\n\n**Significance**: \n\nI think the results are significant and will help to extend the line of research on function space description of Bayesian neural networks. \n\n**Conclusion**:\n\nThe results are obtained for a simple model and seems hard to generalize to other settings or priors but they are very solid. I recommend publication. \n \n\n\n\n**Minor**:\n\n* “In-line” -> in line \n* References: titles should be properly capitalised: Gaussian, NNGP, NTK, Meijer, Bayesian,… The conferences names should be unified. The reference of Novak et al is mentioned twice. \n\nAppendix:\n* “Based in” -> on\n* “We state and proof”, “we proof” \n* “For both distribution”\n* “Well known” -> well-known \n* “Is significantly reduce” \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "7B_Ae2yH-h8", "reviewer": "Reviewer_LLRC", "summary": "Leveraging the Meijer-G function, the authors characterise the predictive priors of fully-connected (finite-width) feedforward induced by independent standard Gaussian weight priors. They show that they can recover existing results concerning infinite width and infinite depth networks, and are also able to tackle different limiting regimes. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clearly written and the technical tools and derivations are introduced in a logical order. Although I was not intimately familiar with the Meijer-G function, I was able to follow the argument of the authors by reading the paper from start to finish without having to understand the details. I like, for example, that the authors started with the linear case, giving the readers the intuition needed to begin to understand the ReLU case.\n\nQuestions:\n1. One future direction the authors mention is extending to other architectures, in particular, conv nets. What about other activation functions? Lemma C.1 especially leverages the piecewise linear nature of ReLU activations on line 616 of the Appendix. Can you also extend this to Leaky ReLU activations? What about Heaviside step activations?\n2. Unless I missed it, I don't think there was any insight into the posterior that is induced. Is there anything you can see here?\n3. This work exclusively applies to the Bayesian setup. What breaks in the NTK setup? I'm guessing the issue arises from the fact that the NTK is no longer close to constant for finite-width. \n4. One part of the paper that confuses me is all of section 6. The authors derive the variance at the output of a finite-width network (equation (24)) by leveraging their previous results. As far as I can tell, this can be derived without the earlier parts of the paper, but perhaps I am missing something. The ReLU function is absolutely homogeneous, ReLU(|a|x) = |a|ReLU(x) for any real number a. Then, factor out the standard deviations from each of the weight matrices so that they are standard Gaussians. The result is a scalar product of all the standard deviations with a random variable with variance (1/2)^L, the same as the result obtained by the authors.\n5. Is it possible to sample from this prior directly, without using a neural network?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adds to the large and important body of work of characterising the prior in function space of a deep learning model implied by the architecture and the prior over the parameters. The paper focuses on Gaussian priors with relu and identity activation functions.

All the reviewers liked the paper. They found it clearly written and appreciated that the paper made an effort to convey the high level results despite the paper using quite heavy machinery (the Meijer-G function) to derive finite network width results. The discussion was substantial and led to a good consensus among reviewers and authors.

The paper adds substantial new insights into an important area of research in deep learning. 

Accept. :-)
Decision: Accept (Spotlight)