Exact marginal prior distributions of finite Bayesian neural networks
Authors: Jacob A Zavatone-Veth
Cengiz Pehlevan
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Bayesian neural networks
inductive bias
infinite-width limits

Abstract: Bayesian neural networks are theoretically well-understood only in the infinite-width limit, where Gaussian priors over network weights yield Gaussian priors over network outputs. Recent work has suggested that finite Bayesian networks may outperform their infinite counterparts, but their non-Gaussian output priors have been characterized only though perturbative approaches. Here, we derive exact solutions for the function space priors for individual input examples of a class of finite fully-connected feedforward Bayesian neural networks. For deep linear networks, the prior has a simple expression in terms of the Meijer $G$-function. The prior of a finite ReLU network is a mixture of the priors of linear networks of smaller widths, corresponding to different numbers of active units in each layer. Our results unify previous descriptions of finite network priors in terms of their tail decay and large-width behavior. 

Exact marginal prior distributions of finite Bayesian neural networks
Jacob A. Zavatone-Vett Veth Department of Physics Harvard University Cambridge, MA 02138
jzavatonevehh@g. hwwpeeeeetteh.....rr..aaa harvardd edu
Cengiz Pehlevan
John A. Paulson School of Engineering and Applied Sciences
Harvard University
Cambridge, MA 02138
cpehlevan@seas harvard.
Abstract
Bayesian neural networks are theoretically well-understood only in the infinite- width limit, where Gaussian priors over network weights yield Gaussian priors over network outputs. Recent work has suggested that finite Bayesian networks may outperform their infinite counterparts, but their non-Gaussian function space priors have been characterized only though perturbative approaches. Here, we derive exact solutions for the function space priors for individual input examples of a class finite fully-connected feedforward Bayesian neural networks. For deep linear networks, the prior has simple expression in terms of the Meijer G-ffunctinn. The prior of a finite ReLU network mixture of the priors of linear networks of smaller widths, corresponding to different numbers of active units in each layer. Our results unify previous descriptions of finite network priors in terms of their tail decay and large-width behavior.
Introduction
Modern Bayesian neural networks (BNNs) ubiquitously employ isotropic Gaussian priors over their weights [1-22]. Despite their simplicity, these weight priors induce richly complex priors over the network's outputs [1, 4-22]. These function space priors well-understood only in the limit of infinite hidden layer width, in which they become Gaussian [4-8]. However, these infinite networks cannot flexibly adapt represent the structure of data during inference, an ability that is key to the empirical successes of deep learning, Bayesian or otherwise [2, 3, 9-14, 16-20, 23, 24J. As result, elucidating how finite-width networks differ from their infinite-width cousins is important objective for theoretical study.
Progress towards this goal has been made through systematic study of the leading asymptotic cor- rections to the infinite-width prior [12, 14-17], including approaches emphasizing the physical framework of effective field theory [13, 14]. However, the applicability of these perturbative ap- proaches to narrow networks, particularly those with extremely narrow bottleneck layers [9], remains unclear. In this paper, present alternative treatment of simple class of BNNs, drawing inspiration from the study of exactly solvable models physics [25-27]. Our primary contributions follows:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We derive exact formulas for the priors over the output preactivations of finite fully- connected feedforward linear or ReLU BNNs without bias terms induced by Gaussian priors over their weights ($3). We only consider the prior for single input example, not the joint prior over the outputs for multiple input examples, as can capture many finite-width effects [9, 12]. Our result for the prior of a linear network is given in terms of the Meijer G-function, which is an extremely general but well-studied special function [28-32]. The prior of ReLU network is mixture of the priors of linear networks of narrower widths, corresponding to different numbers of active ReLUs in each layer.
We leverage our exact formulas to provide simple characterization of finite-width network priors ($4). The fact that the priors of Anite-width networks become heavy- with increasing depth and decreasing width [21, 22], as well as the asymptotic expansions for the priors at large hidden layer widths [12, 15], follow as corollaries of our main results. Moreover, we show that the perturbative finite-wide corrections do not capture the heavy- tailed nature of the true prior.
To the best of our knowledge, our results constitute the first exact solutions for the priors over the outputs of finite deep BNNs. As one might expect from knowledge of even the simplest interacting systems in physics [25-27], these solutions display many intricate, non-Gaussian properties, despite the fact that they are obtained for somewhat simplified setting.
2 Preliminaries
In this section, we define our notation and problem setting. We use subscripts to index layer-dependent quantities. We denote the standard lo inner product of two vectors a, b E R" by b. Depending on context, we use || || to denote the le norm on vectors or the Frobenius norm on matrices.
We consider fully-connected feedforward neural network f: IR - R" with d layers and no bias terms, defined recursively in terms of its preactivations h, as
he x.
(1)
h Wede-1(he-)) f = oa(ba).
(e=1.....dd) d),
(2)
(3)
where ne the width of the l-th layer (i.e., he E R" and the activation functions de act elementwise
[2, 3]. Without loss of generality, we take the input activation function du to be the identity. We consider linear and ReLU networks, with (r(x)) z or (x(x) = max (0, x) for & l..... respectively. As focus on the output preactivations h,, we do not impose any assumptions on the output activation function da.
We take the prior over the weight matrices to be an isotropic Gaussian distribution [1-14, 16-22],
with
[Welss itd. oZ
(4)
for layer-dependent variances of. Depending how one chooses σı-in particular, how it scales with the network width-this setup can account for most commonly-used neural network parameterizations [24]. In particular, one usually takes of 5/ /ne- for some width-independent s² [2-8, 12, 24). This weight prior induces a conditional Gaussian prior over the preactivations at the l-th layer [4-8]
h h (0, ofl|o= (ha-1))l
(5)
where the prior for the first hidden layer is conditioned on the input x, which we henceforth assume to be non-zero. Thus, the joint prior of the preactivations at all layers of the network for given input
x is of the form
ha x) p(haa hh-1))((ba-i / ha-2) -. ..ppp..ppiiii xx).
(6)
To perform single-sample inference of the network outputs with a likelihood function pr(y ha) for some target output y yx)), one must compute the posterior
( ha, x) ba x)
p(ha (x,))=
p(y
x)

---

where p(y xx) f dha (y ha, ()p x) [1, 4-13, 18-20]. Before computing the posterior, it therefore necessary marginalize out the hidden layer preactivations h,.... h obtain the prior density of the output preactivation pa x). Moreover, in this framework, all information about the network's inductive bias encoded in the prior Pa(ha x), as the likelihood is independent of the network architecture and the prior over the weights. This marginalization has previously been studied perturbatively limiting cases [4-9, 12, 14]; here we perform exactly for any width.
To integrate out the hidden layer preactivations, it is convenient to work with the characteristic function (a x) corresponding to the density Pa(hs x). Adopting a convention for the Fourier transform such that
dqu
Pa x)
exp(iqs ha)pa(qa x))
(8)
(2
follows from (5) that this characteristic function is given as
dq dhe (2 exp
pa(qu x)
II
Σ iqe he Σ
(9)
We immediately observe that the characteristic function is radial, i.e., (a(aa pa(ll x). As the inverse Fourier transform of a radial function is radial [33], this implies that the preactivation prior is radial, i.e., pa pah a x) = Pa(] ball x). Moreover, as the prior at any given layer is separable over the neurons of that layer, we can see that pa x) has the property that the marginal prior distribution of some subset of k of the outputs of network with na > k outputs is identical to the full prior distribution of a network with outputs. As detailed in Appendix A, these properties enable us to exploit the relationship between the Fourier transforms of radial functions and the Hankel transform, which underlies our calculational approach.
Exact priors of finite deep networks
Here, we present our main results for the priors of finite deep linear and ReI.U networks, deferring their detailed derivations Appendices and B of the Supplemental Material.
3.1 Two-layer linear networks
As a warm-up, we first consider linear network with single hidden layer. In this case, we can easily evaluate the integral (9) obtain the characteristic function
(2 x)
(1 + (ell
(10)
where we define the quantity = σ r
|||| for brevity. We can now directly evaluate the required Hankel transform to obtain the prior density (see Appendix A.1), yielding
2
h2el
(m1-na)/2
(a
[
Pa (he x) (4Tx3) /2 I(n1/2
K ((n - n2)/2
(11)
2k2
K2
where I is the Fuler gamma function and K K=(z) is the modified Bessel function of the second kind of order 2 [28-30].
Interestingly, we recognize this result as the distribution of the sum of n 2 independent 72-
dimensional multivariate Laplace random variables with covariance matrix 2x [34]. Moreover, we can see from the characteristic function (10) that we recover the expected Gaussian behavior infinite width provided that n² x 1/mı [4-8], as one would expect from the interpretation of this prior as a sum of i.i.d. random vectors. To our knowledge, this simple correspondence has not been previously noted in the literature, though provides a succinet explanation of the slight heavy-tailedness of this prior distribution noted by Vladimirova et al. [21, 22]. The fact that the function space prior heavy-tailed at finite width is particularly important non-Gaussian feature. These results are plotted in Figure 1.
3

---

d=2
d=3
d=4
10
1=
2
10
5
1=10
1=10
n=
10
0
1
2
3
4
5
hg
Then, Figure 1: Priors of deep linear networks of depths d = 2, 3, and 4. In each panel, the prior density plotted only for positive values of the output preactivation has as it is symmetric about zero. For each depth, all hidden layers are of the same width which is indicated by line color. The black line indicates the Gaussian infinite-width limit discussed $4.3. Thick lines show the exact priors, while thin jagged lines show experimental estimates from 10t examples. Further details on the numerical methods used to generate these figures are provided in Appendix E.
3.2 General deep linear networks
now consider a general deep linear network. Deferring the details our derivation to Appendix A, we find that the characteristic function and density of the function space preactivation prior for such a network can expressed in terms of the Meijer G-function [28, 29]. The Meijer G-function extremely general special function, of which most classical special functions special cases. Despite its great generality, it is quite well-studied, and provides powerful tool in the study of integral transforms [28-31]. Its standard definition, introduced by Erdélyi [29], is as follows: Let 0 & m I and 0 Σ S p be integers, and let a1,..., ,,,,, and b....., parameters such that none of aş by positive integers when the Meijer G-function is defined via the Mellin BarnnB integral
......., by be real or complex and
Gm.n p.9
a....... ap by
2Tí / ds 2"
1
II Th ((b - 9) IIR P(1 a, + s)
(12)
b1
ITy [(1 T11-b; + I. [(a + s))
where empty products are interpreted as unity and the integration path C separates the poles of r(b -s) -s from those of [11 ag + +) [28, 29]. Expressing the density characteristic function of radial distribution in terms of the Meijer G-function is useful because one can then immediately read off its Mellin spectrum and absolute moments [28, 29]. Moreover,
exploit integral identities for the Meijer G-function to compute other expectations and transformations of the density [28-31].
With this definition, the characteristic function and density of the prior of deep linear network given as
(qu x) = TaGldd11
nd12121|qu|1"
n1/2..... .................. - na-1/2
(13)
and
Jin Pa
(hs x) (2d m2/2
7d
Gdo
|laal
0, (m - n2)/2, (na-1 )
-na//)
(14)
2d
respectively, where we define the quantities
Ka oalxx| and
II l(ne/2)
(15)
for brevity. Here, the horizontal dash in the upper row of arguments 0d.0 of 'upper' arguments to the G-function, denoted by ai...., Ap in (12), because p 0. For d 2.
Go indicates the absence

---

10
0=4
ng
10
72 7=
0
0
-
10
-010
n,=
11000
=
10%
0
2
3
5
h4
Figure 2: Priors of depth d 4 linear networks with narrow bottlenecks. The left panel shows diagram of depth d = network with two wide hidden layers of widths n, and ng separated by narrow bottleneck of width n22 2. The right panel shows prior densities for networks of this structure with n = ng = 100 and variable bottleneck widths n2,, which is indicated by line color. The prior density is plotted only for positive values of the output preactivation ha, as it is symmetric about zero. The black line indicates the Gaussian limit in which the widths of all three hidden layers taken to infinity, discussed in $4.3. Further details on the numerical methods used generate this figure are provided in Appendix E.
we can use G-function identities to recover our earlier results (10) and (11) for a two-layer network (see Appendix A) [29]. We plot the exact density for networks of depths widths along with densities estimated from numerical sampling in Figure 1, illustratiing that our exact result displays the expected perfect agreement with experiment (see Appendix E for details of our numerical methods).
3. and 4 and various
For any depth, the density (14) has the intriguing property that its functional form depends only on the difference between the hidden layer widths and the output dimensionality. This suggests that the priors of networks with large input and output dimensionalities but narrow intermediate bottlenecks-ag would be the case for an autoencoder-will differ noticeably from those of networks with only few outputs. However, variables. We therefore plot the marginal prior over a single component of the output of a network with bottleneck layer of varying width Figure Qualitatively, the prior for a network with narrow bottleneck layer sandwiched between two wide hidden layers is more similar to that of uniformly narrow network than that of wide network without bottleneck. These observations are consistent with previous arguments that wide networks with narrow bottlenecks may possess interesting priors [9, 35].
challenging to visualize distribution over more than two
3.3
Deep ReLU networks
Finally, we consider ReLU networks. For this purpose, we adopt a more verbose notation in which the dependence of the prior on width is explicitly indicated, writing p (ha; Ka; Ri...., (d-1,nn na) for the prior density (14) of a linear network with the specified hidden layer widths. Similarly, we write P4 ReLU (ha; Kaj ........ Rd-1, na) for the prior density of the corresponding ReLU network. As shown in Appendix B, we find that
ReLU (ha; Had n1,..... nd)) Pd
(2
1)
... (2nd-1
(2nd-1-1)
5(ha)
2 tn
B1
Dd-1
111
Pa (ha; Ka; ka- (16)
2 +
ka-1
11=1
Ad-1=1
where 5(ba) is the nnddimmnsional Dirac distribution. We prove this result by induction network depth using the characteristic function corresponding to this density. The base case follows by direct integration and the binomial theorem, and the inductive step uses the fact that the linear
5

---

a
b
10
d=2
d=3
d=4
10
10
5
1=
100
=
10
2
3
4
5
h
Figure 3: The prior of deep ReLU network. (a) Schematic depiction of the ReLU prior mixture of the priors linear networks of different widths (16). Grey nodes indicate 'inactive' units, while the linear network of active units is shown by the orange nodes.
(b) ReLU prior densities for networks of depths = 2, 3, and 4 and varying width. Here, we choose Ka such that the variance of the preactivations matches that of the linear networks shown in Figure 1. In each panel, the prior density plotted only for positive values of the output preactivation ha, as it is symmetric about zero. For each depth, all hidden layers of the same width n, which indicated by line color. The black line indicates the Gaussian infinite-widthe limit discussed $4.3. Thick lines show the exact priors, while thin jagged lines show experimental estimates from 10t examples. Further details on the numerical methods used to generate these figures are provided in Appendix E.
network prior (14) is radial and has marginals equal to the priors of linear networks with fewer outputs. This result a simple interpretation: the prior for ReLU network a mixture of priors of linear networks corresponding to different numbers of active ReLU units in each hidden layer, along with Dirac distribution representing the cases in which no output units are active. As we did for linear networks, plot the exact density along with numerical estimates in Figure 3, showing perfect agreement.
Properties of these priors
Having obtained exact expressions for the priors of deep linear or ReLU networks, we briefly characterize their properties, and how those properties relate to prior analyses of finite network priors.
4.1 Moments
We first consider the moments of the output preactivation. As the prior distributions zero-centered and isotropic, it is clear that all odd raw moments vanish. However, the moments of the norm of the particular, using basic properties of the Meijer G-function [28, 29], we can easily read off the moments for a linear network as
output preactivation are non-vanishing.
m/2
Em Hallball" 22m/2 m
(m 2 0),
(17)
2

---

where a² = + b)/T() is the rising factorial [28]. This result takes particularly simple form for the even moments m 2k, which case (n/2)* 2- 1== (n 2ÿ). Most simply, for m we have Fin|| h = x² ... ndd-
Similarly, for ReL.U networks, we have
m/2
ke
m/2
ERcLU ball" 2dmm/2 Rid
(18)
2
ke
2
Each term in the product over l expands in terms of generalized hypergeometric functions evaluated at unity [28]. As for linear networks, this expression has a particularly simple form for even moments, ndd Therefore, for identical weight variances, the variance of the output preactivation of ReLU network is 2 times that of linear network of the same width and depth. However, one can compensate for this variance reduction by simply doubling the variances of the priors over the hidden layer weights.
particularly if m 2, for which ERcLU| hh/l? = 2'-2d''' 2''"kkn
Using the property that the marginal prior distribution a single component of the output is identical to the prior of single-output network, these results give the marginal absolute moments of the prior of a linear or ReLU network. Moreover, these results can also be used to obtain joint moments of different components by exploiting the fact that the prior is radial. By symmetry, the odd moments vanish, and the even moments are given up to combinatorial factors by the corresponding moments of any individual component of the preactivation. For example, the covariance of two components of the output preactivation is Eha, = (EA,,) for all i. = I,..., ndd.
4.2 Tail bounds
Vladimirova et al. [21, 22] have shown that the marginal prior distributions of the preactivations of deep networks with ReLU-like activation functions and fixed, finite widths become increasingly heavy-tailed with depth. This behavior contrasts sharply with the thin-tailed Gaussian prior of infinite-
width networks [4-8]. In particular, Vladimirova et al. [21, 22] showed that the prior distributions are sub- Weibull with optimal tail parameter f d/2, meaning that they satisfy
P(haa 2 p) < (eep((pp'))
(19)
for each neuron j E f1....,ndll n all O, and some constant C' > 0if0 2 d/2, but not if 0 < d/2. A sub- Gaussiansin distribution is sub- weibull with optimal tail parameter at most 1 1/2; distributions with larger tail parameters have increasingly heavy tails. As shown in Appendix C, we can use the results of $4.1 to give a straightforward derivation of this result, showing that the norm hall of the output preactivation for either linear or ReLU networks is beibWeibull with optimal tail parameter d/2. Due the aforementioned fact that the marginal prior for a single output of multi-output network is identical to the prior for a single-output network, this implies (19).
4.3 Asymptotic behavior
Most previous studies of the priors of deep Bayesian networks have focused on their asymptotic behavior for large hidden layer widths. Provided that one takes
Kd (n1 ... -..
11/2 11111 Xd
(20)
for Xa independent of the hidden layer widths such that the preactivation variance remains finite, the prior tends to a Gaussian as n1, Ma-1 - oc for fixed d, ny, and na [4-10, 12, 24]. This behavior is qualitatively apparent in Figures and 3. Here, we exploit our exact results to study this asymptotic regime. An ideal approach would to study the asymptotic behavior of the characteristic function (13) and apply Lévy's continuity theorem [36] to obtain the Gaussian limit, but we are not aware of suitable doubly-scaled asymptotic expansions for the Meijer G-function [28, 29]. Instead, use a multivariate Edgeworth series to obtain asymptotic expansion of the density [37]. As
7

---

10
2=2
d=3 d=4
10
n=10
10a-
n=50
n=100
1=
10
0
2
3
5
hg
Figure 4: The width Edgeworth approximation for the prior density is thin-tailed. From left to right, the panels show the priors of linear networks of depths 2, 3, and of varying widths. In each panel, solid lines show the exact prior density (14), while dashed lines show the asymptotic Edgeworth approximation (21). The exact prior density is computed numerically as described in Appendix E.
detailed in Appendix D, we find that the prior of a linear network has an Edgeworth series of the form
1
||ba|* |
Pd
lin (ha)
x) 2
(2
)
na/2
exp
2xx
1
[hall
Iaall 22
1 n²
2(na
22 + na(nn + +2) +
4
ng
x
(21)
The Edgeworth expansion of the prior of a ReLU network is of the same form, with the factor of 1/4 scaling the finite-witth correction being replaced by 5/4, and the variance xÿ re-scaled by 21-d Heuristically, this result makes sense given that the binomial sums (16) will be dominated by kg ne/2 in the larte-width limit.
These results succinetly reproduce the leading finite-width corrections formally written down by Antognini [15] and recursively computed by Yaida [12]. However, importantly, this approxi- mate distribution is sub- Goo-Gussiian: it cannot capture the depth-dependent heaviness of the tails of the true finite-width prior described in $4.2. More generally, one can see that the heavier-tham Gaussian tails of the finite width prior are an essentially nonteertur effect. At any finite order of the Edgeworth expansion, the approximate density for network of any fixed depth is of where f is a polynomial satisfying f [ch exp(-Ihl | /2 = [37]. Such a density sub- Gaussian. In Figure 4, we illustrate the discrepancy between the thin tails of the Edgeworth expansion and the heavier tails of the exact prior. Even at the relatively modest depths shown, the increasing discrepancy between the tail behavior of the approximate prior and the true tail behavior with increasing depth is clearly visible. We emphasize that low-order Edgeworth expansions will capture some qualitative features of the finite-width prior, but not all. It is therefore important consider approximation accuracy on case-by-case basis depending on what features of finite BNNs one aims to study.
the form x2 na/2
5 Related work
As previously mentioned, our work closely relates to program that proposes to study finite BNNs perturbatively by calculating asymptotic corrections the prior [12, 14]. Though these approaches are applicable to the prior over outputs for multiple input examples and to more general activation functions, they are valid only the regime of large hidden layer widths. As detailed in $4.3, these asymptotic results can be obtained as limiting case of our exact solutions, though the Edgeworth series does not capture the heavier-than- than-Gaussian tails of the true finite-width prior. similar
vein, recent works have perturbatively studied the width posterior for Gaussian likelihood
8

---

[12, 13, 38]. Our work is particularly similar in spirit to that of Schoenholz et al. [13], who considered asymptotic approximations to the partition function of the single-example function space posterior. Our exact solutions for simple models provide a broadly useful point of comparison for future perturbative study [12-15, 25-27].
As discussed in $4.2, our exact results recapitulate the observation of Vladimirova et al. [21, 22] that the prior distributions of finite networks become increasingly heavy-tailed with depth. Moreover, our results are consistent with the work of Gur-Ari and colleagues, who showed that the moments we compute should remain bounded large widths [16, 17]. Similar results on the tail behavior of deep Gaussian processes, of which BNNs are a degenerate subclass, have recently been obtained by Lu et al. [39] and by Pleiss and Cunningham [40]. Our approach complements the study of tail bounds and asymptotic moments. Exact solutions provide a finer -rraineaddd characterization of the prior; but is possible to compute tail bounds and moments for models for which the exact prior is not straightforwardy calculable. We note that, following the appearance of our work in preprint form and after the submission deadline, parallel results on exact marginal function-space priors were announced by Noci al. [41].
After the completion of our work, we became aware of the close connection of our results on deep linear network priors to previous work in random matrix theory (RMT). In the language of RMT, the marginal function- spacee prior of deep linear BNN is particular linear statistic of the product of rectangular real Ginibre matrices (i.e.. matrices with independent and identically distributed Gaussian entries) [42]. An alternative proof of the result (14) then follows by using the rotational invariance of what is known in RMT as the one-point weight function of the singular value distribution of the product matrix [43]. However, to the best of our knowledge, this connection had not previously been exploited to study the properties of finite linear BNNs. Further non-asymptotic study of random matrix products, and of nonlinear compositions as in the deep ReLU BNNs considered here, will be an interesting objective for future work [42-44].
Finally, previous works have theoretically and empirically investigated how finite-width network priors affect inference [9-11, 18-20, 23]. Some of these studies observed an intriguing phenomenon: better generalization performance is obtained when inference performed using a "cold" posterior that is artificially tempered as p(ha| x, y for 0 < T < 1 [11, 18, 19]. This contravenes the expectation that the Bayes posterior (i.e., T = 1) should be optimal. It has been suggested that this effect reflects misspecification either of the prior over the weights-namely, that a distribution other than an isotropic Gaussian should be employed -or of the likelihood [1 I]. but the true cause remains unclear [20]. The exact function space priors computed in this work should prove useful in ongoing dissections of simple models for BNN inference. Most simply, they provide finer- mrei-gr understanding of how hyperparameter choice affects the prior than that afforded by tail bounds alone [21, 22, 40]. Though we do not compute function-space posterior distributions, knowing the precise form of the prior would allow one to gain an intuitive understanding of the shape of the posterior for given likelihood [10]. For instance, one could imagine a particular degree of heavy-tailedness in the prior being optimal for a dataset that to some degree heavy-tailed. This could allow one to gain some intuition for when the prior or likelihood is misspecified for a given dataset. Detailed experimental and analytical investigation of these questions is an important objective of our future work.
6 Conclusions
In this paper, we have performed the first exact characterization of the function-space priors of finite deep Bayesian neural networks induced by Gaussian priors over their weights. These exact solutions provide useful check on the validity of perturbative studies [12, 14, 15], and unify previous descriptions of finite-widie network priors [12, 14-17, 21, 22]. Our solutions were, however, obtained for the relatively restrictive setting of the marginal prior for a single input example of a feedforward network with no bias terms. As our approach relies heavily on rotational invariance, is unclear how best to generalize these methods to networks with non-zero bias terms, or to the joint prior of the output preactivations for multiple inputs. We therefore leave detailed study of those general settings as interesting objective for future work.
9

---

Acknowledgments and Disclosure of Funding
We thank A. Atanasov, Bordelon, A. Canatar, and M. Farrell for helpful comments on our manuscript. The computations in this paper were performed using the Harvard University FAS Division of Science Research Computing Group's Cannon HPC cluster. JAZ-V acknowledges support from the NSF-Simons Center for Mathematical and Statistical Analysis of Biology at Harvard and the Harvard Quantitative Biology Initiative. CP thanks Intel, Google, and the Harvard Data Science Initiative for support. The authors declare no competing interests.
References
[1] David JC MacKay, A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448-477 1992.
[2] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT Press, Cambridge, MA, USA, 2016.
[3] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(75533:436-444 2015.
[4] Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
[5] Christopher KI Williams. Computing with infinite networks. Advances in Neural Information Processing Systems, pages 295-301, 1997.
[6] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018. URL https://openreview wtwp:/eeiiie..ne.ouuunn.. net/forum? id-H1-nGgWC-.
[7] Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as Gaussian processes. In International Conference on Learning Representations, 2018. /tt
[8] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian pro- cess behavior, gradient independence, and neural tangent kernel derivation. arXiv preprins arXiv:1902.04760. 2019.
[9] Laurence Aitchison. Why bigger is not always better: finite and infinite neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings Machine Learning Research, pages 166-164. PMLR, July 2020. URL http: http://poocceennse///ee mlr pttps///11///ttciiieeeeeeeeeeeee
html.
[10] Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv:2002.0879), 2020.
[11] Laurence Aitchison. statistical theory of cold posteriors in deep neural networks. In International Conference on Learning Representations, 2021. URL net/forrum2dd=RR133888
[12] Sho Yaida. Non-Gaussian processes and neural networks at finite widths. In Jianfeng Lu and Rachel Ward, editors, Proceedings of The First Mathematical and Scientific Machine Learning Conference, volume 107 of Proceedings of Machine Learning Research, pages 165- 192, Princeton University, Princeton, NJ, USA, July 2020. PMLR. URL http::tt///// htttt:///ppocceiings.. mlr. ppress/v107yaada220a html.
[13] Samuel Schoenholz, Jeffrey Pennington, and Jaseha Sohl-Dickstein. correspondence between random neural networks and statistical field theory. arXiv preprint arXiv: 2017.
[14] James Halverson, Anindita Maiti, and Keegan Stoner. Neural networks and quantum field theory. Machine Learning: Science and Technology, 2021.
10

---

[15] Joseph M Antognini. Finite size corrections for neural network Gaussian processes. arXiv preprint arXiv: 1908. 10030, 2019.
[16] Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from Feynman diagrams. In International Conference on Learning Representations, 2020. URL
[17] Kyle Aitken and Guy Gur-Ari. On the asymptotics of wide networks with polynomial activations. arXiv preprint arXiv:2006.0m 06687, 2020.
[18] Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In Hal Daumé III and Aarti Singh, editors, Pruceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10248-1000 10259.. PMLR, 13-18 Jul 2020. URL http:/ http:::/ppcceeddnng..e mlr. pwwpss/eee/eeeeeeeeppe 1ttes///119//eeeeee000 html.
[19] Vincent Fortuin, Adrià Garriga-Alonso, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. arXiv preprint arXiv:2102.0657/ 2021.
[20] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson What are bayesian neural network posteriors really like? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4629-4640. PMLR, 18-24 Jul 2021. URL http:/ htttp://proceedingg mlr, wwwpppessssvi..eee izmailov2iaaaaammmmmmooommmmm html.
[21] Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in Bayesian neural networks the unit level. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6458-646) 6467. PMLR, 09-15 Jun 2019. URL http: httpp://proceedings mlr. press/resseeeee /v9/s////////// htt////aammmoooaaeeeeeeeee html..
[22] Mariaa Vladimirova, Stéphane Girard, Hien Nguyen, and Julyan Arbel. Sub-Weibull dis- tributions: Generalizing sub-Gaussian and sub-Exponential properties to heavier tailed dis- tributions. Stut, (1))=311, 2020. doi: http:///oi.orr//00 http://066..rr/1000002//s1....1 1002/sta4.318. URL https:
//onlinelibrarr wiley. com hom/doi//at htt//doiaabs///// 1002/sta4...... http///a////...0///aaaaa... 318. c318 sta4.318.
[23] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. arXiv preprint arXiv:2007. 2020.
[24] Greg Yang and Edward Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:201 1. 14522, 2020.
[25] Rodney Baxter. Exactly solved models in statistical mechanics. Dover Publications, Mineola, New York, 2007.
[26] Barry M McCoy and Tai Tsun Wu. The two-dimensional Ising model. Courier Corporation, 2014.
[27] Carl M Bender and Tai Tsun Wu. Anharmonic oscillator. Physical Review, 184(5): 1969.
[28] DLMF, NIST Digital Library of Mathematical Functions. http://dllf...ist.. www.lllmm..istttgo//o. Release 1.1.1 of 2021-03-15, 2021. URL http: http://dle //dfff .ists. mist.gov/. F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.
[29] Arthur Erdélyi, Wilhelm Magnus, Fritz Oberhettinger, and Francesco G. Tricomi. Higher Transcendental Functions. Vol. I. McGraw-Hill Book Company, Inc., New York-Torotto- London, 1953. URL https: //ttt//auttttrrre waatthors.ibbaryyy.. caltech. edweedu/uaoooo //491// Reprinted by Robert E. Krieger Publishing Co. Inc., 1981. Table errata: Math. Comp. v. 65 (1996), no. 215, p. 1385, v. 41 (1983), no. 164, p. 778, 30 (1976), no. 135, p. 675, 25 (1971), no. 115, p. 635, v. 25 (1971), no. 113, p. 199, v. 24 (1970), no. 112, p. 999, v. 24 (1970), no. 110, p. 504, v. 17 (1963), no. 84, p. 485.

---

[30] Arthur Erdélyi, Wilhelm Magnus, Fritz Oberhettinger and Francesco G. Tricomi. Tables of Integral Transforms. Vol. McGraw- Hill Book Company, Inc., New York-Toronto-Londow 1954. URL https:/ aatthrrorrroo... library. caltech. edu/43489/ Table errata: Math. Comp. 66 (1997), no. 220, p. 1766-1767, v. 65 (1996), no. 215. p. 1384, v. 50 (1988), no. 182, p- 653, v. 41 (1983), no. 164, p. 778-779, V. 27 (1973), no. 122, p. 451, v. 26 (1972), no. 118, p.
599, V. 25 (1971), no. 113, p. 199, v. 24 (1970), no. 109, p. 239-240.
[31] Arthur Erdélyi, Wilhelm Magnus, Fritz Oberhettinger, and Francesco G. Tricomi. Tables of Integral Transforms. Vol. II. McGraw-Hii Book Company, Inc., New York- Toronto-London, Table errata: Math. Comp. 65 (1996), no. 215, p. 1385, 41 (1983), 164, pp. 779-780, v. 31 (1977), no. 138, p. 614, V. 31 (1977), no. 137, pp. 328-329, v. 26 (1972), no. 118, p. 599, v. 25 (1971), no. 113, p. 199,
1954. URL https:/ h /athorrs....... wwaarr..iibeee library caltech. vdu/43489/
23 (1969), no. 106, p. 468.
[32] Izrail Solomonovich Gradshteyn and Iosif Moiscevich Ryzhik. Table of integrals, series, and products. Academic Press, 2014.
[33] Elias M Stein and Guido Weiss. Introduction to Fourier Analysis Euclidean Spaces (PMS-32), Volume 32. Princeton University Press, 2016.
[34] Samuel Kotz, Tomasz Kozubowski, and Krzystof Podgorski. The Laplace distribution and generalizations: revisit with applications communications, economics, engineering, and finance. Springer Science & Business Media, 2012.
[35] Devanshu Agrawal, Theodore Papamarkou, and Jacob Hinkle. Wide neural networks with bottlenecks deep gaussian processes. Journal Machine Learning Research, 21(175):1-66, 2020. URL http: / jmlr. org/ ppppps///////0-eeep//// 007 017.html.
[36] David Pollard. A user's 's guide to measure theoretic probability, volume 8. Cambridge University Press, 2002.
[37] lb M Skovgaard. On multivariate Edgeworth expansions. International Statistical Review/Revue Internationale de Statistique, pages 169-186. 1986.
[38] Gadi Naveh, Oded Ben-David, Haim Sompolinsky, and Zohar Ringel. Predicting the outputs of finite networks trained with noisy gradients. arXiv preprint arXiv:2004.01 190, 2020.
[39] Chi-Ken Lu, Scott Cheng-Hsin Yang, Xiaoran Hao, and Patrick Shafto. Interpretable deep gaussian processes with moments. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 613-623 PMLR, 26-28 Aug 2020. URL https: http::/:///occeeinggg//
www.eeiiinnmm........mmmmm..eeee mlr. ppwss/ve08 http//101/11/// /et//bbbb.... html.
[40] Geoff Pleiss and John P Cunningham. The limitations of large width in neural networks: A deep Gaussian process perspective. arXiv preprint (xXiv:2106006529 2021.
[41] Lorenzo Noci, Gregor Bachmann, Kevin Roth, Sebastian Nowozin, and Thomas Hofmann. Precise characterization of the prior predictive distribution of deep ReLU networks. arXiv preprint arXiv:2106..6611, 2021.
[42] Boris Hanin. Random neural networks in the infinite width limit as Gaussian processes. arXiv preprint vrXiv:2107.01562 2021.
[43] Jesper R. Ipsen and Mario Kieburg. Weak commutation relations and eigenvalue statistics for products of rectangular random matrices. Phys. Rev. E, 89:032106, Mar 2014. doi: 10. 1 1103/hyysevEE.. 89 89.032106. URL http://iiink....org/g.. 032106.
1103/Physeev..85
[44] Boris Hanin and Grigoris Paouris. Non-asymptotic results for singular values of Gaussian matrix products. Geometric and Functional Analysis, 31(2):268-33 2021.
12


Reviews:
[{"review_id": "iA2rsgaaPK8", "reviewer": "Reviewer_yWcY", "summary": "This paper derives closed from expressions for the marginal distribution of full-connected ReLU neural networks, with Gaussian distributions over weights, and no biases. (Can be viewed as Bayesian NN priors or NNs at initialisation.) Importantly, this is done for finite widths. This contrasts with much previous work which assumes infinite width (-> GP). It unifies/confirms some results on finite width networks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers a worthwhile contribution, is well written, and seems technically sound (disclaimer: I have not reviewed the mathematical derivations in appendix A & B). I can see it becoming well cited and useful for future work.\n\nThe main limitation is the reduced scope considered by the paper – fully-connected networks, ReLU non-linearities, no biases, Gaussian weight distributions. Also, it’s only the marginal distribution that’s considered – not the covariance between two inputs as is often of interest in the BNN/GP literature. That said it’s a useful theoretical result in itself.\n\nWhilst the paper was good at reminding in several places this limited scope, the current title risks being misleadingly general. I’d like to suggest aligning the paper’s title with the scope, something like: “Exact conditional distributions of finite neural networks at initialisation” or “Exact marginal prior distributions of finite Bayesian neural networks”, which I think is a more honest representation of the contents of the paper. I’d be interested to get other reviewers’ opinions on this.\n\nWhilst this is a theoretical paper, it would be nice to discuss possible potential impacts of the results. Is the heavy-tailedness of deeper networks a problem? I appreciated the discussion on the cold posterior, but wasn’t clear exactly how the paper’s results contribute to that issue? Are there other consequences, e.g. ideas for initialisation schemes? \n\n\n__Minor__\n\nPerhaps denote eq 14 as p^lin_d since this is used later on?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Was done well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VPji4t5f4vE", "reviewer": "Reviewer_jHrx", "summary": "The paper calculates the function-space prior (marginally for single outputs) induced by an independent Gaussian prior on the weights for deep linear and ReLU finite NNs without biases. This prior is expressed using the ​​Meijer G-function for deep linear networks, and weighted sums of these functions for deep ReLU networks. This makes clear that at finite widths, the prior is increasingly heavy-tailed with increasing depths. Moreover, finite-width corrections are unable to capture this heavy-tailedness as it is not a perturbative effect.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality: This is the first exact characterization of the function-space prior of finite-width NN that I know of. While the expression of that prior is complicated, relating to a preexisting class of special functions is useful for theorists. “Deep Neural Networks as Gaussian Processes” Lee et al. should be cited in the introduction.\n\nQuality: The high level argument presented in the main text is reasonable, but I did not read the proof in the supplement in detail. The results seem correct and are supported by experimental evidence in the figures.\nAlthough it might be out of scope for this paper, the statistical implications of the heavy-tailed of the prior are under explored. Some experiments showing how this leads to improved performance on heavy-tailed data would be interesting. Similarly, are there any benefits to inference from knowing the exact form of the prior for finite widths (other than the improved understanding of the prior induced by the hyperparameters)?\n\nClarity: The calculation of the priors is technical but the major points are presented clearly in the main text. The figures are clear and emphasize the main points of the paper, although they are quite similar. The paper might be improved if the authors presented this information in alternate ways for different figures.\n\n- Eq. (6) has a typo: h_2 -> h_{d-2}\n- What is the dash in Eq. (14)?\n\nSignificance: This result is important and will be of interest to theorists at NeurIPS. There are many interesting follow up questions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IB4SVqF4Z_l", "reviewer": "Reviewer_XUQc", "summary": "This paper considers random finite-width, finite-depth fully-connected neural networks and derives exact formulas for the probability distribution for neural network preactivations, conditioned on a single input, at initialization. The cases considered include deep linear networks and deep Relu networks without biases. These prior distributions are expressable in terms of Meijer G-functions and are rotationally invariant (dependent only on the magnitude of preactivations). The exact priors are visualized for different widths and depths and compared to exact results for infinite-width networks (where the result is a Gaussian distribution) and perturbative results for large but finite-width networks (where the result can be expressed using an Edgeworth series). Notably, the exact results -- which are non-perturbative and have no constraints on depth and width -- reveal a long and heavy tail in the distribution which is not evident in the large width limit.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper addresses a timely research topic -- trying to theoretically characterize the prior and posterior distributions of deep neural networks. Previous work has largely used other methodology; the technical derivations in this paper (e.g. connection to Meijer G-functions) seem original, to the best of my knowledge. \n\nQuality: The quality of the work appears solid, and I could not identify any mistakes (however, I have not checked the derivations in detail, specifically the appendix).\n\nClarity: The paper is well written & paced and straightforward to read.\n\nSignificance: The relative significance of this paper is my reason for a slightly lower score. I think the paper makes a nice and publishable (in some form) contribution to the literature on this topic, but I am concerned it may be of limited interest relative to other papers publishable at NeurIPS. The contexts in which the authors are able to derive results is a bit narrow in scope: a single input (multiple input case is left for future work), no biases (this is perhaps not of greatest concern, however), and treating only the prior distribution. It is illuminating to see that the exact results display heavy tails and are qualitatively different from large width results, although similar insight has appear in previous work (Vladimirova, et al).\n\nOther comments: \n--Given that the scope of the work is limited to a single input, I would suggest a small modification to the paper title that somehow makes this clear \n--Another work that is of some relevance (since they consider the case of a single input and marginalization over intermediate variables) is https://arxiv.org/abs/1710.06570.\n--A paper with overlapping results to Matthews, et al. (Ref 5) is Lee, et al. https://arxiv.org/abs/1711.00165 and is missing from the citations.\n--Could the authors comment on the connection of their work to the literature on deep Gaussian processes?\n--Is the notation on L113 intended as written? ({a} and {b} series ranging from 1 ... p and no 1 ... q)\n--In addition to Ref 34 & 35 cited in L228, I believe Ref 10 also considered corrections to the finite-width posterior / inference.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors are transparent about the limitations. Omitting discussion of societal impact is ok for this type of paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Dc9CWTfwGQT", "reviewer": "Reviewer_ZUme", "summary": "While infinitely wide Bayesian neural networks (BNNs) hidden units distributions are studied well enough due to the Gaussian process limit, hidden units at finite widths are now under the telescope. The main result of the paper is a derivation of the exact characterization of priors in function space through Meijer G-functions. The results are obtained for Gaussian priors, ReLU and linear activation functions. \n\n**Advantages**: \n* It is the first accurate description of hidden units distributions.  \n* The results are in line with other works on the heavy-tailed nature of hidden units, non-Gaussian corrections, and bounded moments.  \n\n**Limitations**: \n* Only linear and ReLU functions, \n* Only Gaussian priors, \n* Without the bias term. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \nThe paper is original and the first to find an accurate description of hidden units distributions. There is also a concurrent work called “Precise characterization of the prior predictive distribution of deep ReLU networks” that appeared after the submission and I guess the authors have seen it already.  \n\n**Clarity**: \n\nI find the paper well-written and clear. \n\n**Conclusion**:\n\nThough the results are obtained for a simple model and it seems hard to generalize to other settings or priors, I find the work worth publishing. \n \n\n**Minor**: \n\n* The notation “output prior” seems not accurate for me. As it is about hidden units outputs, I would precise or change to one of following: “function space prior”, “prior predictive”, “hidden units prior”. \n* “Our approach complements to the study” -> without to ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper derives an exact solution for function-space prior expression (marginally for single outputs) induced by an independent Gaussian prior on the weights for deep linear and ReLU finite fully connected feedforward NNs without biases. This can be viewed as Bayesian NN priors or NNs at initialisation for finite width networks. This prior is expressed using the ​​Meijer G-function for deep linear networks, and weighted sums of these functions for deep ReLU networks. This makes clear that at finite widths, the prior is increasingly heavy-tailed with increasing depths. Moreover, finite-width corrections are unable to capture this heavy-tailedness as it is not a perturbative effect.

All reviewers point out that the paper is well-written and clear. Also work presented in the paper is solid and no noticeable issues or mistakes were identified. Reviewers mention that `this is the first exact characterization of the function-space prior of finite-width NN` (`jHrx`). It’s been noted that the technical methodology advanced using the connection to Meijer G-functions is novel in the deep learning theory community. 

There are few weaknesses pointed out. One significant one was the limitation to the single input rather than multi inputs. In this regard, reviewers `XUQc` and `yWcY` raised concern with the current title being too broad and authors agreed to modify it to more precisely describe  the contribution. The authors agreed to modify it as "Exact marginal prior distributions of finite Bayesian neural networks".  Another limitation is requirement for particular architecture (deep linear, ReLu) and weight distribution (`ZUme`, `yWcY`, `XUQc`).

Overall, the AC believes this paper is a timely,  worthwhile and significant step towards characterizing prior distribution of finite networks despite some restrictive settings. AC’s opinion is that the limitations are outweighed by the interesting theoretical outcomes (voiced also by reviewer `yWcY` and `jHrx`)  which opens up new effects (e.g. heavy-tailedness) uncapturable by perturbative corrections. There are lots of interesting questions and follow up work that this paper would ignite, thus it would be worth sharing with the broad NeurIPS audience. 

Decision: Accept (Spotlight)