Exact marginal prior distributions of finite Bayesian neural networks

1 Introduction . Modern Bayesian neural networks ( BNNs ) ubiquitously employ isotropic Gaussian priors over their weights [ 1–22 ] . Despite their simplicity , these weight priors induce richly complex priors over the network ’ s outputs [ 1 , 4–22 ] . These function space priors are well-understood only in the limit of infinite hidden layer width , in which they become Gaussian [ 4–8 ] . However , these infinite networks can not flexibly adapt to represent the structure of data during inference , an ability that is key to the empirical successes of deep learning , Bayesian or otherwise [ 2 , 3 , 9–14 , 16–20 , 23 , 24 ] . As a result , elucidating how finite-width networks differ from their infinite-width cousins is an important objective for theoretical study . Progress towards this goal has been made through systematic study of the leading asymptotic corrections to the infinite-width prior [ 12 , 14–17 ] , including approaches emphasizing the physical framework of effective field theory [ 13 , 14 ] . However , the applicability of these perturbative approaches to narrow networks , particularly those with extremely narrow bottleneck layers [ 9 ] , remains unclear . In this paper , we present an alternative treatment of a simple class of BNNs , drawing inspiration from the study of exactly solvable models in physics [ 25–27 ] . Our primary contributions are as follows : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • We derive exact formulas for the priors over the output preactivations of finite fullyconnected feedforward linear or ReLU BNNs without bias terms induced by Gaussian priors over their weights ( §3 ) . We only consider the prior for a single input example , not the joint prior over the outputs for multiple input examples , as it can capture many finite-width effects [ 9 , 12 ] . Our result for the prior of a linear network is given in terms of the Meijer G-function , which is an extremely general but well-studied special function [ 28–32 ] . The prior of a ReLU network is a mixture of the priors of linear networks of narrower widths , corresponding to different numbers of active ReLUs in each layer . • We leverage our exact formulas to provide a simple characterization of finite-width network priors ( §4 ) . The fact that the priors of finite-width networks become heavy-tailed with increasing depth and decreasing width [ 21 , 22 ] , as well as the asymptotic expansions for the priors at large hidden layer widths [ 12 , 15 ] , follow as corollaries of our main results . Moreover , we show that the perturbative finite-width corrections do not capture the heavytailed nature of the true prior . To the best of our knowledge , our results constitute the first exact solutions for the priors over the outputs of finite deep BNNs . As one might expect from knowledge of even the simplest interacting systems in physics [ 25–27 ] , these solutions display many intricate , non-Gaussian properties , despite the fact that they are obtained for a somewhat simplified setting . 2 Preliminaries . In this section , we define our notation and problem setting . We use subscripts to index layer-dependent quantities . We denote the standard ` 2 inner product of two vectors a , b ∈ Rn by a · b . Depending on context , we use ‖ · ‖ to denote the ` 2 norm on vectors or the Frobenius norm on matrices . We consider a fully-connected feedforward neural network f : Rn0 → Rnd with d layers and no bias terms , defined recursively in terms of its preactivations h ` as h0 = x , ( 1 ) h ` = W ` φ ` −1 ( h ` −1 ) ( ` = 1 , . . . , d ) , ( 2 ) f = φd ( hd ) , ( 3 ) where n ` is the width of the ` -th layer ( i.e. , h ` ∈ Rn ` ) and the activation functions φ ` act elementwise [ 2 , 3 ] . Without loss of generality , we take the input activation function φ0 to be the identity . We consider linear and ReLU networks , with φ ` ( x ) = x or φ ` ( x ) = max { 0 , x } for ` = 1 , . . . , d − 1 , respectively . As we focus on the output preactivations hd , we do not impose any assumptions on the output activation function φd . We take the prior over the weight matrices to be an isotropic Gaussian distribution [ 1–14 , 16–22 ] , with [ W ` ] ij ∼ i.i.d . N ( 0 , σ2 ` ) ( 4 ) for layer-dependent variances σ2 ` . Depending on how one chooses σ ` —in particular , how it scales with the network width—this setup can account for most commonly-used neural network parameterizations [ 24 ] . In particular , one usually takes σ2 ` = ς 2 ` /n ` −1 for some width-independent ς 2 ` [ 2–8 , 12 , 24 ] . This weight prior induces a conditional Gaussian prior over the preactivations at the ` -th layer [ 4–8 ] : h ` |h ` −1 ∼ N ( 0 , σ2 ` ‖φ ` −1 ( h ` −1 ) ‖2In ` ) , ( 5 ) where the prior for the first hidden layer is conditioned on the input x , which we henceforth assume to be non-zero . Thus , the joint prior of the preactivations at all layers of the network for a given input x is of the form p ( h1 , . . . , hd |x ) = p ( hd |hd−1 ) p ( hd−1 |hd−2 ) · · · p ( h1 |x ) . ( 6 ) To perform single-sample inference of the network outputs with a likelihood function pl ( y |hd ) for some target output y = y ( x ) , one must compute the posterior p ( hd |x , y ) = pl ( y |hd , x ) pd ( hd |x ) p ( y |x ) , ( 7 ) where p ( y |x ) = ∫ dhd pl ( y |hd , x ) pd ( hd |x ) [ 1 , 4–13 , 18–20 ] . Before computing the posterior , it is therefore necessary to marginalize out the hidden layer preactivations h1 , . . . , hd−1 to obtain the prior density of the output preactivation pd ( hd |x ) . Moreover , in this framework , all information about the network ’ s inductive bias is encoded in the prior pd ( hd |x ) , as the likelihood is independent of the network architecture and the prior over the weights . This marginalization has previously been studied perturbatively in limiting cases [ 4–9 , 12 , 14 ] ; here we perform it exactly for any width . To integrate out the hidden layer preactivations , it is convenient to work with the characteristic function ϕd ( qd |x ) corresponding to the density pd ( hd |x ) . Adopting a convention for the Fourier transform such that pd ( hd |x ) = ∫ dqd ( 2π ) nd exp ( iqd · hd ) ϕd ( qd |x ) , ( 8 ) it follows from ( 5 ) that this characteristic function is given as ϕd ( qd |x ) = ∫ d−1∏ ` =1 dq ` dh ` ( 2π ) n ` exp ( d−1∑ ` =1 iq ` · h ` − 1 2 d∑ ` =1 σ2 ` ‖q ` ‖2‖φ ` −1 ( h ` −1 ) ‖2 ) . ( 9 ) We immediately observe that the characteristic function is radial , i.e. , ϕd ( qd |x ) = ϕd ( ‖qd‖ |x ) . As the inverse Fourier transform of a radial function is radial [ 33 ] , this implies that the preactivation prior is radial , i.e. , pd ( hd |x ) = pd ( ‖hd‖ |x ) . Moreover , as the prior at any given layer is separable over the neurons of that layer , we can see that pd ( hd |x ) has the property that the marginal prior distribution of some subset of k of the outputs of a network with nd > k outputs is identical to the full prior distribution of a network with k outputs . As detailed in Appendix A , these properties enable us to exploit the relationship between the Fourier transforms of radial functions and the Hankel transform , which underlies our calculational approach . 3 Exact priors of finite deep networks . Here , we present our main results for the priors of finite deep linear and ReLU networks , deferring their detailed derivations to Appendices A and B of the Supplemental Material . 3.1 Two-layer linear networks . As a warm-up , we first consider a linear network with a single hidden layer . In this case , we can easily evaluate the integral ( 9 ) to obtain the characteristic function ϕ2 ( q2 |x ) = ( 1 + κ22‖q2‖2 ) −n1/2 , ( 10 ) where we define the quantity κ2 ≡ σ1σ2‖x‖ for brevity . We can now directly evaluate the required Hankel transform to obtain the prior density ( see Appendix A.1 ) , yielding p2 ( h2 |x ) = 1 ( 4πκ22 ) n2/2 2 Γ ( n1/2 ) ( ‖h2‖ 2κ2 ) ( n1−n2 ) /2 K ( n1−n2 ) /2 ( ‖h2‖ κ2 ) , ( 11 ) where Γ is the Euler gamma function and Kν ( z ) is the modified Bessel function of the second kind of order ν [ 28–30 ] . Interestingly , we recognize this result as the distribution of the sum of n1/2 independent n2dimensional multivariate Laplace random variables with covariance matrix 2κ22In2 [ 34 ] . Moreover , we can see from the characteristic function ( 10 ) that we recover the expected Gaussian behavior at infinite width provided that κ22 ∝ 1/n1 [ 4–8 ] , as one would expect from the interpretation of this prior as a sum of i.i.d . random vectors . To our knowledge , this simple correspondence has not been previously noted in the literature , though it provides a succinct explanation of the slight heavy-tailedness of this prior distribution noted by Vladimirova et al . [ 21 , 22 ] . The fact that the function space prior is heavy-tailed at finite width is a particularly important non-Gaussian feature . These results are plotted in Figure 1 . 3.2 General deep linear networks . We now consider a general deep linear network . Deferring the details of our derivation to Appendix A , we find that the characteristic function and density of the function space preactivation prior for such a network can be expressed in terms of the Meijer G-function [ 28 , 29 ] . The Meijer G-function is an extremely general special function , of which most classical special functions are special cases . Despite its great generality , it is quite well-studied , and provides a powerful tool in the study of integral transforms [ 28–31 ] . Its standard definition , introduced by Erdélyi [ 29 ] , is as follows : Let 0 ≤ m ≤ q and 0 ≤ n ≤ p be integers , and let a1 , . . . , ap and b1 , . . . , bq be real or complex parameters such that none of ak − bj are positive integers when 1 ≤ k ≤ n and 1 ≤ j ≤ m. Then , the Meijer G-function is defined via the Mellin-Barnes integral Gm , np , q ( z ∣∣∣∣ a1 , . . . , apb1 , . . . , bq ) = 1 2πi ∫ C ds zs ∏m j=1 Γ ( bj − s ) ∏n k=1 Γ ( 1− aj + s ) ∏q j=m+1 Γ ( 1− bj + s ) ∏p k=n+1 Γ ( ak + s ) , ( 12 ) where empty products are interpreted as unity and the integration path C separates the poles of Γ ( bj − s ) from those of Γ ( 1− ak + s ) [ 28 , 29 ] . Expressing the density or characteristic function of a radial distribution in terms of the Meijer G-function is useful because one can then immediately read off its Mellin spectrum and absolute moments [ 28 , 29 ] . Moreover , one can exploit integral identities for the Meijer G-function to compute other expectations and transformations of the density [ 28–31 ] . With this definition , the characteristic function and density of the prior of a deep linear network are given as ϕlind ( qd |x ) = γdG 1 , d−1 d−1,1 ( 2d−2κ2d‖qd‖2 ∣∣∣∣ 1− n1/2 , . . . , 1− nd−1/20 ) ( 13 ) and plind ( hd |x ) = γd ( 2dπκ2d ) nd/2 Gd,00 , d ( ‖hd‖2 2dκ2d ∣∣∣∣ −0 , ( n1 − nd ) /2 , . . . , ( nd−1 − nd ) /2 ) , ( 14 ) respectively , where we define the quantities κd ≡ σ1 · · ·σd‖x‖ and γd ≡ d−1∏ ` =1 1 Γ ( n ` /2 ) ( 15 ) for brevity . Here , the horizontal dash in the upper row of arguments to Gd,00 , d indicates the absence of ‘ upper ’ arguments to the G-function , denoted by a1 , . . . , ap in ( 12 ) , because p = 0 . For d = 2 , we can use G-function identities to recover our earlier results ( 10 ) and ( 11 ) for a two-layer network ( see Appendix A ) [ 29 ] . We plot the exact density for networks of depths d = 2 , 3 , and 4 and various widths along with densities estimated from numerical sampling in Figure 1 , illustrating that our exact result displays the expected perfect agreement with experiment ( see Appendix E for details of our numerical methods ) . For any depth , the density ( 14 ) has the intriguing property that its functional form depends only on the difference between the hidden layer widths and the output dimensionality . This suggests that the priors of networks with large input and output dimensionalities but narrow intermediate bottlenecks—as would be the case for an autoencoder—will differ noticeably from those of networks with only a few outputs . However , it is challenging to visualize a distribution over more than two variables . We therefore plot the marginal prior over a single component of the output of a network with a bottleneck layer of varying width in Figure 2 . Qualitatively , the prior for a network with a narrow bottleneck layer sandwiched between two wide hidden layers is more similar to that of a uniformly narrow network than that of a wide network without a bottleneck . These observations are consistent with previous arguments that wide networks with narrow bottlenecks may possess interesting priors [ 9 , 35 ] .

Summary: This paper considers random finite-width, finite-depth fully-connected neural networks and derives exact formulas for the probability distribution for neural network preactivations, conditioned on a single input, at initialization. The cases considered include deep linear networks and deep Relu networks without biases. These prior distributions are expressable in terms of Meijer G-functions and are rotationally invariant (dependent only on the magnitude of preactivations). The exact priors are visualized for different widths and depths and compared to exact results for infinite-width networks (where the result is a Gaussian distribution) and perturbative results for large but finite-width networks (where the result can be expressed using an Edgeworth series). Notably, the exact results -- which are non-perturbative and have no constraints on depth and width -- reveal a long and heavy tail in the distribution which is not evident in the large width limit.