Deep Kernel Posterior Learning under Infinite Variance Prior Weights
Authors: Jorge Loria
Anindya Bhadra
Venue: ICLR 2025 Poster (2025)
Keywords: Kernel methods
Deep Gaussian processes
Infinite variance priors
Deep Bayesian neural networks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Neal (1996) proved that infinitely wide shallow Bayesian neural networks (BNN) converge to Gaussian processes (GP), when the network weights have bounded prior variance. Cho & Saul (2009) provided a useful recursive formula for deep kernel processes for relating the covariance kernel of each layer to the layer immediately below. Moreover, they worked out the form of the layer-wise covariance kernel in an explicit manner for several common activation functions, including the ReLU. Subsequent works have made the connection between these two works, and provided useful results on the covariance kernel of a deep GP arising as wide limits of various deep Bayesian network architectures. However, recent works, including Aitchison et al. (2021), have highlighted that the covariance kernels obtained in this manner are deterministic and hence, precludes any possibility of representation learning, which amounts to learning a non-degenerate posterior of a random kernel given the data. To address this, they propose adding artificial noise to the kernel to retain stochasticity, and develop deep kernel Wishart and inverse Wishart processes. Nonetheless, this artificial noise injection could be critiqued in that it would not naturally emerge in a classic BNN architecture under an infinite-width limit. To address this, we show that a Bayesian deep neural network, where each layer width approaches infinity, and all network weights are elliptically distributed with infinite variance, converges to a process with $\alpha$-stable marginals in each layer that has a conditionally Gaussian representation. These conditional random covariance kernels could be recursively linked in the manner of Cho & Saul (2009), even though marginally the process exhibits stable behavior, and hence covariances are not even necessarily defined. We also provide useful generalizations of the recent results of Loría & Bhadra (2024) on shallow networks to multi-layer networks, and remedy the prohibitive computational burden of their approach. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.

Published as a conference paper at ICLR 2025
DEEP KERNEL POSTERIOR LEARNING UNDER INFI- NITE VARIANCE PRIOR WEIGHTS
Jorge Loría'
Anindya Bhadra
Department of Computer Science
Department of Statistics Purdue University West Lafayette, IN, USA bhadraEpurdue, edu
Aalto University
Espoo, Finland
jorge. wew.eeaallaaaooo....oooo
ABSTRACT
Neal (1996) proved that infinitely wide shallow Bayesian neural networks (BNN) converge to Gaussian processes (GP), when the network weights have bounded prior variance. Cho & Saul (2009) provided a useful recursive formula for deep kernel processes for relating the covariance kernel of each layer the layer imme- diately below. Moreover, they worked out the form of the layer-wise covariance kernel in an explicit manner for several common activation functions, including the ReLU. Subsequent works have made the connection between these two works, and provided useful results on the covariance kernel of a deep GP arising as wide limits of various deep Bayesian network architectures. However, recent works, including Aitchison al. (2021), have highlighted that the covariance kernels obtained in this manner are deterministic and hence, precludes any possibility of posterior a
representation learning, which amounts to learning a non-degenere random kernel given the data. To address this, they propose adding artificial noise to the kernel to retain stochasticity, and develop deep kernel Wishart and inverse Wishart processes. Nonetheless, this artificial noise injection could be critiqued in that it would not naturally emerge in a classic BNN architecture under an infinite- width limit. To address this, we show that a Bayesian deep neural network, where each layer width approaches infinity, and all network weights are elliptically dis- tributed with infinite variance, converges a process with a-stable marginals each layer that has a conditionally Gaussian representation. These conditional random covariance kernels could be recursively linked in the manner of Cho Saul (2009), even though marginally the process exhibits stable behavior, and hence covariances are not even necessarily defined. We also provide useful gen- eralizations of the recent results of Loría & Bhadra (2024) shallow networks to multi-layer networks, and remedy the prohibitive computational burden of their approach. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.
INTRODUCTION
The study of deep kernel processes the infinite-width limit of deep Bayesian neural networks can be traced back to the work of Cho & Saul (2009), and further developed by Damianou & Lawrence (2013); Wilson et al. (2016); Aitchison al. (2021), among others. This connection is built upon the foundational work of Neal (1996), who proved under the condition of bounded variance network weights, the infinite-width limit of shallow (one hidden layer) Bayesian neural network (BNN) is Gaussian process (GP), whose covariance kernel depends on the choice of the nonlinear activation function. Neal (1996) studied two bounded activations explicitly: the sign function and tanh, and worked out the covariance kernels in these cases: respectively the exponential and the squared ex- ponential covariances. Subsequently, Williams (1996) obtained explicit expressions for the kernel under an error function activation. Through standard use of what is now known as the kernel trick, these ex expressions then allow one to work on with the covariance matrix induced by the kernel at the
"Corresponding author

---

Published as a conference paper at ICLR 2025
observation points for the purpose of nonparametric estimation and prediction under GP model, rather than having to work with a possibly anfinie-dimensionn and nonlinear feature space.
The importance of the work of Cho & Saul (2009) in the deep GP literature over the early works on shallow networks is that they extend the kernel processes to deep architectures, which emerge as infinite width limits of deep BNNs under bounded variance weights in various architectures, such as simple deep feedforward BNN (Lee et al., 2018; de G. Matthews et al., 2018), or deep convolutional BNNs (Garriga-Alone Alonso et al., 2018). In fact, the tensor program of Yang (2019) shows that a limiting GP would appear under an arbitrary network architecture under bounded variance priors. The work of Cho & Saul (2009) provides an explicit recursive formula relating the kernels at each layer to the layer below, so that one can evaluate the covariance matrix of the features in layer l + 1, using only the covariance matrix of the features in layer f. However, in the case of deep Gaussian processes, the covariance kernel in each layer is deterministic (non-stochastic) and the downside is that any possibility of learning the posterior distribution of the features disappears (Aitchison, 2020), as the posterior distribution of degenerate point mass kernel is again degenerate point mass.
To address the deterministic limit, Aitchison et al. (2021) propose deep inverse Wishart processes (DIWP), which they argue approximates neural network GPs and have convenient properties for learning a variational posterior. Before introducing DIWP they also propose deep Wishart pro- cesses (DWP), which work with a noisy version of the sample covariance matrices for finite-width DNNs, the downside being a deterministic infinite-width limit. While these approaches succeed in making the covariance kernel random through artificial noise injection, so that the posterior is non-degenerate, these processes do not result naturally as an infinite-width limit as in Neal (1996) or Lee et al. (2018). Consequently, the choice of the artificial noise distribution could itself be consid- ered a somewhat subjective design parameter, even though some recommendations and desiderata for such choices do exist, such as the deterministic limiting kemel is a centrality parameter of the distribution assigned to the random kernel, which is ensured in both DIWP and DWP.
Although deep GPs emerging as the wiint limit of deep BNNs are definitely interesting in their own right, there are some limitations of Gaussian scaling limit. One is the lack of feature learning mentioned above. Second is the mean-square continuity of GPs, which means GP prior cannot model jump discontinuities or even just localized smoothness. The third limitation is one noted by Neal (1996), that in the case of multiple outputs, GPs cannot learn the covariance structure under i.i.d. weights, as the covariance is always zero under an isotropic Gaussian limit. To address these limitations we develop d-stable kernel processes as the infinite- width limit of deep BNNs with infinite prior variance at each layer. The realizations of o-stable random variables are able model jumps (Kyprianou, 2018, Section. 1.2), and their ability to learn features via a stochastic kernel is developed in this paper.
1.1 PRIOR WORKS ON INFINITE- INFINITE-WIDTH LIMITS OF BNNs UNDER INFINITE VARIANCE PRIORS AND OUR CONTRIBUTIONS IN CONTEXT
The literature on the scaling limits of infinitely wide BNNs with weights that have infinite variance priors has traditionally focused on the properties of the limiting process and less on posterior in- ference under such prior processes (analogous to kriging under GP priors). The first non- Gaussian limit result of its kind was given by Der & Lee (2005), who proved that in a single hidden layer neu- ral network the limiting process is sub- Gaussaan with stable margins, and derived the characteristic function of the process. Recent extensions to deep feedforward networks are by Peluchetti et al. (2020), Favaro al. (2023) and Lee al. (2023); and the convolutional setting by Bracale et al. (2022). To the best of our knowledge, the first computationally viable method for posterior infer- ence under these prior processes was explored by Loría & Bhadra (2024). However, their method has two major drawbacks. The first is that the method is limited to shallow networks. The second is the computational complexity that scales exponentially the input dimension. Specifically, with n training points of dimension T, the computational complexity of their method is O(n Although Loría & Bhadra (2024) provided considerable evidence in one and two dimensions that a stable process prior is beneficial when the true underlying data-generating function contains jump-type discontinuities (which arise with zero probability under a GP prior), the second drawback, i.e., the exponential complexity in I, is especially limiting, since it makes it infeasible to apply the method in all but one or two dimensions, as also evidenced by our numerical experiments presented later in this paper. An intuitive explanation for this exponential complexity is that Loría & Bhadra (2024)
2

---

Published as a conference paper at ICLR 2025
work in the feature space with sign activation function in the hidden layer, for which the resulting separating hyperplanes require enumeration over each of the I dimensions on whether the output of the activation function is marked as 1 or O. Enumerating over all points results in the exponential in complexity. Avoiding this combinatorial enumeration is of critical importance for large
To address these serious limitations, the current work eschews working in the feature space al- together, and focuses on kernel methods. Setting aside the issue of computational complexity for moment, a key observation of Loría & Bhadra (2024) is that the limiting stable process has a conditionally Gaussian representation, i.e., is a Gaussian process, conditioned on some positive a-stable variables. To see the main idea, we introduce the necessary notations on a- stable random variables at this point. Define X ~ S(a, 3) an a-stable random variable with index parameter a E (0, and skewness parameter B E [-1.1, by its characteristic function: (t) El (itxx)
exp(-It]00 [11"01111wea t)]), with w( L) tan( tan(arr/2)sign) (2)sgn(())) for / 1, (2/xr) log( (()(g()))))((() (Samorodnitsky & 'Tqqqu, 1994, p. 5), with no closed form to the density of X in general, apart from specific a. A remarkable property of a-stable variables is that the a = 2 case is Gaussian, with the usual Gaussian limit for the scaled sum established by the classical central limit theorem (CLT); but they possess infinite variance for < 2, which does not admit a Gaussian scaling limit due to an inapplicability of the classical CLT. The generalized CLT (Gnedenko & Kolmogorov, 1968), that still applies, gives the non-Gaussian scaling limit in this case, and is stated in Appendix A. Two cases of interest are: symmetric o--saable random variables, which have 3 0: and positive a-stable. which requires a 1 and / 1 and which we denote by St. Equation 5.4.6 of Uchaikin & Zolotarev (1999) states for og E (0, 1) and for all positive A one has: cxp(-X"0) = Jo cxp(->s)ps== (s)ds, where Psto (-) is the density function of a positive at--table random variable, with no closed form in general. Using = f2 ag = a/2 and the fact that 12 = 16]2 obtain for E (0, 2), a Gaussian scale mixture representation over a random scale parameter s for the symmetric a-stable characteristic function as:
and t)
exp(-111)) = / exp(-I2s)sst, (5)
(1)
This observation is exploited in Theorem 1 of Loría & Bhadra (2024), represent the posterior of the stable process as a Gaussian process posterior, conditional on mixing positive a a/2-stable variables, which results in posterior sampling analogous the GP case. However, in computing the covariance kernel of this GP, they suffer from the aforementioned combinatorial bottleneck.
Elliptical a-stable random vectors (Samorodnitsky & Taqqu, 1994, Definition 2.5.1) centered at zero have characteristic function defined similarly to the scalar case as: Φz(t) = EE(exp(it"2) exp(-(t" Lt) )taa/), where Σ is positive definite matrix, termed the shape parameter. The main advantage of zero-centered elliptical a-stable vectors is that they admit the Gaussian mixture rep- Z _ $1/2G, where _ denotes equality in distribution, S St and G (O, with S, G independent, which can be shown using a multivariate extension of Equation (1).
resentation:
What the current work does is to find a recursive formula for deep kemee processes exploiting the technique of Cho & Saul (2009) to relate the conditional stochastic covariance kernels recursive manner, resulting in a far more computationally practical procedure in the kernel space rather than one in the feature space. One must note that although the limit processes are conditionally Gaussian for each layer, they possess stable margins. Nevertheless, the conditionally Gaussian representation naturally results in a stochastic covariance kernel in each layer, whose posterior can be learned given the data, to enable data-dependent representation learning. In this way, the proposed method bypasses the need for artificial noise injection in the manner of Aitchison et al. (2021).
1.2 SUMMARY OF MAIN CONTRIBUTIONS
1. The development of a novel deep a-stable kernel process, arising as the infinite-width limit of a deep BNN under elliptical infinite variance priors on the weights at each layer. We further present conditionally Gaussian representation of the resulting process at each layer, with a recursive formula relating these conditional covariance kernels, even when covariances for the marginal processes do not exist any layer.
3

---

Published as a conference paper at ICLR 2025
2. Theoretical and numerical demonstrations that the covariance kernel in the conditionally Gaussian representation for each layer is stochastic, allowing for learning a non-degenerate posterior of the kernels and identifying clear demarcation with the deep GP literature.
3. A Markov chain Monte Carlo method for posterior sampling, which allows out-of-sample prediction at new inputs as well as uncertainty quantification of the predictions via the full posterior predictive distribution.
4. Numerical demonstrations in simulations and on benchmark UCI data sets that our method performs better in prediction than the competing methods, offers full predictive uncertainty estimates, and is considerably less time- cone-conuumin than the method of Loría & Bhadra (2024) that is limited to shallow (one hidden layer) BNNs.
DEEP a-STABLE KERNEL PROCESSES AS INFINITE-WIDHH LIMITS OF DEEP BAYESIAN NEURAL NETWORKS UNDER HEAVY-TAILED WEIGHTS
Consider a deterministic input e R' with corresponding one-dimensional output (xx). We
define an L layer feedforward deep neural network (DNN) with L- 1 hidden layers by the recursion:
M w
(s0), (
f(et1l( (x)
(1)
5(()
j=1.... 1..... Me-1: (= I.....LL 1,
(2)
ML
(xx) w 1 (x),
= Σ 1=1
(L)
where f(1) x, 9(-) is nonlinear activation function, and Mr is the width of the f-th layer. Neal (1996) proved under mild conditions, with L 2 and the weights w!?) ind. Ma M/-10/ 1), by taking the infinite-width limit M2 - oo for the last layer of this shallow network, the result- ing stochastic process is Gaussian process. Cho & Saul (2009) provided recursive formula for relating the layer-wise covariance kernels when the activation function g(-) is of the form gs( [c>0). for 5 non-negative integer, and for any number of hidden layers while tak- ing the limits M1 + "y....,M,, + co. Unlike these works, we use infinite variance prior weights at each layer and study the resulting infinite-wide limit posterior. For the first layer we take the weights Wij
(1) indep. (O, sm,t), ). where (1) iidd. S2/2" and the bias of the first layer is sim- sm.t
ply taken to be 6()) s.i.d. ~ (N 1). For the subsequent layers, we set (0) as i.i.d. (O 1) and the weights (w for i = 1I...,,,,, and j = I,..,,,,,, M2+1) 1) of each layer é > are given a distribution with infinite variance, as specified below in Theorem 1. This theorem establishes the resulting infinite-win width limit as a non- non-Gaussian process with stable marginals. Further. similar to Tsuchida al. (2019), it provides conditionally Gaussian representation. conditioned on mixing positive a/2 stable variables at each layer. The covariance kernel of this conditional GP can be de- rived explicitly under several commonly used activation functions, similarly to Cho & Saul (2009). Once this connection is established, posterior inference and prediction becomes feasible even in this infinite-variance regime.
Theorem 1. For j 1...., Me-1 and 1...., n, define: 2)0 (xxx) M.
+ EM, w fl (x)))))),
(6) where (.()) = (s 1/2 with 8 i..dd positive a/2 stable random variables for 2, 2....... L,
and without loss of generality, wh are ii.d. with zero mean and unit variance and independent of
s Then, the weights w/f) have infinite variance and as Mc + xc, the limiting distribution of
z = (x)) (0)) (1) (xx)), in the l-th layer is elliptical a-stable, with characteristic function:
(
① 1200 (t) = ccxp= (-spp-(tt ((t))(()))))))
where t = (t1,..., ta) and the n x n random matrix {() is positive definite with probability 1. As such, the limiting 2 admits the representation: n (2) (6)
8 E(0
NO, (8)
8 (11())) where
4

---

Published as a conference paper at ICLR 2025
(8) S For activation functions (r(C) 5 10115500 15>03, and wig i.i.d. standard Gaussian, the
(6)
matrix ( is recursively given as a function of Df ( and $ ((-1) by:
(0 = = [(1 + (e-1) x(x
(1 + 84 2/,
(0-11)
h )]
6/2
Jo(O
0 k.h = cos { {1+ ((-1)) ((--1)
I + +
((-1)))
((e-1) 11/2
[| [+ st
(((-1) (
(e-1)
11/
k.
k,k
for ki = I,...,R, n, = == ==2..... 2...., L; where [0) can be computed explicitly for 6 E N us-
ing Eq. (4) of Cho & Saul (2009), and I(I) xS [ X", where X [,, X, s diag(s (1)
( 11, ,1,t), with m.
i.d. 5ap m m I.......
A proof is in Appendix B. For completeness, we note that go({) and gi (C) correspond to respectively the step function and ReLU activations, for which (0) as derived by Cho & Saul (2009). This result has similarities to those previously obtained by both Lee et al. (2018) and de G. Matthews et al. (2018). However, we emphasize the key difference that in our case the kernels of (())(()))))))))))))))) at all layers are random for a < 2, conditional on scales with a positive a/2-stable distribution, although they are positive definite with probability one. The limiting kernel for the Gaussian (a + 2) case reduces to deterministic one, as in earlier works, since St 5a/2-1 converges to a degenerate point mass at with probability one. This permits an explicit characterization of deep kernel process with marginally infinite variance at each layer via a conditional deep Gaussian process. Two significant gains, when compared to Loría Bhadra (2024) are obtained: (1) there is need for exponential complexity computations in the kernel space as opposed to in the feature space, and (2) our derivations work with multi-layer BNNs, in contrast to the results of Loría & Bhadra (2024) that are limited to BNNs with single hidden layer.
0 and J1 sin(0)+((T cos(0),
The next proposition describes the posterior predictive density implied by the probabilistic structure in Theorem 1. The posterior predictive density we present combines the simplicity of predicting in GPs while ameliorating the challenges posed by variables that have infinite variance.
Proposition 2. Consider a vector y of real-valued observations with corresponding inputs [X1, [xi...., xn)²", where E R!! for all Consider the model of Equations (2) and (3) and suppose we observe data with an additive error: yk- (x+) + Ek where itd
~ N(00, Then, under the
setting of Theorem 1, and taking Mt + 00 for all l, m new inputs X [x++1. xn+ml
the posterior predictive distribution of the observations y" is given by:
y" y, X, X', (1 (( ~N ( where,
44 ((n+))((nmm) A
A (n++))(444)
A ((+11)(nnm),1nm A Imm in A A (n+))))))))
( iid. st St/2i m.
(0) iid. Sapz for t 2,..., L; m m= l.......;
8
and A (() + I mm where SXL) is the matrix obtained in the last layer in Theorem I with the inputs, in order, X'I and In denotes the identity matrix of size (n + m).
Proof of Proposition is in Appendix C. This proposition enables computation of the kernel poste- rior and prediction, and is key to the numerical demonstrations presented later. We also remark that the purpose of Proposition 2 is to tackle regression tasks rather than classification. However, similar results can also be obtained for classification. Note that for a 2, the conditional kernel A is stochastic matrix, which we verify numerically in Sections 3 and 4.
Making use of Proposition 2, the next algorithm provides Markov chain Monte Carlo (MCMC) procedure for sampling from the posterior predictive density p(y", (0(0 Algorithm 1 obtains valid sample of the posterior density p(y* y, X, X*) 2: S(") x,,,
by classic results in the MCMC literature (Chib & Greenberg, 1995). Our algorithm also provides valid samples of the posterior stochastic covariance matrix induced by the kernel at the design points. We remark that simulating in previously unseen locations can be done in an off-line manner, by simply utilizing the scales and the collection of input points, which permits efficient prediction algorithm without
5

---

Published as a conference paper at ICLR 2025
Algorithm A Metropolis- sampler for the posterior predictive distribution of the deep a-kernel process.
Require: Observations E RR,, with T-dimensional input variables X E [R*x new input variables X E R"*x, and number of MCMC iterations T.
Output: Posterior predictive samples {y? Jam samples of stochastic matrix {Σ)
12(")7, 17=1-
Simulate starting scales ( +.(0): +.()))
(1) Ltd. 5ā,2 for l = 2...., L using the algorithm of Cham-
bers et al. (1976).
for = 1, ,..., do
Simulate ((1)
+.(1) (0
(t (t-1))222 using Algorithm 2.
for {= 2, 2,.,,, L do
Simulate 8 (2)
+.(1)) ly. {a (K) +.lt- k+t using Algorithm 2.
end for Simulate $4.(0) using Algorithm 3.
Compute p4ī and A using 2 in Proposition 2.
Simulate y? (y, () Nmi,,
end for return [y: and (At)T=1-
need to resample the scales. We also make use of a half-Cauchy prior (Gelman, 2006) on the variance the errors o², to specify fully-Bayesian model. A full implementation, with examples, is freely available at: https:::...... /ttttas:/giiee /ttp//ggtttt//t///// com/ aoriaa//daaieoo wwwpaa/deapppaaaaaaaaaaaaaaaaaaa kernel.
Following the construction of Theorem ensures we have defined valid kernel process that is con- sistent under marginalization, in the manner described by Aitchison et al. (2021). Specifically, the T/=1 process { is a deep kernel process with probability 1. meaning that they induce a distribu- tion on positive definite matrices of different sizes which are consistent under marginalization. The proof of this follows from a similar argument as in Aitchison et al. (2021), and reduces to their result under fixed scales. For completeness we provide further elaboration in Appendix E.
Additionally, the process ( has implicit dependence a, the index of the a-stable random variables. For a = 2, the characteristic function of 2 becomes that of Gaussian random variable, and ()) deterministic. Remarkably this kernel process is stochastic when a 2. The importance of having a stochastic deep kernel process is that it allows us to learn features, which deep Gaussian processes are not able do, highlighted by Yang et (2023). We formalize this notion in the following proposition, with proof in Appendix F.
Proposition 3. For < 2 the posterior distribution of the features Zi is given by
p(250 xx,)) f (6 (E. S(1 M2 X,y)
II
ds II
m=1
This implies that the features also depend on the observations. For a = 2, the posterior distribution of the features are independent of the observations, and hence, cannot be learned from the data.
The importance of this proposition is that we are able to verify that the deep kernel process we have defined, the features do depend on the observations. As such, there is feature learning only when a < 2, but when a = 2 we return to a deterministic kernel and are not able to learn the features. As an additional benefit, the proposition provides straightforward procedure for sampling from the posterior of the features. Namely, we can sample the scales from p( (x ( H S X,y), and then sample from 2, (111)) s()) (1 (40 (()))))))))))))))) using the posterior samples of (0)

---

Published as a conference paper at ICLR 2025
Mattral Information
Cootioora
10 Distance
Figure 1: Decay of the conditional mutual information for the deep a-kernel process as a function of the distance between the inputs with L = 2, ô = 1. The limiting Gaussian case (a = 2) is also included.
3 NUMERICAL EXPERIMENTS
3.1 CONDITIONAL MUTUAL INFORMATION AS A FUNCTION OF
Figure displays the conditional mutual information for different values of a at varying distances over uniform grid in one-dimension with L 2, 8 for the proposed deep a-kernel process, ie., a process with one hidden layer. For a GP, the mutual information is simply function of the correlation, but this quantity is not well defined in case, since there is well-defined covari- ance. As such, we need to refer to the conditional mutual information (Cover Thomas, 2006, p- 23), which can be computed using the conditionally-Gaasiic representation and numerical integra- tion via Monte Carlo methods. The expressions of conditional mutual information and details of computation are provided in Appendix G. The key finding that the conditional mutual information decays at a slower rate for smaller a. The long memory behavior of the conditional mutual infor- mation suggests the developed deep a-kernel processes be especially adept at picking up distant relationships, which would be modeled as nearly independent under GP.
3.2 POSTERIOR PREDICTION AND UNCERTAINTY QUANTIFICATION UNDER DISCONTINUITY
We display results on functions with jumps in one, two, and ten dimensions. The reason we focus specifically discontinuous true functions is that they are misspecified under GP prior under any covariance function, which can model varying levels of smoothness, but not lack of mean square continuity. In contrast, heavy-tailed priors enable posterior inference on a broader function class, e.g., those belonging to Besov space; see recent theoretical works by Agapiou & Castillo (2024) and extensive numerical works Loría Bhadra (2024). The comparisons include a Bayesian GP method (Gramacy & Taddy, 2010) and a frequentist approach on GPs (Dancik & Dorman, 2008), as well as the DIWP and NNGP method as implemented by Aitchison al. (2021), and the Stable method by Loría & Bhadra (2024). We refer our proposed method Da-KI
shorthand for deep a-stable kernel process. For the deep kernel methods Aitchison et al. (2021) we train all the models using the same hyperparameters they use, with 8000 total steps with 10-2 as the step-size for the first half, and 10-3 for the second half. For the Da-KP method we run 3000 MCMC simulations with a 8 1. For fair comparison, the three kernel methods all use are shallow with one hidden layer. The following simulation settings are considered, respectively in one, two, and dimensions, all under discontinuous truth.
meaning they
1. The true function in one dimension is f({) 5 x 1 (E>0)) and generate observations as v(E) = f{) + E; E (/ 0.52). For training consider 40 equally-spacee input points on 1, and predict on 100 out-of-sample points.
2. two dimensions we use the function f(E,, E2) = x 14s, >o) +5 1 1/g >0) and generate v(E1, = f() + E E [0, 0.52), using a x 7 uniform grid on [-1, 1]2 for training, and similar 9 grid for testing.

---

Published as a conference paper at ICLR 2025
3. In the ten-dimensional setting 6sign({1 + +-5) ) +6sign(E6 + + (Sign({s + + 6sign((10) and generate u(E) = f with ~ (O, 0.5*). The design points are generated by & vid. Unif(-0.5, 0.5). We generate 20 splits, each with 300 training and 300 testing observations.
the function f(E) 6sign(E1) Ssign({2 + Ex) +
Under these simulation settings, we provide in Table I the prediction errors for the competing meth- ods by presenting the root mean squared error (RMSE) and mean absolute error (MAE). Our method obtains smaller out-of-sample prediction errors compared to the other methods, and is generally a close second to the Stable method of Loría & Bhadra (2024). However, the running time of our method is a fraction of the time of Loría & Bhadra (2024), which we highlight in Supplementary Section H. As a result of this high run time owing to the exponential complexity computations, the results for the Stable method are only available in one and two dimensions. Figure 2a compares the function fit for the one-dimensions function. Figure 2b displays the uncertainty quantification (UQ) by presenting 90% posterior predictive intervals, where the benefits of the methods using stable priors over GP priors are clear under discontinuity, as is the inability of GP to model such functions. We display similar figures for the examples in two and ten dimensions in Appendix H. Furthermore, the out-of-sample prediction errors for this case demonstrate the superior performance of methods that incorporate a-stable random variables. Appendix H.2 shows the percent of observa- tions that are covered in the 90% posterior predictive intervals for the simulation settings under all methods. Appendix H.3 we present additional results, including an ablation study on the param- eter (Appendix H.3.3) and visualizations of the posterior quantiles of the random kernel matrix (Appendix H.3.4).
dltt
4ytu
Baya
Method
Method
MLE
deha KF
D1M 7
Bays
XNGP
-2
ayden EP
plup
NXG2
Sa
NNGP
Stusio
(a) Function for the different methods.
(b) 90% posterior predictive intervals the Bayesian methods.
Figure 2: Function fit and uncertainty quantification for the competing methods for 1-d function with single jump.
Table 1:
Out-of-sample errors in numerical examples, in twenty different splits. Best in bold. Stable not available for more than 2 dimensions.
One Dimension
Two Dimensions
Ten Dimensions
Method
RMSE (SD)
MAE (SD)
RMSE (SD)
MAE (SD)
RMSE (SD)
MAE (SD)
Da-KP
0.57 (0.05)
0.45 (0.04)
0.86 (0.09)
0.67 (0.07)
8.08 (0.38)
6.48 (0.33)
DIWP
1.08 (0.04)
0.84 (0.03)
1.69 (0.05)
1.36 (0.05)
8.92 (0.38)
7.21 (0.32)
GP Bayes
0.69 (0.05)
0.52 (0.04)
0.90 (0.06)
0.70 (0.06)
10.39 (0.63)
8.32 (0.52)
GP MLE
0.77 (0.06)
0.57 (0.04)
1.19 (0.08)
0.92 (0.07)
8.32 (0.38)
6.68 (0.34)
NNGP
1.08 (0.04)
0.84 (0.03)
1.69 (0.05)
1.36 (0.04)
8.92 (0.39)
7.21 (0.34)
Stable
0.52 (0.03)
0.42 (0.03)
0.57 (0.08)
0.45 (0.04)
3.3 THE EFFECT OF DEPTH ON PREDICTION FOR a-KERNEL PROCESSES
We numerically investigate the effect of depth on predictive performance in the synthetic data sets of Section 3.2, with L = 2 results given there. As can be seen from Table 2, there is no meaningful difference between deep and shallow kernel processes for these simulation settings. Recent works
8

---

Published as a conference paper at ICLR 2025
have theoretically investigated fundamental barriers to trainable depth of deep neural networks, exceeding which actually results in degraded performance (Schoenholz et al., 2017). A similar theoretical investigation is beyond the scope of the current work, but conjecture that the resultant stable process can capture a function class rich enough with even one hidden layer, so that the effect of depth becomes a secondary issue for these test functions. This is a very different situation from a deep NNGP, which can only capture functions that are mean square continuous, with the effect of depth primarily manifesting richer covariance kernel, but not in a non-Gaussian process.
Table 2:
Average (SD) out- sample errors for deep a-KP, with 5 = 1, a = 1, in the three simula- tion scenarios with varying layer depth, over twenty different splits.
One Dimension
Two Dimensions
Ten Dimensions
L
RMSE (SD)
MAE (SD)
RMSE (SD)
MAE (SD)
RMSE (SD)
MAE (SD)
0.56 (0.05)
0.45 (0.04)
0.82 (0.07)
0.65 (0.05)
8.09 (0.38)
6.49 (0.34)
0.57 (0.05)
0.45 (0.04)
0.81 (0.07)
0.64 (0.05)
8.10 (0.39)
6.49 (0.34)
11
0.58 (0.05)
0.45 (0.04)
0.85 (0.06)
(0.05)
8.11 (0.40)
6.51 (0.35)
16
0.58 (0.05)
0.46 (0.04)
0.88 (0.06)
0.69 (0.05)
8.11 (0.39)
6.50 (0.34)
4
APPLICATIONS TO UCI DATA SETS
4.1 OUT-OF-SAMPLE PREDICTIVE PERFORMANCE
We apply the five methods (Da-KP, DIWP, NNGP, GP Bayes, and GP MLE) to the three well-known data sets from the UCI repository: Boston, Yacht, and Energy. Due the size of these data sets and the On computational complexity, the Stable procedure was not feasible to run. To this end, we split each of the data sets in 20 different folds, training in 19 and testing in the remaining fold; and repeat the process for each of the folds. We display the average RMSE and average MAE in Table 3, with the respective standard deviations. We find that in two out of three cases (Energy and Yacht), Da-KP has the best predictive performance, while being close second on the Boston data set. In Appendix I.1 we display the running times for the considered methods, and in Appendix I.2 display the coverage of the credible intervals.
Table 3: Comparison of -f-mple errors in twenty splits of the three UCI datasets, using the
Da-KP (a = 1, 1, L. = 2). DIWP, NNGP (Aitchison et al., 2021), GP Bayes, and GP MLE. Number of observations denoted by n and number of inputs by T. Stable method not available for I> >>> 2. Best in bold.
Boston (n 506, 1 = 13)
Energy (n 769, 7 = 8)
Yacht (n 308, 7 = 6) MAE (SD)
Method
RMSE (SD)
MAE (SD)
RMSE (SD)
MAE (SD)
RMSE (SD)
Da-KP
2.59 (0.73)
1.78 (0.35)
0.46 (0.07)
0.32 (0.05)
0.31 (0.12)
0.16 (0.05)
DIWP
2.85 (0.89)
2.01 (0.41)
0.48 (0.06)
0.34 (0.04)
0.60 (0.20)
0.30 (0.10)
NNGP
3.00 (0.87)
2.04 (0.43)
2.18 (0.23)
1.57 (0.18)
3.88 (0.87)
2.58 (0.49)
GP Bayes
2.58 (0.75)
1.76 (0.35)
0.68 (0.06)
0.51 (0.04)
0.48 (0.23)
0.22 (0.07)
GP MLE
3.93 (1.02)
2.58 (0.51)
0.49 (0.06)
0.34 (0.04)
0.52 (0.34)
0.25 (0.11)
Stable
4.2 NON-GAUSSIAN FEATURE LEARNING
In this section we numerically investigate whether Da-KP indeed learns a non-degenerate and non- Gaussian posterior of the features. Specifically, we display in Figure 3 the first two features in the hidden layer of a two layer Da-KP for the Energy data set. The posterior of features in the f-th layer is given by: z (f) E(A) y -V (()). Note that after marginalizing over [(), which has stable dependence, Proposition indicates we would obtain a non- non-Gaussian distribution. Figure confirms the non-Gaussia In this figure we include the 2-d heat map of posterior samples with

---

Published as a conference paper at ICLR 2025
1-d marginal box plots, which show far more values in the tails than what be expected under Gaussianity. The normal q-q plots also confirm the heavy tails. We include similar figures for the Boston and Yacht datasets in Appendix J. The non-Gaussianity of 2j also serves confirmation that () indeed stochastic, since otherwise we would obtain Gaussian features.
(a) Scatterplot of the first two features, with box (b) Normal 9-9 plot of the first two features. plots for the marginals.
Figure 3:
Posterior distribution of the features for the Energy data set using 10k posterior MCMC samples with Ik burn- in samples.
CONCLUDING REMARKS
We present a viable posterior inference procedure for deep o-stable kernel processes arising as the infinite-width limit of a BNN under infinite variance prior weights each layer. Through convenient representation of elliptical a-stable random vectors as a Gaussian mixture with respect to positive a/2 stable variables acting as random scales, we obtain an explicit relationship with deep conditionally Gaussian process, upon which the result of Cho & Saul (2009) can be leveraged to link the conditional covariance kernels a recursive manner, resulting in computationally viable method. We also present extensive evidence of benefits in prediction and uncertainty quantification under a true data generating function that contains jump-type discontinuities. Compared to the approach of Loría & Bhadra (2024), the current work overcomes the serious limitations of being restricted to one hidden layer and exponential complexity computations.
Several future directions could naturally follow from the current work. Although MCMC is gold standard for full posterior uncertainty quantification, for computational scalability, one may look for variational approximation to the posterior predictive density derived. Applying techniques such as inducing points (Snelson & Ghahramani, 2005), would also help scale the algorithm to data sets with larger sample sizes.
Our results indicate successful feature learning under non-Gaussian stochastic processes that is not possible under GP (shallow deep) and achieves this without any artificial noise injection in the manner of Aitchison et al. (2021). In the current work, we have not made any efforts of feature selection, which appears to be a promising direction under the infinite-width limits of suitable spar- sity inducing architectures, such as dropout (Srivastava et al., 2014), that should now be possible to study following our work and leveraging its main ideas. More general activation functions, as in Tsuchida et al. (2021), may also be considered.
Although in this paper we consider the infinite- width limit of deep Bayesian neural networks, an- other interesting GP limit arises due to the dynamics of the stochastic gradient descent (SGD) noise during the training of the network, resulting in the neural tangent kernel (NTK) (Jacot et al., 2018), under the assumption of bounded variance SGD noise. However, recent works, including Simsekli et al. (2019), have found empirical evidence that SGD noise can be heavy- In this regime, one should expect non- non-Gaussian analog for the NTK, with random kernel, whose data-dependent posterior should be possible to study using techniques similar to ours.
ACKNOWLEDGMENTS
Bhadra was supported by U.S. National Science Foundation Grant DMS-201437
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sergios Agapiou and Ismaël Castillo. Heavy-tailed Bayesian nonparametric adaptation. Annals of Statistics, to appear, 2024.
Laurence Aitchison. Why bigger is not always better: finite and infinite neural networks. In International Conference on Machine Learning, pp. 156-164. PMLR, 2020.
Laurence Aitchison, Adam Yang, and Sebastian W Ober. Deep kernel processes. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 130-140 140. PMLR, 18-24 Jul 2021. URL https: http///rrceeeng...ceee///. mlr. wrwppprssss..... v1 htt.////aattttttaaaa.aaaaaa html.
Patrick. Billingsley. Probability and measure. Wiley series in probability and mathematical statis- tics. Wiley, New York, NY, 3rd ed. edition, 1995. ISBN 0-471-00710-2.
Daniele Bracale, Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Infinite-channel deep stable convolutional neural networks, 2022.
John M Chambers, Colin L Mallows, and B W Stuck. A method for simulating stable random variables. Journal of the American Statistical Association, 71(354):340-344, 1976.
Siddhartha Chib and Edward Greenberg. Understanding the Metropolis-Hastingg algorithm. The American Statistician, 49(4):327-333 1995. ISSN 00031305. URL http: /www. jstor. org/staale/22884
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.). Advances in Neu- ral Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL. https: http:///oooeeennneeeeee/e neuriiiggnnnnnunooee cc cccpaper_rpceceeeceeeeeeeeeeeeee htae//fles/paprr////// file/ 7751ec33ea4feebbbrr...
- pdf.
M. Cover and Joy A. Thomas. Elements of information theory. Wiley-Interscience,, Hoboken, NJJ, 2nd ed. edition, 2006. ISBN 1-118-58577-1.
Andreas Damianou and Neil D Lawrence. Deep Gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Garrett M Dancik and Karin S Dorman. mlegp: statistical analysis for computer models of biological systems using R. Bioinformatics, (4(17):1966, 2008.
A. P. Dawid. Some matrix-variate distribution theory: Notational considerations and a Bayesian application. Biometrika, 681))265-227, 1981. ISSN 00063444. URL http:htp////. /www jstor. ozg/stable/233582
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahra- mani. Gaussian process behaviour in wide deep neural networks. In International Confer- ence on Learning Representations, 2018. URL https: /ttp://////peeeeee.././ net forum? did= H1 -1-nGgWC-.
Ricky Der and Daniel Lee. Beyond Gaussian processes: On the distributions of infinite networks. In Proceedings of the 18th International Conference on Neural Information Processing Systems, 05, pp. 275-282, Cambridge, MA. USA, 2005. MIT Press.
Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Deep stable neural networks: Large- width asymptotics and convergence rates. Bernoulli, 29(3):2574 - 2597, 2023. doi: 10.3150/ BEJ1553.
22-BEJ1553. URL https: aodiioog//o000 -3.50222BB111553
Adrià Garria-AAlono, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net- works as shallow Gaussian processes. In International Conference on Learning Representations, 2018.
Andrew Gelman. Prior distributions for variance parameters in hierarchical models (comment ticle by Browne and Draper). Bayesian Analysis, 133):515- 5344, 2006. doi: 10. 1211////466-11 BA1I URL https:/ hos:://daa //o.i .or./orgg/1/ 1

---

Published as a conference paper at ICLR 2025
Boris Vladimirovich Gnedenko and Andrei Nikolaevich Kolmogorov. Limit distributions for sums of independent random variables, volume 2420. Addison dadison-wesley, 1968.
Robert B. Gramacy and Matthew Taddy. Categorical inputs, sensitivity analysis, optimization and importance tempering with tgp version 2, an R package for treed Gaussian process mod- doi: 10.18637/jss.0333.000 URL
els. Journal of Statistical Software, 33(6): 2010.
https: /wwww//.../ www.ssstttt..ooo org
.rg. v3/3/333////////// 106/.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen-
cralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Andreas E Kyprianou. Stable Lévy processes, self-similarity and the unit ball. ALEA. Latin Ameri- can Journal of Probability & Mathematical Statistics, 15, 2018.
Hoil Lee, Fadhel Ayed, Paul Jung. Juho Lee, Hongseok Yang, and François Caron. Deep neu- ral networks with dependent weights: Gaussian process mixture limit, heavy tails, sparsity and compressibility. Journal of Machine Learning Research, 24(289):1-78, 2023.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks Gaussian processes, In International Confer- ence on Learning Representations, 2018. URL httph:tt/////...tt https///pppeeeeeeeeee net/ forum? idd= BIEA-M-O2.
Jorge Loría and Anindya Bhadra. Posterior inference on shallow infinitely wide Bayesian neu- ral networks under weights with unbounded variance. In 40th Conference on Uncertainty in Artificial Intelligence (UAI 2024), 2024. URL https:/ J9 397bdMR7Lv.
/ net www.:reeiwww.eeeooommmmmmm forum?idd
Radford M Neal. Priors for infinite networks. Bayesian Learning for Neural Networks, Lecture Notes in Statistics, pp. 29-53. Springer New York, New York, NY, 1996. ISBN 0387947248.
Stefano Peluchetti, Stefano Favaro, and Sandra Fortini. Stable behaviour of infinitely wide deep neural networks. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1137-1146. PMLR, 26-28 Aug 2020. URL https:// proceedings .mlr. wwwps/eessvvieeee vio8 /ac/0///eelle pelpchetti22bbb html.
G. Samorodnitsky and Murad. Taqqu. Stable non-Gaussian random processes: stochastic mod- els with infinite variance. Stochastic modeling. Chapman & Hall, New York, 1994. ISBN 0412051710.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dicksteri Deep information propagation. In International Conference on Learning Representations, 2017.
Umut Simsekli, Levent Sagun, and Mert Gurbuzhalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pp. 5827-5837. PMLR, 2019.
Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Ad- vances Neural Information Processing Systems, volume 18, 2005.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, (5(1):199-1995. 2014.
Russell Tsuchida, Fred Roosta, and Marcus Gallagher. Richer priors for infinitely wide multi-layer perceptrons. arXiv preprint arXiv: 191 1.1297, 2019.
Russell Tsuchida, Tim Pearce, Chris van der Heide, Fred Roosta, and Marcus Gallagher. Avoiding kernel fixed points: Computing with ELU and GELU infinite networks. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9967-9977, 2021.
12

---

Published as a conference paper at ICLR 2025
Vladimir V. Uchaikin and Vladimir M. Zolotarev.
Chance and Stability. DE GRUYTER, URL httppa///aci.rrg/.....o
December 1999. doi: 10.1515/9788110933999
9783110935974.
Christopher Williams. Computing with infinite networks. In M.C. Mozer, M. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL https: / http://proceeeeineseeeeeeeeeeeee
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernll learning. In Arthur Gretton and Christian Robert (eds.), Proceedings of the 19th Interna- tional Conference Artificial Intelligence and Statistics, volume 51 of Proceedings Ma- chine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 11 May 2016. PMLR. URL https: / //proceedings.. ml press v55 /wi ssnn 6 html.
Adam X. Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives a deep generalisation of kernel methods. In 40th International Conference on Machine Learning, 2023. URL https: httttpp://poocedinggg. mlr. 2202 agn.//..ng2kk.. html.
Greg Yang. Tensor programs I: wide feedforward recurrett neural networks of any architecture are Gaussian processes. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. URL http: / /ariivv. httg/bb////////egg///gg///aaa 1910. 12478.
13

---

Published as a conference paper at ICLR 2025
A THE GENERALIZED CENTRAL LIMIT THEOREM
(Uchaikin & Zolotarev, 1999, p. 62) Let X1..... Xn be independent and identically distributed random variables with the distribution function Fx (2) satisfying the conditions:
Theorem 4.
-x) cx
0c,
Fx(x) ~ dr I + 00,
with μr > 0. 0, O. Then, there exists sequences and bn such that the distribution of the centered and normalized sum Zn = bn 2 Xi - an), converges in distribution to the stable distribution with parameters:
t=mmnn (1,, 2), B
C d'
as n - 0o, i.e., Fz. (x) + G(x; 3) for all x where G is continuous, where Fz,, is the c.d.f of Zeo and G is the c.d.f. of an a-stable random variable with symmetry parameter B. The coefficients On and by are given in Table 4.
Table 4: Centering and normalizing coefficients a, and b, for the generalized central limit theorem.
an
bn
0 0< <
μ
0
</ // (2) (a) sin 1/an))))))nn
p=
(c
d)n log(n)
+cc djn/2
1 1<H <
(c+d)\/((2((1) siaii)ini((((//// -
2
nE ]
log(n))) (c+ d) (n ()
2
nE
x
[(1/2)Var (X)] [2)
B PROOF OF THEOREM
First, the marginal variance of w unbounded:
V(w!f) v[(+4) ( E $491 + Elss
1 oc,
since for Stable (a/2) random variables, the expectation is unbounded. Next, we have:
1 M
Mi
2 x
(xx) = (s (2())/22
i f1 (xx).
M.
M
Since aif are i.dd with unit variance by assumption, have, as Me- - 00:
Mr
Σ
(! (xx) 3 N00,
M
by an application of the classical central limit theorem, where 2 denotes convergence in distribution. Thus,
(xxx D (s (=10))/2N(00
( which which is an a-stable random variable for each j, k. and x = v(J (xx) {8 (11-- 1
does not depend on since the s! (xx) are i.i.d. across
The conditional covariance between two features in layer one is given by:
Cov(z)" (xx). 23 (xx)
s())
Cov
Σ
(1)
1/2 ty
Im,ks Ks Σ (5 1/2 fm
Tmm (!!)
my
m/==1
1/2
Vem Σ
(1) 1/2
xm, x Cor(w))) (0 (s())
m'
m'=l
m=1
Σ
(1)
my
14

---

Published as a conference paper at ICLR 2025
As such the conditional covariance matrix 2j first covariance matrix, proceed by induction over é, with the note that in the case E = 2 what we denote by ( corresponds to s Next we obtain the characteristic function of 2) for a finite
(1) | is Σ(1) XS
x Now that we have the
Mr. assuming that 1) 8+ (É-1) are given. To this end, for t (I1...., (n) E R
f ( 10 , -0- (t) = E
(6) exp it 2
((--1)
+ 8
(E) ((-1)
Me
M 1/2 wig 51 (xx()) Ix(--)) s/. ,,8 (4-11)
(6-1)
E
exp
km 1
12
Mr
bk Mx 1/2 (1)
1. (xx))
E((e_1) off + ((--1)
E
exp
Mz
E
exp
i(5 (0) 1/2 M 1/2
Σ
w
(
tass (x (xx)
| $10 ((-11)
1=1
k=1
Mr
(1) i(8 1/) M
1/2
(1) (8)
[((-1) (0) (e-11)
E
exp
1
(4)
where we define A/0 Σk= 1x ( (xx). which independent and identically distributed over i, and the expectation E(-) is over the distribution of wn! Next, the expectation of wij zero since both factors of the product are independent and the mean of one of them is zero, and the variance is EE (x (xx))² ({--)), (.-"))
(=()) (0) is
Taking the limit M₁- oo of the expression inside the exponential in Equation (4), we obtain:
lim (54 1/2, M1
(8) 1/2
Mr
0 2 (6 (610)12 0,
(
4ns (xa))
11 (e-1)
1, 84
= (6 x 115 (0)
since s(0) and ij are independent and n (8) denotes the normal random variable on the RHS of
first line of the above display, which can be seen to be i.i.d. over j. Define the matrix Σ(0 with
I ElA! fxx (xx) [(--)), s (f-1), which is a valid covariance matrix and does not depend on Thus, since ( a deterministic function of (e-1)) and 8 (f-1), we have:
lim & f piu (t) = exp { (1/2)((""))))))))) M
by an application of Lévy's continuity theorem (Billingsley, 1995, Th. 26.3). As such, 20 500 is an elliptical a-stable random vector.
((-1) being standard Next, we proceed with the calculation of [(0 [(--1), 2 (f-1) in the case of mi
Gaussian weights and when 98(C) = [C For this, note that (=)6-1) (xx): = ((-1) (xx))² |
I((-1), 54 (f-1) ~ NO (f-)))((((( where (x( is two-by-two symmetric matrix with diago- (x1(-1),
nal (E , (1(-1)). and off-diagonal entries ((E-1) follows that:
(190- ((e-1) + 2 (6-1)) (xx) 65t 1(-1)) + (x.))) T 1 sf (e-1) lid N (
(5)
where U is by--by matrix with all entries equal to one, with the independence over the js and conditional on 1) 8 (f-1)). Then for k. = 1, I,..., ,
20 = E15 (xx)), (x(n) (xx) ((-11), s4 ((-1) /
(2)d22G
15

---

Published as a conference paper at ICLR 2025
where in the second line p((i, (2) is the density of the random vector in Equation (5), It follows that:
SaJe
<puT 2 6it2
+ G3z2) )
2/ exp
2p
dGidGz
2Tuu1U2
p²
2(11 (p)
0102
!s 00
1 2(1 p²)
= =nfug
n2 exp
[n] 2pm 12 + nēl dmp
2T /1
1² Jo
where the second line follows by the bivariate normal p.d.f. where es is the standard deviation for the i-th entry implied by Equation (5), is the respective correlation and by evaluating the indicator function. In the last line we use the change of variables 77 ///De u. This last expression corresponds
to multiple of Equation (15) of Cho & Saul (2009), where p² cos² (0). Explicitly:
+ (1) 24
(1111 of 1 + 5 RR,k
1/2
=1 -0))())))
1 ++88 +
P
2/ 1/2
Finally:
(( k.h =
[uf [2
6/2 Jo(0), cOs (p),
as required.
C PROOF OF PROPOSITION 2
The expressions for the conditional expectation and variance, μe and A are a direct consequence of the posterior predictive distribution of Gaussian random vectors. We omit the details.
D SUPPLEMENTARY ALGORITTMM
The following algorithms are used in Algorithm 1. They compute the covariance matrix E(L) from the random scales. The algorithm used is an independent sample Metropolis-Hastnngs sampler where propose from the prior, and accept with probability corresponding to the ratio of the likelihoods.
11+1) Algorithm A Metropolis Hastopol sampler for ( $
Require: Scale of the next layers 4
(f+1) (L) and E{e-)), the p.d. matrix of the previous layer.
(E),* ~ St using the algorithm of Chambers et al. (1976).
Propose 4
if f= 1 then
Compute (z) using the formula from Theorem 1.
end if
Compute, using the proposed scale, (I)++)) (2(4))) with the formulas in Theorem 1. Simulate U (0, 1).
U p(y ((E()))))/ly EX4) then
(0).
Return 8
else
Return
end if
E (EOH IS CONSISTENT UNDER MARGINALIZATION
Consistency under marginalization means that for K G(K)), where both K, Kn are p.d. n x n matrices, then principal P x P submatrix K2 of K, also induces a distribution Kj G(K)), and
16

---

Published as a conference paper at ICLR 2025
Algorithm 3 A Metropolis Hastings sampler for 84
(1.
Require: Vector of observations y, a, previous iteration of 8+ and (
Propose $ St/2 using the algorithm of Chambers al. (1976).
Compute (I(D))* (L),, using the formula in Theorem 1.
Simulate U (0,
if U p(y (E(4))))/p() E(4) then
Return 8+ (L),²
else
Return 8
end if
those distributions are consistent, i.e., if we consider the full p.d.f. of K₁ and integrate out the entries that do not belong to K1 we obtain the same p.d.f. This is a property that is known for Wishart and inverse Wishart matrices (Dawid, 1981).
First, consider the set of positive random scales in the first layer ( fixed. Note that E is a p.d. matrix for any number of observations n. since the obtained cross product in Appendix B is valid covariance function. Similarly, for P < n any P P principal submatrix (T())): (wlog) is a consistent submatrix, since the random scales are fixed.
Now, consider principal submatrix (wlog) in the layer: (E()),:) and consider $ fixed. Then (5((+"))),;...> is a deterministic function of (E:,)) since (e) is fixed. This implies that (I(++1)) ):P.1:P is consistent under marginalization, as the only randomness comes from 84 over which we can integrate.
F PROOF OF PROPOSITION
First consider < 2. The posterior of the features is given by:
L
p(z, (0 X,y) = / p(z) (0)
13 1
1 5 X,y)
II
ds
(9
II
d
t=2
m=1
/ plz))) 11 (000 |E=2 ( x)
x xp((s 12 s X,y) ds
II
ds ds
(6)
Employing the elliptical a-stable characterization of Theorem 1, the distribution of (6) is given by:
(8) (6) (0) 2 S AO g()).
As such, we have that the posterior of (E) can depend y only through the posterior of (0(() y for = 2,..., L, since the ()()) is deterministic function of a fixed design matrix and conditional on the random scales 8
2
((!)
Now, when a = 2. all (f are equal to one with probability (i.c., they are degenerate Dirac point mass at one) and therefore the posterior (() y is also degenerate point mass 1. Thus, the posterior of the features cannot depend on the observations when a = 2, i.e., in the Gaussian case.
G CONDITIONAL MUTUAL INFORMATION
For a bivariate Gaussian vector the mutual information is easily computed by: (112)) log(1 1 p²), where p is the correlation coefficient between the two entries of the vector (Cover & Thomas, 2006,
17

---

Published as a conference paper at ICLR 2025
p. 252). However, recall that in non ma-Gaussian setting, marginal moments correlations not defined and we need conditional construction. Consider three random variables Y1, Y2, S, the conditional mutual information (Cover & Thomas, 2006, p. 23) is given by
Ji J log
p(11, 1s)
7(Y; Y2
p(y1 s)p(y2 8) p(u1, y2 ()y()) dy2ds
/ DKIlp(Yi, () p(V s)p s))p(s)ds,
where DKL is the Kullback-Leibie divergence. In our case, consider Y1, Y22 two observations from the deep a-kernel process, and S corresponds to the conditional scales. Since given S the observations Seaittionnll-Gaausiii
apply the formula that employs the correlation and integrate over the scales to obtain the mutual information. Specifically, we use:
I(Y;; Yo = - S log(1 -Yy,,,
- (s))p(sdss
To perform the integration we use Monte Carlo samples {1 JE-2: S and use the correlation induced by the (())
H ADDITIONAL NUMERICAL RESULTS FOR SIMULATED DATA
H. TIMING OF SIMULATIONS
In Table we display the running time for the six methods employed in the simulations described in Section 3. All the times are seconds and display CPU times. We avoid any type of parallelization throughout our experiments.
Table 5: Average (SD) time (s) for the six methods in the numerical examples for one, two, and ten dimensions.
Method
2D
10D
Da-KP
181.23 (15.3)
200.19 (31.16)
9713.51 (212.10)
DIWP
206.56 (10.24)
312.29 (21.17)
14509.31
(2783.10)
GP Bayes
17.26 (0.44)
21.35 (0.24)
729.78 (26.96)
GP MLE
0.08 (0.00)
0.18 (0.01)
49.89 (2.51)
NNGP
131.03 (11.99)
167.85 (6.36)
11044.99 (1927.03)
Stable
33.14 (5.64)
1456.72 (85.45)
H.2 COVERAGE OF CREDIBLE INTERVALS
In Table we display the percent of observations in the out-of-sample observations captured by the 90% credible intervals produced by the Bayesian methods. The GP MLE method is frequentist method and as such does not produce valid credible intervals. The Stable method is not available for 10 dimensions. The methods that stable random variables are the only ones that consistently manage to be close to the nominal coverage of 90%. The other methods typically have coverage much below 90%, especially in high dimensions.
H.3 VISUALIZATIONS OF SIMULATION RESULTS
H.3.1 PLOTS FOR TWO DIMENSIONS
Figure 4 displays the function fit for the example in two dimensions and the residuals with the accompanying credible intervals. We do this for the first split of the dataset. Recall that the true function is f(E,. 52) = x [ + 5 x 1 ()) The Da-KP and the Stable method are the only ones that able to capture the jump without oversmoothing, and with narrower credible intervals.
18

---

Published as a conference paper at ICLR 2025
Table Average (SD) percent of out-of-sample observations captured by the 90% credible intervals in the twenty splits of the simulation examples.
Method
ID
2D
10D
Da-KP
90.85% (5.12)
92.47% (5.63)
89.70% (2.42)
DIWP
92.70% (1.89)
95.86% (1.22)
60.47% (3.08)
GP Bayes
94.85% (2.43)
94.14% (3.02)
65.17% (5.82)
MLE
NNGP
92.50% (1.79)
95.99% (1.12)
60.65% (2.82)
Stable
91.65% (4.40)
91.11% % (4.63)
da
(a) Function for different methods.
(b) Residuals versus predictions the methods that produce valid credible intervals.
Figure 4: Function fit and residuals with 90% credible intervals for the six methods in the simulations in two-dimensions.
H.3.2 PREDICTIONS AND UNCERTAINTY QUANTIFICATION IN HIGH DIMENSIONAL SIMULATION
In Figure 5 we display the a scatter plot of residuals (observation prediction) with the prediction on the first split of the simulation example for 10 dimensions. Note that the Da-KP and GP MLE have the smallest range of residuals, while the other methods have a wider range of residuals.
Precitttn
Figure 5: Residual plot for first split of the high dimensional example with 90% credible intervals.
H.3.3 ABLATION STUDY ON a FOR THE ONE- DIEEENINNIO EXAMPLE
In Figure 6, display the predicted values (solid lines), along with the 25th to 75th percentiles for the posterior predictive intervals (shaded regions) on a finer grid (1000 points) for the example in one dimension considered in Section 3.2, for 0.5, 1, 1.5, 1.95, 1.99. It is apparent that values of
19

---

Published as a conference paper at ICLR 2025
0.5
1.5
1.95
1.99
5.0
CZ
0.5
2.5
1.5
1.95
0.0
1.99
2
11
0
2
2
-1
0
1
2 2
-1
- -1 0 1 - -1
0
1
2
Figure 6: Comparison of predictions (solid lines) and 25th to 75th percentile posterior predictive intervals (shaded regions) in one dimension for different values of a for the DaKP.
closer (the Gaussian limit) tend oversmooth the fit around the jump discontinuity at zero and in general give wider posterior predictive intervals.
H.3.4 POSTERIOR DISTRIBUTION OF THE KERNEL IN THE LAST LAYER FOR EXAMPLES IN ONE AND TWO DIMENSIONS
In Figure we display the quantiles of the posterior of the kernel matrix in the last layer for the one and two dimensional examples shown in the main text. Recall that the training points are on equally-spaced grids. Figure 7 serves evidence that the kernel is indeed stochastic, with non- degenerate posterior. Furthermore, the values obtained in the 75th percentile are quite high relative to the median and is indication of heavy-tailed distribution.
1500
20
200
40
25h
SOLL
75A
10
20
08
50
Figure 7: Posterior quantiles (left: 25, center: 50, and right: 75) of the distribution of the kernel in the last layer for the examples in 1 dimension (top), and 2 dimensions (bottom).
20

---

Published as a conference paper at ICLR 2025
I ADDITIONAL NUMERICAL RESULTS FOR UCI DATA
Figures 8, 9 and 10 displays a scatter plot of the predictions and residuals (observation prediction) of the test in the first split the UCI datasets, well as the credible intervals for said obser- vations. The predictions of the Da-KP are much closer to the observations, while the DIWP and NNGP present worse predictions relative to the observations. Moreover, the width of the credible intervals of the Da-KP vary, while the DIWP and NNGP are much more uniform. The results indi- cate that the credible intervals the Da-KP capture more observations than the competing methods. Figure 11 displays the trace plot for the log-likelihood of the fit of the deep a-kernel process in the first split of the Boston dataset, indicating a fast mixing.
-10
MLC
NSCP
40
Figure 8: Comparison of predictions and observations in the test set for the first split of the Boston dataset, with 90% credible interval.
10
10
20
42
Figure 9: Comparison of predictions and observations in the test set for the first split of the Energy dataset, with 90% credible interval.
I.1 TIMING OF UCI EXPERIMENTS
In Table 7 we display the average running time for each of the methods, except the Stable method the input dimensions of the data all greater than 2. Overall the GP MLE and GP Bayes are faster than the kernel process methods. The kernel process methods have similar running time in each dataset.
I.2 COVERAGE OF CREDIBLE INTERVALS FOR UCI DATASETS
In Table 8 we display the average coverage of the 90% credible intervals for the UCI datasets in the same splits used Section 4. The Da-KP consistently is close to the nominal 90% coverage, while the other methods do not have consistent coverage close to the nominal 90% percent.
21

---

Published as a conference paper at ICLR 2025
DWMP
G7 SAN9
Resiaia
GP
Prediction
Figure 10: Comparison of predictions and observations in the test set for the first split of the Yacht dataset, with 90% credible interval.
130
-000
lag hog-Baufoood
-250
-000
Iteraton
Figure 11: Traceplot of the log-likelihood for the first split of the Boston dataset.
Table 7: Average (SD) time (s) for the methods in the three UCI datasets over twenty different splits. Times for the Stable method are not available.
Method
Boston
Energy
Yacht
Da-KP
22320.89 (690.23)
36154.63 (1812.99)
4590.63 (149.31)
DIWP
20104.97 (3877.92)
41291.76 (15973.11)
22177.2 (10017.53)
GP Bayes
1047. (6.51)
2151.64 (12.37)
271.82 (2.21)
GP MLE
142.06 (6.68)
390.52 (9.36)
22.18 (1.20)
NNGP
18242.54 (8474.21)
36872.04 (21713.89)
9177.82 (3803.24)
Stable
J FEATURE LEARNING IN ADDITIONAL DATASETS
In Figures 12 and 13 we provide the scatter plot of the features of the Boston and Yacht data sets under Do-KP, along with the corresponding box plots and q-q plots for each of the features. Similar to the Energy dataset, the Boston features display heavy-tailed behavior, while those for Yacht are closer normality.
22

---

Published as a conference paper at ICLR 2025
Table 8: Coverage (SD) of the 90% credible intervals for the methods in the UCI datasets over twenty different splits. Stable method is not available, and GP MLE does not produce credible intervals.
Method
Boston
Energy
Yacht
Da-KP 91.86% (5.14)
87.40% (3.91)
95.32% (2.86)
DIWP
57.35% (7.50)
48.77% (6.05)
90.00% (6.61)
GP Bayes
93.24% (4.43)
97.01 % (1.79)
95.00% (3.22)
GP MLE
NNGP
68.53% (8.26)
68.77% (5.45)
67.58% (7.65)
Stable
Nomea Thansool
(a) Scatterplot of the first two features, with box plots for the marginals.
(b) Normal 9-9 plot of the first two features.
Posterior distribution of the features for the Boston data set using 10k posterior MCMC samples with 1k burn-in samples.
Figure 12:
(a) Scatterplot of the first two features, with box (b) Normal 9-9 plot of the first two features. plots for the marginals.
Posterior distribution of the features for the Yacht data set using 10k posterior MCMC samples with Ik burn-in samples.
Figure 13:
23


Reviews:
[{"review_id": "o7pc81civr", "reviewer": "Reviewer_a3tm", "summary": "The authors consider Bayesian neural networks in the infinite-width limit under infinite variance priors on the weights. They show that the resulting covariance kernel becomes a random process that depends on the given data, in contrast to the finite variance prior case, where the kernel is deterministic. Additionally, they demonstrate that the representation in each layer is Gaussian when conditioned on the associated covariance realization. In the experiments, they show that the data-dependent stochastic covariance enables feature learning, and that the resulting predictive Gaussian process achieves improved performance on benchmarks.", "questions": "Q1. Can we summarize the key points as follows: Under a specific infinite variance prior, the covariance kernel becomes a stochastic process at each layer. Even if the marginal does not exist, sequential sampling is still possible. Furthermore, conditioning on a covariance sample at a given layer yields a Gaussian representation, while marginalizing over the covariance leads the representation to follow a heavy-tailed distribution centered at 0.\n\nQ2. You could perhaps include a 1-dimensional regression example to illustrate the model’s behavior for varying $\\alpha$. Additionally, it might be helpful to show the covariance matrix to demonstrate its properties and the correlation among input data.\n\nQ3. Could you clarify the technical differences between the current paper and Loria & Bhadra (2024), which makes the approach feasible?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The technical aspects of the paper appear solid, although I have not checked the results in detail.\n- The experiments seem to effectively support the theoretical claims.", "weaknesses": "- The paper is somewhat challenging to approach, given its niche topic and highly technical content. It also feels quite text-heavy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khq4NyaG7f", "reviewer": "Reviewer_Qsid", "summary": "**Summary**\nNeal's infinite width neural networks, even when extended to the deep setting, do not account for representation learning. That is, they result in a GP with a fixed covariance function. This is unlike typical but intractable Bayesian neural networks, which are capable of representation learning. The authors consider heavy tailed priors, elliptically distributed with infinite variance, and show that the network converges to a process with $\\alpha$-stable marginals. Each layer is conditionally Gaussian, even though the covariances are not necessarily even defined.  The main technical tool is the CLT for alpha-stable RVs (Theorem 4 from another paper, with alpha=2 being the classical CLT), and a (conditionally Gaussian) Gaussian mixture representation of the characteristic function of the alpha-stable RV (equation (1)). Furthermore, a computationally viable MCMC method for doing inference is also provided. The derived method is demonstrated on some UCI benchmarks, showing better prediction than other methods, and being faster than a previous heavy-tailed method.", "questions": "**Questions:**\n- I don't understand this sentence, could you please clarify? \"The third limitation is one noted by Neal (1996), that in the case of multiple outputs, GPs cannot learn the covariance structure, as the covariance is always zero under a Gaussian limit.\" Are you saying that for a vector-valued neural network output, any two outputs are independent? This would be false, even in the finite variance setting, as long as one considers non-isotropic Gaussian weights (which is relatively straight-forward). E.g. see weights W in section 1 of \"Avoiding Kernel Fixed Points: Computing with ELU and GELU infinite networks, AAAI 2021\".\n- Another related work might be \"Richer priors for infinitely wide multi-layer perceptrons\". In particular, Theorem 1 and Proposition 2 seem to show that the resulting posterior predictive distribution also marginalises out an $\\alpha$-stable distributed scale matrix in the covariance matrix. This seems similar to equation (19), where a scale and a location parameter are integrated out in the posterior predictive, and the resulting model is a kind of infinite mixture of GPs. Although I think the motivation for considering such infinite mixtures is different: this paper focusses on heavy tailed whereas the other one focusses on exchangeability, and considers more \"mild\" mixing. Still, it could be nice to motivate the current approach both from the perspective of heavy-tailed priors and from the perspective of exchangeability in the hidden layers.\n- Is it possible to construct deep $\\alpha$-stable processes for gradient flow (extending NTK), rather than BNNs?\n- As far as I can tell from the proof of Theorem 1 in Appendix B, rectified monomial activations are not strictly required for the proof to work. All that is required is that a recursive equation for the conditional covariance is available, which is true beyond this activation for other activations (see e.g. the works I mentioned above). It might be nice to extend the theoretical statement to more broad activation functions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths:**\n- The proposed model is capable of representation learning. Specifically, Proposition 3 nicely shows that the feature at layer l depends on the data X, y observed in training, which is unlike many finite width models. It is nice to have this notion of representation learning formalised in this simple way.\n- The idea of the paper is relatively straightforward: everything is conditionally Gaussian given this scale parameter, which induces the heavy-tailed behaviour. I see this simplicity as a strength, as it doesn't require complicated constructs to convey a nice and useful result.\n- The method is demonstrated, firstly on some artifical but enlightening discontinuous functions (where it either almost matches the shallow stable process or is able to compute a posterior where the shallow stable process fails), and also on some UCI benchmarks, where it usually outperforms other methods and the previous shallow stable process is intractable. There are also some nice graphics comparing the posterior predictives of other models in 1D and a QQ plot showing non-Gaussian behaviour of the features. More extreme non-Gaussian behaviour is shown in the appendix (in particular, Figure 10 - I am wondering why the authors did not choose to include this as their main non-Gaussian demonstration in the main paper?)\n- Appendix B and D, which contain the proofs, appear to not have any major technical errors.", "weaknesses": "**Weaknesses:**\n- Unless I am mistaken, \"The key finding is that the conditional mutual information decays at a slower rate for smaller α\" should be \"The key finding is that the mutual information [which itself is computed via MCMC as an expected conditional mutual information] decays at a slower rate for smaller α.\" Figure 1 shows a mutual information, not conditional mutual information.\n\n\n**Minor:**\n- Theorem 1. $J_\\delta(\\theta)$ can be computed explicitly, and this is obvious to people familiar with the result of Cho & Saul (as mentioned in the following paragraph). But some readers might find it alarming to see $J_\\delta(\\theta)$ before it is defined. Perhaps the result of Cho & Saul could be made more explicit before theorem 1 (i.e. move $J$ up in the page)\n- Figure 1 and 2 labels are too small. Sans-serif font clashes with the text in the main paper.\n- last sentence of first paragraph of section 4.2 seems to have odd grammar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w5tbCAylar", "reviewer": "Reviewer_3kZC", "summary": "The article presents a deep kernel process with infinite variance priors including a recursive formula and experiments illustrating dependence on the inputs and behavior for discontinuous targets.", "questions": "* Could the authors please offer more comments on the motivation for considering this particular model? \n* Could the authors please offer comments on what the concrete and significant technical advances are?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The article \n* provides a discussion of references, theoretical results, and experiments. \n* discusses potential benefits in prediction and uncertainty quantification. \n* suggests feature learning that is not possible under a Gaussian process.", "weaknesses": "1. My main concern with the article is the writing and presentation, which I think need to be improved. \nThe abstract gives a long discussion of prior works but ideally it should instead give a crisp description of the main points in the article. The lengthy discussion in the introduction comments on prior works and perceived limitations, but does not provide a sufficiently concise and clear description of the objective, motivation, and contributions of the present work. Terminology could be introduced more clearly in preliminaries, rather than in prior works. The writing is in parts repetitive and the thread is not sufficiently clear.  \\\n&nbsp;\n\\\nSuggestions proposed by the ICLR Associate Program Chair: \n    1. Revise the abstract to focus on succinctly stating the key contributions and results of the paper, rather than discussing prior work.\n    2. Restructure the introduction to clearly state the objectives, motivation and main contributions upfront before discussing related work.\n    3. Add a preliminaries section to introduce key terminology and concepts.\n    4. Review the paper to remove repetition and improve the logical flow of ideas between sections.  \\\n&nbsp;\n\n2. My second concern is the motivation and innovations, which I do not find sufficiently clearly articulated in the article. \nFor Theorem 1 it is mentioned that the result permits characterization of the deep kernel process in an infinite variance setting and for Proposition 3 that it results in a posterior distribution of the features that depends on the observations. I can see that there are differences to prior works and that feature learning is an interesting topic, but in my opinion it is not sufficiently clearly explained what the motivation is for formulating this particular deep kernel process. Another point that is not described in sufficient clarity is what the concrete and significant conceptual and technical innovations are that go into obtaining the proofs and how they could be useful for obtaining further relevant advances.  \\\n&nbsp;\n\\\nSuggestions proposed by the ICLR Associate Program Chair: \n    1. Provide a clearer explanation of why an infinite variance setting is important or beneficial for deep kernel processes.\n    2. Discuss more explicitly how the ability to learn features from observations addresses limitations of previous approaches.\n    3. Highlight the key technical challenges in proving Theorem 1 and Proposition 3, and how overcoming these challenges advances the field.\n    4. Explain potential applications or extensions of their technical innovations beyond this specific model.\n    5. Provide a clearer explanation of why this particular deep kernel process formulation was chosen and what advantages it offers.  \\\n&nbsp;\n\n3. Description of the experiments. \nIn my opinion the motivation and description of the experiments are not sufficiently clearly presented.  \\\n&nbsp;\n\\\nSuggestions proposed by the ICLR Associate Program Chair: \n    1. Provide clearer motivation for each experiment, explaining how it relates to the theoretical results.\n    2. Improve the description of experimental setups, including details on datasets, baselines, and evaluation metrics.\n    3. Discuss the implications of the experimental results in more depth, relating them back to the theoretical contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCYDU7pknF", "reviewer": "Reviewer_xtbp", "summary": "Although deep kernel processes allow for more flexibility than standard GPs, they still assume smoothness and finite variance, which is unsuitable for modelling data with heavy tails or discontinuities. This is particularly relevant to representation learning, where deterministic covariance kernels limit the ability to learn non-degenerate, data-dependent representations. In contrast, $\\alpha$-stable distributions are well-suited to problems were large fluctuations or discontinuities can be expected. Taking inspiration from these strengths, in this paper the authors assign $\\alpha$-stable distributions with infinite variance to the weights in each layer of a BNN, such that the BNN now converges to a deep $\\alpha$-stable kernel process rather than GP in the infinite-width limit.\n\nThey also develop conditional Gaussian representations of $\\alpha$-stable processes that allow for recursive computation of the stochastic kernels at each layer. Through a combination of numerical experiments on synthetic discontinuous functions and evaluations on widely-used datasets from the UCI repository, the authors demonstrate that the deep $\\alpha$-stable kernel process outperforms competing approaches when modelling discontinuous or heavy-tailed data. However, simpler models or GPs with traditional Gaussian priors might be preferable for smoother data due to their lower computational cost.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Due to my limited familiarity with α-stable processes, I didn’t review all of the theoretical details. However, the paper reads as well-structured and complete, and the writing clearly highlights the contributions in the context of prior work with a solid mix of theoretical grounding, ablation studies, and performance benchmarks on public datasets.\n- The problem statement is otherwise well-described, and I think this work should be of interest to future work on BNNs by addressing core limitations of smoothness and finite variance in existing models. I also wonder whether the non-Gaussian priors explored in this work could also be considered in other deep models?\n- I appreciated the extensive numerical experiments that showcased the model’s performance in controlled settings that properly validate it meets the intended criteria of modelling non-Gaussian behaviours. I do wish the section real-world datasets was more compelling however.\n\n**Post-rebuttal update**\n\nRaised score from 6 to 8.", "weaknesses": "Although the paper is well-written overall, the dense presentation of different concepts makes it challenging to intuitively grasp without visual aids or simplified examples. I would recommend the inclusion of more figures illustrating how the α-stable kernel process differs from traditional GPs, ideally showing its behavior on simple synthetic data to highlight the flexibility and stochasticity introduced by their method. Some detail can likely also be moved to the appendix instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6FnlBhrpOr", "reviewer": "Reviewer_o61u", "summary": "This paper considers an infinite-width Bayesian neural network, with infinite-variance weights.  Interestingly, in this setting, representations/kernels do not become deterministic as usual in infinite-width limits, but remain stochastic, and hence learnable.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical contributions are extremely strong.  They consider an extremely interesting regime, and the question of how to avoid deterministic kernels in infinite-width networks is a critical one for Bayesian neural networks.\n\nThey have an extensive and excellent Appendix.", "weaknesses": "The experiments are weaker, but this can be expected in a paper with a mainly theoretical focus and contribution, such as this one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)