Bayesian Treatment of the Spectrum of the Empirical Kernel in (Sub)Linear-Width Neural Networks
Authors: Ouns El Harzli
Bernardo Cuenca Grau
Venue: ICLR 2025 Poster (2025)
Keywords: infinite bayesian neural networks
kernel theory
random matrix theory
Primary area: learning theory

Abstract: We study Bayesian neural networks (BNNs) in the theoretical limits of infinitely increasing number of training examples, network width and input space dimension. Our findings establish new bridges between kernel-theoretic approaches and techniques derived from statistical mechanics through the correspondence between Mercer's eigenvalues and limiting spectral distributions of covariance matrices studied in random matrix theory. 
   Our theoretical contributions first consist in novel integral formulas that accurately describe the predictors of BNNs in the asymptotic linear-width and sublinear-width regimes. Moreover, we extend the recently developed renormalisation theory of deep linear neural networks, enabling a rigorous explanation of the mounting empirical evidence that hints at the theory's applicability to nonlinear BNNs with ReLU activations in the linear-width regime.
   From a practical standpoint, our results introduce a novel technique for estimating the predictor statistics of a trained BNN that is applicable to the sublinear-width regime where the predictions of the renormalisation theory are inaccurate.

Published as a conference paper at ICLR 2025
BAYESIAN TREATMENT OF THE SPECTRUM OF THE EMPIRICAL KERNEL IN LIBEAR-AR-- NEU- RAL NETWORKS
Ouns El Harzli
Bernardo Cuenca Grau
Department of Computer Science University of Oxford Oxford, UK ouns wwwarrallle axx.ac.uu .uk
Department of Computer Science University of Oxford Oxford, UK berna rdo. cuenca. gwwaeagrrauuu....ooo... ac uk
ABSTRACT
We study Bayesian neural networks (BNNs) in the theoretical limits of infinitely increasing number of training examples, network width and input space dimen- sion. Our findings establish new bridges between kernel-theoretic approaches and techniques derived from statistical mechanics through the correspondence be- tween Mercer's ' cigenvalues and limiting spectral distributions of covariance ma- trices studied in random matrix theory. Our theoretical contributions first consist in novel integral formulas that accurately describe the predictors of BNNs in the asymptotic linear-wer width and sublinear- width regimes. Moreover, we extend the re- cently developed renormalisation theory of deep linear neural networks, enabling a rigorous explanation of the mounting empirical evidence that hints at the theory's s applicability to nonlinear BNNs with ReLU activations in the linear-width regime. From practical standpoint, our results introduce novel technique for estimating the predictor statistics of trained BNN that is applicable the sublinear-width width regime where the predictions of the renormalisation theory are inaccurate.
INTRODUCTION
Bayesian Neural Networks (BNNs) are a variant of neural networks that incorporate Bayesian infer- ence techniques to mitigate overfitting, enable learning from small datasets, and capture uncertainty in predictions (Neal, 2012: Gal, 2016). In BNN, prior probability distributions are specified for weights and biases. During training. the posterior distribution, which represents the updated knowl- edge about the parameters after observing the data, is updated using Bayes' rule. A trained BNN can be interpreted as an infinite ensemble of neural networks where each individual contribution in the ensemble is weighted by the posterior probability of its parameters given the training data. Although computing the posterior distribution is intractable and difficult to approximate, BNNs have gained significant traction with the development of effective estimation techniques (Gal, 2016; Blei al., 2017). BNNS demonstrate generalisation performance on par with deep neural networks trained us- ing gradient descent (Lee et al., 2020; Magris & Iosifidis, 2023). BNNs also showcase improved sensitivity to out-of-distribution examples (Gal, 2016) and the ability to estimate uncertainty.
In effort analyse the generalisation properties of BNNs, researchers study idealised views of fully-connected neural architectures defined by the input dimension, the layer widths, and the ac- tivation function. As the width approaches infinity in each layer (the NNGP limit), the functions generated by random weight selection converge in distribution to Gaussian process (GP) (Ras- mussen Williams, 2006). The covariance function of such GP, called the NNGP kernel, can be recursively defined by proceeding on layer by layer basis (Lee et al., 2018). This perspective based on kernel and GP theory has inspired formalisms that mimic different aspects of the behavior of BNNs in the infinite-width limit Aitchison et al. (2021), including representation learning Yang et al. (2023). Simultaneously. generalisation error of related kernel and random features models (Canatar al., 2021: 1: Simon al., 2023). These formulas often rely on the spectral universality assumption (SUA), which simplifies the derivations by approximating the eigenfunctions of the kernel with independent Gaussian entries
has led to the development of analytical formulas to estimate the

---

Published as a conference paper at ICLR 2025
(Karoui, 2010; Cheng & Singer, 2013; Fan & Montanari, 2015). Extensive research is being devoted to study the accuracy of the SUA (Liu et al., 2021; Lu & Yau, 2023; Bosch et al., 2023).
In addition to the NNGP limit, BNNs have also been studied under the linear-width limit (also referred to as thermodynamic limit or proportional limit) where the network's 's width, the number of training examples and the dimension of the input space are taken simulataneously to infinity while keeping constant and bounded ratios between them (Engel et al., 2012). By employing techniques from statistical mechanies, such saddle point approximations (Seung & Sompolinsky, 1992; Li & Sompolinsky, 2021), the replica method (Barbier et al., 2018: Canatar et al., 2021), and random matrix theory (Wigner, 1955; Livan et al., 2018; Fan & Wang, 2020), researchers have studied the mean and variance of the output generated by trained BNNs in this setting. A recent theoretical work (Cui al., 2023) has derived the predictor learned by non-linear BNNs in the case of Gaussian data. More recently, publinear-widhe regimes, where the width (or the input dimension) is small compared to the number of data points (Maillard al., 2024), and related scalings (van Meegen & Sompolinsky, 2024) have been studied, and the emergence of strong feature learning has been demonstrated in these scenarios.
One of the most prominent results in this literature is the renormalisation theory (Li & Sompolin- sky, 2021) of linear BNNs (i..., those without non-linear activations) in the linear-width regime, which establishes that the mean predictor and the predictor variance of the BNN coincide with that of Bayesian linear regression, but surprisingly the variance must be renormalised by a factor de- pendent the training data and problem dimensions. Subsequent developments have provided more detailed analysis on the linear setting including non-asymptotic results (Hanin & Zlokapa, 2023), and comparison with deep random feature models (Zavatone-Vett) et al., 2022). It remains an open question, however, whether the insights from the renormalisation theory for linear BNNs can be extended to non-linear networks, as suggested by empirical evidence (Li & Sompolinsky, 2021: Ariosto et al., 2023), and how the theory should be adapted to sublinear-widd width regimes, where discrepancies with empirical results have been observed (Li & Sompolinsky, 2021).
Our Contributions In this paper, we establish new connections between the kernel-theoretic per- spective associated with the NNGP limit and the statistical mechanics viewpoint associated with the linear-widts width and sublinear wiith limits, and contribute new insights to the generalisation properties of BNNs. First, we demonstrate that training a (non-linear) BNN in the linear-width and sublinear- width limits result in predictor with identical mean and variance to that of GP regression with modified NNGP kernel, and we observe that the Mercer spectrum (Mercer, 1909; Minh et al., 2006) of this kernel is known in the linear-widhe regime. Second, using this observation, sary and sufficient conditions (on the data and the architecture) for the application of renormalisation theory to non-linear BNNs in the linear-width limit. These conditions also provide a criterion for determining the applicability of the spectral universality assumption (SUA) from kernel theory in the context of BNNs. Third, we present initial findings on a sublinear-width regime where the rele- vant quantities are simultaneously taken to infinity while the number of training examples remains proportional to the product of the network width and the dimension of the input space. In particular, we provide a novel mechanism for estimating the mean and variance of the predictions of non-lineat BNNs this setting, for which renormalisation theory is not applicable.
prove neces-
PRELIMINARIES
We use standard notation for real-valued vectors v E RR", matrices A E Rm and their transposes v and AT. We use a, to denote the vector in the i-th row of A. The Moore-Penrose pseudo-invers, of a matrix A is denoted as At (Moore, 1920).
Neural networks. A fully-connected neural (FCN) architecture with L layers is tuple f ((W') (b²) Jisee, {or) (ises))) Each layer & E (1.... [.} of width N₂ is given by weight matrix W Ne 1, bias b' E R and activation function of R + R. On input x E Ro, network of sets x = x and then computes recursively on the depth the sequence of pre-activations he and activations x as follows, where the network's 's output (xx is given by xt:
h w +be
2

---

Published as a conference paper at ICLR 2025
We assume that all but the last layer have the same width N.. For the last layer, assume width Ny = 1 (ensuring a real-valued output), b" = 0 and o Idg (ensuring linearity). In this setting, the weights W are referred to as the readout weights (Li & Sompolinsky, 2021).
Kernels. A kernel on No is a positive semi-definite symmetric function K No x R R. By Mercer's theorem (Minh et al., 2006), given distribution x p(x) with compact support on R:N, there exist unique countable collections of Mercer's eigenvalues (Ae) and eigenfunctions (Pi)ieN such that K (x, x') = L; (x) and (yp: are orthonormal w.r.t. the data distribu- da,j for all i.j. By Riesz's 's theorem, there exists a Hilbert space H and a feature map Φ: No + H. such that (xx, x') = (((x).((xx)) (xx)) Kernel regression amounts to linear regression in the corresponding Hilbert space: when trained data X. the prediction a new point x" is given by & x x xx where the vector kx- x is given by (kx-.),, = K (xx", x;) and the kernel matrix Kx.x is given by (Kx.x),, = xxx, Although the kernel's eigen- functions exhibit the described structure, the spectral universality assumption (SUA) is commonly adopted. The SUA posits that, as P increases, the eigenfunctions can be approximated by inde- pendent Gaussian entries: (x((X)) N(HK, where μk and 0 depend on the kemne K and the data distribution p(x), but not on specific instances and j. The SUA works well in practice (Karoui, 2010; Cheng & Singer, 2013; Fan & Montanari, 2015; Liu et al., 2021; Simon et al., 2023; Lu & Yau, 2023; Schröder et al., 2023), and research focuses on identifying conditions under which it holds.
tion: Ex-p(x) (xx(())
Random feature maps. Let θ represent all parameters of f up to layer L - 1. The random feature map ((O,.) ) No + RV is a nonlinear transformation (random in 9) mapping the input and the activation x 1-1 By definition, f(x) = (WL)T ((O,, x), and to highlight the parameter dependency we denote it as fe.w.. The random feature map is associated to random kernel K-N.No (x, x') + d (Ø(O, x), $(O, x')) expressed as the inner product between the corresponding random feature map evaluations. For this kernel, the Hilbert space H R is thus known.
Training set. The training set yX consists of P examples sampled i.i.d. from an unknown distribution PN, with compact support on No x R. We assume that in the limit Ng -> oc, P PN, converges (in distribution) to well-defined distribution with compact support over I8 x R noted lim NN.-- We denote each example by (x:, W:), so that X (x1,..., xp) R Px and (V) ..... yr) E R.P. We denote the evaluation of the random feature map the training set by (0,
(Ø(0, x1), xi),..,, ((O, xp))² E RNPP; this induces an empirical kernel matrix N.N.No (X, X) given by N |XX 0) 6XX θ) E RPxP. The training data X also induces an 9
empirical distribution (x(x)= = (EE1 8x. (x) with 6x. the Dirac measure.
BNNs. We assume a prior distribution over parameters (O, W with weights sampled i..dd from (0, +)) and biases sampled i.i.d. from.((O, (0, 1); this yields density p(O, W4) that is product of Gaussian densities. The posterior distribution given the training data is given by Bayes' rule:
0 w = (OO, W O, W4)
ply|X,
X(y[X)
where x(y) θ W is the likelihood of the data given set of parameters, and p(y[X) / 9, W Ip(,, W W=W is the marginal likelihood (or evidence). We assume Gaus-
sian likelihoods, i.c. p(y X, θ, W²) ~ NN ((,, X)" W 6(0, ()). Calculating the pos- terior distribution, which is the essence of BNN training, is analytically intractable and remains core challenge (Gal, 2016). In practice, the posterior distribution is estimated via variational infer- ence (Blei al., 2017) or Monte-Carlo simulation methods (Rasmussen, 1995).
Given the posterior distribution, the predictor defines a distribution over functions fe,wz with (9, W4) ~ p(O, w The mean-squared generalisation error is defined for any new point (x*, v²) as the expectation over the predictor error: Eee, Only the mean and variance of the predictor are needed to calculate it. ((1 - (
3

---

Published as a conference paper at ICLR 2025
Gaussian processes and NNGPs. A GP g over a space R is random scalar field such that its evaluation at any collection of finitely many points (g(x1), .....,....... g(z)) follows a multivariate Gaus- sian distribution. A GP is determined by mean function μ I No + E. and covariance function k No x No + R. which describe respectively the mean of the Gaussian distribution at each point and the covariance between the Gaussians at any two points. The covariance function of GP is a kernel (Rasmussen & Williams, 2006). We note GP(u K). Gaussian process regression consists in performing Bayesian inference using Gaussian process as the prior distribution over functions. The prediction distribution of GP regression with prior GP(O, K) trained on the data y, is given, on new point x", by NK x kxx XXY, (xx",xx xx) K x xx..xks xx x). The mean prediction of GP regression coincides with the prediction of kernel regression with the same kernel.
Applying successively the central limit theorem to each layer, the infinite-width limit of equa- tion yields a GP, called the Neural Network Gaussian Process (NNGP). If we let the width N - co. the h GP(mt, k²) are independent and defined inductively by layers as fol- lows for all x, x' -R--No and each x"'x' Then, h GP(p-11, Kt-1) and the covariance functions K K*(x, x') are given by Eeptt 1,Ke 1) (of (x))o' (lf-1(x'))) The covariance function K is the NNGP ker- nel (Daniely al., 2016), denoted as KL KNNGP. infinite-widht limits involve various subtleties (Matthews et al., 2018). and we follow the approach in Lee et al. (2018) where infinite limits are taken sequentially. In this limit, the number of examples P and the input dimension No remain fixed. Furthermore, we will investigate more comprehensive limits where P. .N, and No all tend to infinity simultaneouslly first while maintaining constant and bounded ratios a = &f and ogp No (linear-width regime), then while P N No. thus a + oo and ag - oc ssublinear-widee regime).
1,... L. First, AE (²(x) O and (x,,'')
Random matrix theory. Random matrix theory (Wigner, 1955; Livan et al., 2018) is the study of the spectral distributions of large matrices of random variables. The spectral measure Fp for a given matrix, with eigenvalues X; is given, for x E R. by Fr(x) == } Er= 8x, (x). where 5x, (z) represents the Dirac measure centered at the eigenvalue A₁- When the matrix is random, the spectral measure becomes a random measure, called the empirical spectral distribution. Our focus lies in studying weak convergences (convergences in distribution) of the spectral measures towards nonrandom measures (Geronimo & Hill, 2002). A sufficient condition for weak convergence of measures is to have pointwise convergence in their Stieltjes transforms (Geronimo & Hill, 2002). We rely on a famous result in random matrix theory. Consider W E R Nx'' a random matrix with i..d, entries drawn from (0, 1) and V nonrandom positive semi-definite matrix. Suppose that & has a limiting spectral measure p. and let P.N - co with fixed ratio a == N. then the random matrix WWT 1/2 has a limiting nonrandom spectral measure PhIp X Pastur map of p. denoted M Zx,, is defined by the Stieltjes transform solving the Marchenko-Parte equation (Marchenko & Pastur, 1967; Fan & Wang, 2020). It also appears in the free probability literature as the free multiplicative convolution between the probability measures p and (Mingo & Speicher, 2017). When considering the specific case where V Ip (identity matrix of size P). then P represents the Dirac measure at and we recover the well-known Marchenko-Paue distribution, denoted as P. p. Furthermore, we denote as PMP X p == PMP X (..( P p)) the composition of successive Marchenko- Pastur maps.
p. The Marchenko-
3 BNNS AS MODIFIED GP REGRESSION
First we state our definitions of lincar-width and sublincar-width regimes. Assumption 3.1 (Linear-width regime). Assume that N with the ratios a, au E (0, +00).
P and P No - on as P,N, No -> co
Assumption 3.2 "Sublincar-widde regime). Assume that N.No ratio E (0, +00).
P + as P. + oo with the
Our first aim in this section is to showcase the emergence of a modified NNGP kernel during the training of BNNs in the linear-wide width and sublinear-widit limits. We then study the Mercer's 's spec- trum of the modified NNGP kernel and exploit it to extend the renormalisation theory encompass
4

---

Published as a conference paper at ICLR 2025
nonlinear networks in the linear-width regime. Finally, outline the fundamental arguments sup- porting the expansion of this theory to the sublinear-width width regime.
3.1 THE MODIFIED NNGP KERNEL
Mercer's theorem applied to the random kernel Kg and the data distribution px decom-
poses the kernel into terms of eigenvalues and eigenfunctions as follows: K K.N.Noo (x, x') = P.N.No with spec-
This defines a random spectral measure Pe trum given by the eigenvalues and random eigenfunctions 9Pk No. (-) Here, the dependency in P stems from the empirical training data distribution. We use the correspondence between Mercer's eigenvalues and the limiting spectral measure of the corresponding empirical kernel matrix show that, to estimate the infinite random matrix Kg P.N.NNo (x, X), there is no need examine the joint distribution of its eigenvalues, as Mercer's 's eigenvalues can be sampled independently. This is a cru- cial observation because the correlations between kernel matrix eigenvalues the classical eigen- decomposition is an obstacle in the computation of the posterior distributions.
Σx k
pk Pp P. (x):px
Theorem 3.3. Assume that Assumption 3.1 (respectively. Assumption 3.2) holds. Assume that for each ls N there is a random function 4x 2..00 R + R (respectively, (p2) such that up NNNNNNo (xx)) converges in distribution to fkk 0.:00 (x) (respectively, (x)), where x lim P No. Assume that the spectrum of K K. (X, X) (respectively, the strictly positive support of the spectrum) admits P.N.No
limiting nonrandom measure pa,00 (respectively, (²)) Consider the random matrix 4 AA",, with & E PxXM, Di.k := 0,000 (5x) (respectively, 4%) and A E RM AAx11 := 5x,12x with each X follows independently p0000 (respectively, p²) and each &, follows independently lim PNo N.N (X,X) X) and o converge (in distribution) to the same Then, the random matrices e distribution over 12 NxN in the limit
M
P
- OC.
Proof. In the linear-width (respectively, the sub-linear width limit), the positive semi-definiteness of any matrix extracted from NNN.No and px maintained (the limit of a positive sequence remains positive), and this suffices to characterise the kemnel property over a compact subset of an infinite- dimensional space (Saitoh & Sawano, 2016). Thus, there random kernel Km.00 (respectively,
KT) defined over IR which characterises the convergence in distribution of K P.N.No (X.)). As per Mereer's theorem, Kg.00 (respectively, KT) also defines a random spectral measure pe 0,,00 (re- spectively, PT) associated with its Mercer's 's eigenvalues, By Baker's ' result (Baker, 1977) stating the kernel matrix to the Mercer eigenvalues of the respective kernel, it follows that Pe a.90 (respectively, Pb) is the limiting spectral distribution of the random matrices Kg NoN (X, X) in the linear-width limit (respectively, the sublinear wiith regime). By assump- tion, this spectral measure (respectively, the strictly positive support of this spectral measure) is nonrandom Pe 0,00 = =pa.00 (respectively, P3 = p²). Thus, we can reformulate the empirical ker- nel matrix corresponding to the random kernel K-. (respectively. Ky) as TT where ò are drawn independently according to po:00 (respectively, p²). Since the spectral measure no longer depends on θ, the eigenvalues can be sampled independently from the eigenfunctions. It follows N.N.No (X, X) and & A converge to the same distribution over NNNN
e
convergence of eigenvalues
that K 9
This result is non-trivial and only holds if the spectrum admits a nonrandom limit: this is the key argument that allows us to disregard, in the limit, the correlations between eigenvalues when using the Mercer decomposition. Note that the distribution of eigenfunctions & 2.000 and 1 are not known in general. We will justify in the next section the SUA as a means for alleviating this limitation. Similarly, we will denote with o evaluations of the eigenfunctions on an unseen data point x*.
The modified NNGP kernel is the random kernel KB.00 (respectively, K2) defined over RNN in the linear-wide regime (respectively, the sublinear-wied regime). In the limit, the feature map is not known explicitly, but it must exist by Riesz's 's representation theorem.
'Expression 4 A is not the usual eigendecomposition of square matrix: the evaluations of eigenfunc- tions yield rectangular (infinite) matrices. This decomposition is enabled by Mercer's 's theorem and applies to kernels.
5

---

Published as a conference paper at ICLR 2025
The nonrandom spectral measure is known in the linear-width regime. Observe that, in the linear- width regime, for many cases of interest (including ReLU activations), the limiting spectral measure Pe aa indeed no longer depends on ② and hence becomes nonrandom measure. To this end, let us first consider the kernel random matrix K KNNGP(X. X) associated with the NNGP kernel KNNGP- El Harzli et al. (2024) have shown that, under mild assumptions on the activation functions of (namely measurability and Lipschitz continuity), K KNNGP (X, X) admits a limiting nonrandom spectral measure PNNOP as No + Oo with constant ratio ag; and furthermore, in the linear- width limit, the limiting spectral distribution of the same random matrix as K Po (X. X) but where the interior widths have already been taken to infinity (i.e. when the linear-width limit only pertains to the last-layer width) is PM M do PNNGP. By immediate induction, successively applying the lincar-width limit to the hidden-layer widths and keeping the remaining interior widths infinite until reaching the input layer, it follows as direct corollary of Theorem 2 in El Harzli et al. (2024)
that, in the width limit, K No (X, X) also admits limiting nonrandom spectral measure given by the composed Marchenko-Pain Pastur maps PMP LL PNNGP.
3.2 TRAINING BNNS WITH THE MODIFIED NNGP KERNEL
We can now study the predictor statistics of trained BNNs in the linear-width limit and the sublinear- width limit. In particular, the following theorem provides integral formulae to estimate, under the SUA, the first and second moments of the trained BNN using only the limiting spectral measure. In this section, the results hold indistinctively of the linear-width or the sublinear-width limit, so to simplify notations, we will note p for both pa:00 and p" and Ke for both Ka.00 and Kg
Theorem 3.4. Assume that Assumption 3.1 or Assumption 3.2 holds. Let p be the nonrandom spectral measure characterising the modified NNGP kernel Ke,, and assume that the SUA holds.
The mean (f)(x,, y) and variance ((61))((x" (x"", .) of the predictor associated to a BNN with training data (X. y) is given by expressions equation 2 and equation 3 respectively."
S u- (
( A T --
ply, $ X) X
dp (p(A) D&DD
(2)
((f{))= (8 / AΦ' Ax" Φ. A rt - '' ply, D X
X) dp (A) D&DD' (3)
where D46 is a standard Gaussian matrix measure, Dij wiid N obtained from the SUA; the likelihood is given by ply, 11A,X) N'o" y, A); the marginal likelihood is given by ply X) = f ply, &/A, X)dp (A) Dep.
The integral forms equation 2 and equation 3 provide a new estimation of the predictor statistics of a trained BNN. While these expressions are exact only in the limit, we will present empirical evi- dence suggesting that they constitute a reasonable approximation. A practical challenge arises from the need to estimate the spectral distribution p(A), which often involves diagonalising a kernel ma- trix. This process can be computationally intensive, especially for large training datasets; however, it is worth mentioning that in many applications of BNNs, where the training datasets are rela- tively small, this computational difficulty becomes less significant. In particular, in the linehr-width regime, p(A) is the Marchenko-Pan Pastur map of the empirical spectral distribution of the NNGP kernel PNNGP which can be computed by diagonalising the NNGP kernel Cho & Saul (2009) to fixed-point equation in the Stieltjes transform space (Marchenko & Pastur, 1967; Fan & Wang, 2020).
PMP X oo
estimate PNNGP and solving numerically the Marchenko-
Theorem 3.4 and its proof also offer valuable new perspectives on the applicability of the SUA. In particular, in the last steps of the proof, the probability density of Φ & no longer appears directly in the integral. For given A, x", if each orthogonal matrix 4 4 has a non-zero probability of
"This result first appeared in the context of neural networks in Fan & Wang (2020). The result by El Harzli al. (2024) extends
more general setting.
'Here. X is the infinite matrix representing the linear width or bollinear-widhh limits of the training data. To be completely rigorous, we should write f(x', X. y) lima,N.No- fr,NNN (x" No. X.,No, yr). but by slight abuse of notation the same notation is used for both. In practice, one would use finite (but large) objects in calculations.
6

---

Published as a conference paper at ICLR 2025
occurring, the integral spans uniformly over the entire space of orthogonal matrices of size P x M. This is useful because in the limit of infinite dimensions, this space coincides with that of Gaussian matrices with independent entries (independent infinite Gaussian vectors orthogonal). Remarkably, this property precisely corresponds to the SUA in kernel theory (Karoui, 2010; Jacot al., 2020; Simon al., 2023), which posits that, in terms of the generalisation error statistics in kernel regression, the eigenfunctions can be approximated by Gaussian matrices with independent entries, denoted P ~ (uK, k Note that this approximation is not the same the Gaussian equivalence assumption (Schröder et al., 2023; Cui et al., 2023), which assumes Gaussianity of the predictor (here, the eigenfunctions are Gaussian but the predictor non- non-Gaussian). To the best of our knowledge, this marks a first connection between BNNs and the SUA from kernel theory (i.e. applied to Mercer's eigenfunctions).
We can now reframe the question concerning the correctness of the SUA approximation as follows: given A and X, is the case that all orthogonal matrices + have non-zero probabilities (according to Q) to satisfy Ke (X, X) D A If this condition holds, the SUA is applicable and Gaussian eigenfunctions can be used for estimating equation 2 and equation 3. Since the prior is a Gaus- sian matrix, any matrix has non-zero probability of occurrence. thus it suffices to show that for any orthogonal o there exists a θ such that (x(X, = TAT". In particular, show that the SUA always holds in the linear case: for any orthogonal o, there exists O such that XTOTex = $ With a non-linearity, the problem is less obvious: is there a θ such that 600, X) ( = - for any orthogonal + ? In the next section, we show that, in the linear- width limit, this criterion about ((,,.) and X is a necessary and sufficient assumption for the renormalisation theory to hold.
is easy to
3.3
AN EXTENDED RENORMALISATION THEORY
This section only concerns the linear wichh regime. In this limit, we can explicitly derive the results of our integral estimators because the corresponding limiting nonrandom spectral measure is known El Harzli et al. (2024) (see Paragraph 3.1).
The renormalisation theory for linear BNNs establishes that, in the thith limit, the marginal likelihood p X) follows multivariate Gaussian with mean vector y and covariance matrix Mg Ko. with Kg Ng x and ug the renormalisation factor fulfilling the fixed-point equation 1 1- 140 =
a(1 with ro- Ko "y This result obtained in Li & Sompolinsky (2021) by
4 successively applying the saddle point method when integrating out the weights θ, W
The following theorem shows that this result generalises to BNNs with nonlinear activations if and only if the SUA is correct (i.e., it gives the correct estimate for the marginal likelihood). Here, exploit the characterisation the correctness of the SUA developed as a corollary of Theorem 3.4.
Theorem 3.5. Assume that Assumption 3.1 holds. Let WNNGP fulfil the fixed-point equation
UNNGP a(1
TNNGP
I. "NNGP
with TNNGP bb KNNGP -ty. The marginal likelihood for nonlinear BNN verifies ply) X) N(y, KNNGKKNNGG) if and only if. for given A,X and orthogonal 4, there exists O such that (0, x)* ((0, X) TT
Proof. In the linear case, the true NNGP kernel is simply KNNGP(x, x') No x" x'; hence, PNNGP the limiting spectral distribution of the kernel random matrix Kg, which we denote Pn- The renormalisation theory of linear networks thus implies that:
Ctp
f ply, A X)d (DMP 8 po) (A) ~ of
(5)
This identity is exact in the linear-width limit and holds in general without assumption on X, y, as long the integral DΦ is uniform the space of orthogonal matrices.
Assume the SUA holds in the nonlinear case. Thus, we can express the marginal likelihood as p(y X) ply, A X)d (p R 20 PNN (A) DΦ. Furthermore,
freely interchange
7

---

Published as a conference paper at ICLR 2025
the role of Ko and KNNGP(X, (X, X) in equation 5. Indeed, it suffices to consider the linear case and new training dataset X which exhibits the same covariance structure No XXT as that of (NNGP X). As result, a similar equation to equation applies to nonlinear networks by replacing the linear kernel (x, x') + No xxx"x x' with the true NNGP kernel in the equations, provided that the SUA holds:
/ ply, PIA Xjd (PMP R
N PNNGP) (A)Da ~ Ny, "NNGGP KNNGP)
Conversely, if the SUA does not hold, the integral with respect to & does not span the space of orthogonal matrices, the identity equation 6 is no longer exact (all integrands are stricly positive), nor is the renormalisation. Thus, the SUA is necessary and sufficient for the renormalisation to hold.
This result characterises the renormalisation theory in the nonlinear case and describes a continu- ous transition between an accurate and a poor approximation. Specifically, if the SUA significantly deviates (the feature map spans small fraction of the space of orthogonal matrices) then the equiv- alence equation 6 also deviates substantially from the correct value. For example, in the spiked kernel case, which occurs for one step of feature learning Dandi et al. (2024), the orthogonal matri- ces pernissible for constructing the prior kernel is significantly constrained, thus we anticipate that the spectral universality assumption would fail in this scenario. Conversely, if the SUA is nearly accurate (meaning that the feature map encompasses a large portion of the space of orthogonal ma- trices) then equation closely approximates the true marginal likelihood. Thanks to these insights, future research on BNNs can benefit from research advances on the accuracy of the SUA (Liu et al., 2021).
3.4 APPLICATION TO THE SUBLINEAR- WIDTH REGIME
In this section, we consider the application of our integral estimators to the sublinear-width regime. Assume that Assumption 3.2 holds. In this regime, the ratios = N width regime tend to infinity and hence are no longer bounded. Here, the renormalisation theory breaks even in the linear case, because the random matrix K. N...N.No (X,X) X) becomes degenerate and its limiting spectral distribution is the Dirac distribution at 0 and equation no longer holds. A mismatch with the predictions of the renormalisation theory has indeed been observed empirically for high values of a and Oig (Li & Sompolinsky, 2021), hence the need for new theory.
P. and ag = No
P. from the linear-
Remarkably, our kernel-theoretic description of BNNs (Theorem 3.4) still holds as its validity relies only on the dot product of random feature maps (e),)) defining a random kernel (see proof of Theorem 3.3). This remains true for the sublinear-width regime (as well as for other regimes of interest). Additionally, zero eigenvalues in Mercer's 's decomposition can be disregarded since they do not contribute to the kernel evaluation. An alternative perspective is that, when calculating the
first two moments, one takes into account the Moore-Penrose pseudo-inverse No x) of the kernel random matrix. Consequently, only the contributions from the strictly positive support of the limiting spectral distribution estimators for the mean and variance of the predictor remain applicable under the SUA. The only missing element in the argument is whether p( A |X x*), which is now the strictly positive support of the limiting spectral distribution of K.P.N.No (X, X) (rescaled to integrate to 1), also converges to a nonrandom spectral measure (in order to apply Theorem 3.3). Thus, precisely characterising the asymptotic behavior of this Mercer's random spectral measure and assessing the SUA this regime is an interesting avenue for further research. Note however that this approach only concerns the predictor statistics and is not derived in weight space, thus one limitation of the approach that we anticipate is that it might be difficult to characterize (strong) feature learning from this standpoint.
considered (see Theorem 3.3). As result, our integral
Although we don't tt have an analytical formula for the limiting spectral measure, it is still possible to numerically compute the strictly positive support of the random matrix KP P.N.No (X, X) and use our integral forms equation 7 and equation 8 to estimate the predictor of trained BNNs in this regime.
8

---

Published as a conference paper at ICLR 2025
18
ratas
Palmee
0.0
20
420 550
520
1000
168th M
Figure 1: Comparison with Li & Sompolinsky (2021) in the lincar-width regime. On the left, the linear setting on our synthetic dataset with No 500 and P 200. On the right, the nonlinear setting on the subset of MNIST. In both cases, the blue line computed using the fixed-point equa- tion and the orange dots are the ratio between the result of our integral estimator equation 3 and the variance of Bayesian linear regression (respectively, NNGP regression) on the left (respectively, on the right). In the nonlinear case, we use large width N = 10000 to estimate the NNGP kernel matrix for ReLU.
4 EXPERIMENTS
We consider synthetic dataset generated by a multivariate Gaussian ~ NNO0, 1 INo) to which we apply a linear teacher and noise y = 3 x + with (O, of) and B (1,..., 1)T. We also consider a subset of MNIST restricted to classes "0" and "1" of size P 105 and with No 784 pixels per image.
Ng
Our first experiment verifies that our estimators coincide with the predictions theory in the linear-width limit both for a single hidden-layer network with ReLU activations and a linear network with a hidden layer. We computed the renormalisation factors using the fixed- point equation 4 and used equation 2 and equation predictor in our approach. To compute equation 2 and equation we first computed the Marchenko- Pastur maps of the empirical spectral distributions (of the NNGP kernel) by solving numerically the Marchenko- Pastur fixed-point equation in the Stieltjes transform space (Marchenko & Pastur, 1967; Fan & Wang. 2020); then, we relied the SUA estimate the integral forms. In second experiment, we simulated the sublinear-width regime P x N No (for which the renormalisation theory breaks, see Figure 14 in Li Sompolinsky (2021)) using small value of N, (thus making ag high). We compared our estimators for the regime as described in the previous section with the predictions of BNNs trained with variational inference using the library Pyro (Bingham al., 2019). For the spectral distribution, we computed the strictly positive support of the empirical spectral distributions by sampling and diagonalising the empirical kernel matrices several times and shuffling the eigenvalues; we continued to use the SUA for eigenfunctions.
renormalisation
estimate the mean and the variance of the
As shown in Figure 1, our estimates align with the renormalisation theory in the linear-width limit. As shown in Figure 2, for a regime where the renormalisation theory is inaccurate, our estimators provide reasonable matches to the actual predictions. These results suggest that our estimates are better suited regimes where key assumptions of the renormalisation theory do not hold. Sources of discrepancies include: the fact that we used finite values of P, N. No (whereas the theory is only exact in the limit); the SUA may not be fully accurate in this configuration; the limiting spectral measure may not be nonrandom.
5 CONCLUSION
In this paper, we have explored bridges between BNNs trained under interesting idealised limits and kernel theory, which enable extension of the renormalisation theory non-linear networks. From a practical standpoint, our theory offers a new way to estimate the prediction of BNNs with better accuracy in the sublinear-wied regime. Finally, hope that the theory developed here will

---

Published as a conference paper at ICLR 2025
++ Iwtconlloirrrii
0.03
250
600
1000 220 Nicth
2000
Wilthh
Figure 2:
Sublinear-widre regime. Mean and variance of the predictor against the width of the synthetic dataset with / 200 and No 40. In both cases, the
single ReLU hidden-layer blue line is computed using the probabilistic predictions a BNN trained with variational inference on the synthetic data, and the orange dots correspond to integral estimates.
motivate further research on the application of existing kemnelthe results in the context of BNNs.
REFERENCES
Laurence Aitchison, Adam Yang, and Sebastian W Ober. Deep kernel processes. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 130-140. PMLR, 18-24 Jul 2021. URL hwt.h:tt///.mcm httpss//ppccceedeeeeee mlr www.sppppssseee.e. paress/v139/aiiee http:///////acciisniaa.aa htm...
S. Ariosto, R. Pacelli, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo. Statistical mechanics of deep learning beyond the infinite-width limit, 2023.
C. H. Baker. The Numerical Treatment of Integral Equations. Oxford University Press, 1977.
Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal rors and phase transitions in high-dimensional generalized linear models. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pp. 728-731. PMLR, 06-09 Jul 2018. URL https: http///eoccceeeeeeeeeeeee mlr pww.sspesss.... https///5//b/bbrbb
html.
Eli Bingham, Jonathan P. Chen. Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. Journal of Machine Learning Research, 20(28):1-6, 2019. URL http:// jmlr
jmlrr www.///gpee.eee... aaaerr//22// 18-0003.ttt..
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statis- ticians. Journal of the American Statistical Association, 112(518):859 877788997777777.88888 apr 2017. doi: 10. 10.10800012221552211118888 2017.
URL bitpss://dii.. http://ooo.orgg//////// org/ wttpr///.............. 10011122222111
David Bosch, Ashkan Panahi, and Babak Hassibi. Precise asymptotic analysis of deep random fea- ture models. In Gergely Neu and Lorenzo Rosasco (eds.), Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 4132-4179. PMLR, 12-15 Jul 2023. URL https: http::://prcceeeee html.
htttp:://prcceedingg hRtttps::/proceeee
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model align- ment explain generalization in kemel regression and infinitely wide neural networks. Na- ture Communications, 12(1), may 2021. doi: 10. 10.10088/444447-001-330---- URL https: //doi. o
10

---

Published as a conference paper at ICLR 2025
Xiuyan Cheng and Amit Singer. The spectrum of random inner-product kemnel matrices. Random Matrices: Theory and Applications, 02(04))1350010,, 2013, doi: 10.1442222000326333550000
URL https::p//// /w..ooioooooo...oo org. htt.......2//22122211331333
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural informa-
tion processing systems, 22:342-350, 2009.
Hugo Cui, Florent Krzakala, and Lenka Zdeborova. Bayes-optimal learning of deep random net- works of sitensive-width.. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Con- ference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 6468-6521. PMLR, 23-29 Jul 2023. URL https:/ httpss//proceeeee /ptppppeeoogga.p///ee/pep.e...ee mlr wwwps/eee2222. cui23b. html.
Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M. Lu, and Bruno Loureiro. A ran- dom matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities, 2024. URL https: aarxivivivv.////i//.aa//aaaaavva org/ abs / 2410. 18938.
A. Daniely, Frostig, and Singer. Toward deeper understanding of neural networks: The power of initialization and dual view on expressivity, Advances in Neural Information Processing Systems, 29, 2016.
Ouns Harzli, Bernardo Cuenca Grau, Guillermo Valle-Pérez, and Ard A. Louis. Double-descent curves in neural networks: A new perspective using gaussian processes. Proceedings of the AAAI Conference on Artificial Intelligence, 18366:11856-1184 Mar. 2024, doi: 10. (0.1169//aai.v3/0 v38i10. 29071. URL https: jojs aaai. org/ index. php/AAAT/ www.Aa/aaailleee.omeeoo httpe//wwww//////////// 20711.
A. Engel, Otto von Guericke, and C. Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2012.
Z. Fan and Z. Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Zhou Fan and Andrea Montanari. The spectral norm of random inner-product kernel matrices. Probability Theory and Related Fields, 173:27-85, 2015.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
J.S. Geronimo and T.P. Hill. Necessary and sufficient condition that the limit of stieltjes transforms is stieltjes transform. Journal of Approximation Theory, 2002.
Boris Hanin and Alexander Zlokapa. Bayesian interpolation with deep linear networks. Pro- ceedings of the National Academy of Sciences, 120023))e2301344512,, 2023. doi: 10. 0073/ppas.230134512 URL https: /wwp///wwww//////////////.////// ppw.ppaasssss..pp.o.o org/ wtt///////bbb/o///e 10..0...aa........ 2301345120.
Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clement Hongler, and Franck Gabriel. Ker- nel alignment risk estimator: Risk prediction from training data. In H. Larochelle, Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural formation Processing Systems, volume 33, pp. 15568-15578. Curran Associates, Inc., neurips cc/ pdff.
2020. URL file/b httee//2386522577777411711123333 41w.4441777711122223223330003
Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38(1), feb 2010. doi: 10. 1/1214/0--0 URL https::////.. /oii org/10. 1211222220F0 200222008-066-95-- a0s648.
Lec, S. Schoenholz, Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33, 2020.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks gaussian processes. In International Confer- ence on Learning Representations, 2018. URL https: httpss://pperreveek net /forumiid- B1 EA-M-O7.
11

---

Published as a conference paper at ICLR 2025
Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The back- propagating kernel renormalization. Physical review X, 11(3), 2021.
Fanghui Liu, Zhenyu Liao, and Johan Suykens. Kemel regression in high dimensions: Refined analysis beyond double descent. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceed- ings The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 649-657. PMLR, 13-15 Apr 2021. URL https: http://procceee i ngs ml r. press v1 / iullb. html.
Giacomo Livan, Marcel Novaes, and Pierpaolo Vivo. Introduction to Random Matrices. Springer International Publishing, 2018. doi: 10. 10.1007/978-3-311-70888 URL https: //o. .ecooioorr... 10. 10.100782F990
319 00855-C.
Yue M. Lu and Homg-Tizr Yau. An equivalence principle for the spectrum of random inner-product kemel matrices with polynomial scalings, 2023.
Martin Magris and Alexandros Iosifidis. Bayesian learning for neural networks: algorithmic survey, Artificial Intelligence Survey, 2023. doi: wttp://.........0000//0000002220
Antoine Maillard, Emanuele Troiani, Simon Martin, Florent Krzakala, and Lenka Zdeborová. Bayes-optimal learning of an extensive-width neural network from quadratically many samples, 2024. URL https: / arxiv. org/ abw/a//.....// 03733.
V.A. Marchenko and L.A. Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheskit Sbornik, 72, 1967.
A. G. de G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour wide deep neural networks. arXiv preprint arXiv: 1804.1 2018.
J. Mercer. Functions of positive and negative type, and their connection the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, pp. 209415 446, 1909. doi: http://doi.org/ 10. 1909,0016
James A. Mingo and Roland Speicher. Free Probability and Random Matrices. Springer, 2017. doi: http://doi.org/I1 1007/978- 1-4939-6942-5.
H.Q. Minh, P. Niyogi. and Y. Yao. Mercer's theorem, feature maps, and smoothing. In Learning Theory, pp. 154 168. Springer, 2006.
E.H. Moore. On the reciprocal of the general algebraic matrix. Bulletin of the American Mathemat- ical Society, 1920.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Carl Rasmussen. A practical monte carlo implementation of bayesian learning, In D. Touretzky, M.C. Mozer, and M. Hasselmo (eds.), Advances in Neural Information Processing Systems, vol- ume 8. MIT Press, 1995, URL 20000//82220000002000ee222232332 paper/1995/ file/ httpe///95///eeee222222222222232
neurips wwteirripscccppppppp wwtpccppppppplllepee
C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Saburou Saitoh and Yoshihiro Sawano. Springer, 2016. doi: https://doi.org/10..00 978-981-10-0530-5.
Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference Machine Learning. volume 202 of Proceed- ings of Machine Learning Research, pp. // proccedding ngs. ml press v202 schroder23a..... html.
PMLR, 23-29 Jul 2023. URL https:
12

---

Published as a conference paper at ICLR 2025
H. S. Seung and H. Sompolinsky. Statistical mechanics of learning from examples. Physical Review
A., 45, 1992.
James Simon, Madeline Dickens, Dhruva Karkada, and Michael Deweese. The eigenlearning framework: A conservation law perspective kernel ridge regression and wide neural net- works. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:
net :
Alexander van Meegen and Haim Sompolinsky. Coding schemes in neural networks learning clas- sification tasks, 2024. URL httttt:////xxxly http://aaavvvvaa//e orooorrg////oo
16689.
E. Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathe- matics, 62, 1955.
Adam X. Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. theory of representation learning gives deep generalisation of kernel methods. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2023. URL https: httpppptoooon....e/oo http:///ooooonnn..........// w hamg////222///2kkk.k./ html.
PMLR, 23-29 Jul
Jacob A. Zavatone- Veth, William L. Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep bayesian linear regression. Phys. Rev. E, 105:064118, Jun 2022. doi: 10.1 10. 105.064115 URL https:/ //l:://lnnkk aps. ort.
PhysRevE. 105. 064118
13

---

Published as a conference paper at ICLR 2025
A PROOF OF THEOREM 3.4
We calculate the conditional expectation (f)(x, X,y, y. )) and variance ((6f) ,,,, 6) of the predictor by marginalising over the readout weights W
(0'x', - / w 6(0, 610.X)
(x1)) = / | 60, (0,x))
O)dW - ((,,,,,,,)))
where pl (W4 ,,,,, θ can be expressed by Bayes rule using Gaussian likelihoods. The result can be expressed analytically and yields the same prediction as GP regression with prior GP(O, K N.o::
(f)x,, X,, 0) pk..N.N. (x",
No (x.", X)I K
N.N.No (X. (X.X))
(x))) Kg N.N.No (x' xx) K.
[k No (x", X)]
K P.N.No
(X. X)]
kg
P.N.No (x' X).
To marginalise over θ ~ X. y), we perform the change of variables Q + (Φ* 4, A), re- lying on the fact that all quantities of interest involving O can be expressed in the limit solely using eigenvalues and eigenfunctions, namely Ko(X,X) T ke(x", X) = $A and
Ke(x",x") ,x*) T A Since + R Rx has orthogonal rows, o¹ x (Φ -1 and
क. ($ -1 Φ. This allows us to express the mean and variance of the predictor as follows:
= (5) 39/= / (5. "TTTTTT"" π A -'-bly) p(A, क a X,x). P(ylA, D 2. xxxx)
dAd
X)
((688)] -0'"" T -'
P(y A, 4, Φ-, X, d
p(A, $ 9 X,
(8)
p(y(X)
Furthermore, it holds that p(A, 4 x
= P
A, X, x" (P)AX,
and also p(y A, D Φ. X, X,x*)
p
A.X.x if pf
4 A X. x*) O, which yields:
2
( (-= / ( T ΛΦ क
Tt Tー p(A(X,x') ply. 1 + A X,
dAd
ply X)
((617) =/
// /100' A x-
A -0"
A x'
p(A x*) ply, 4, Φ. | X. )*) '
p(y X)
where the integral over A restricted to segments where P(A X, x²) + 0 and the integrals over p(h, (b, A xxx We obtain equations equation 2 and equation 3 by replacing d& and dl by standard Gaussian matrix measures and the density of A by the spectral measure.
& and 4 are restricted to where p(
14


Reviews:
[{"review_id": "CnsolpbkYQ", "reviewer": "Reviewer_LMnA", "summary": "This paper studies bayesian neural networks in the linear ($P/N = \\text{constant}$ for data $P$ and width $N$) regime and sub-linear ($P/(N N_0) = \\text{constant}$) regime. The authors operate under a spectral universality assumption, which treats the eigenfunctions of the limiting kernel as random with a covariance determined by the posterior kernel. The authors argue that this spectral universality assumption is logically equivalent to a kernel renormalization theory which was derived for deep linear networks. This kernel renormalization is a scale shift in the kernel by a variable $u$ which needs to be solved for self-consistently. In the sublinear regime, the kernels are rank-deficient which the authors acknowledge by allowing for a singular spectrum and utilizing the pseudo-inverse.", "questions": "1. What do the authors suspect would happen for networks in the $N \\gg P$ regime with mean-field / $\\mu$P scaling that preserves strong feature learning? In this case, [theory](https://arxiv.org/abs/2205.09653) predicts that the kernels are non-random but the preactivations can become non-Gaussian as was also shown in the [Bayesian setting](https://arxiv.org/abs/2406.16689). Do the spectral universality assumptions break down?\n2. What do the authors think of the spectral universality assumption for cases where the matrices [pick up low rank spikes](https://arxiv.org/abs/2410.18938) such as learning single or multi-index models.  Do spiked kernels violate the kernel renormalization theory?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper studies the important problem of characterizing feature learning in nonlinear Bayesian neural networks and provides an original idea to apply a spectral universality idea to characterize feature learning. It further aims to justify some of the recent applications of kernel renormalization theory to nonlinear networks.  Showing that the spectral universality assumption implies and is implied by the kernel renormalization picture is an interesting contribution. The authors also provide a few experiments to support their claims.", "weaknesses": "However, the spectral universality assumption is not proven directly and the distribution of kernel eigenfunctions in either the proportional regime or the sublinear regime has not been characterized outside of the spectral universality assumption. The experiments are somewhat limited. I have a number of questions below, which if addressed could lead me to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oJgDU5sXRN", "reviewer": "Reviewer_U5Zu", "summary": "The paper gives integral formulas describing the outputs of BNNs in the linear and sublinear width regimes.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical contributions are extremely strong, especially around the integral formulas describing the outputs of BNNs in the linear and sublinear width regimes.", "weaknesses": "The primary weakness is around the experimental results.  These are restricted to MLPs on very simple datasets.  Though I would be happy to consider an argument that the results should generalise and/or that it would be prohibitively difficult to get results outside this setting.\n\nThe experimental results are presented very poorly.  For instance:\n* The plots do not have labelled x and y-axes.\n* The legends are very confusing. As an example: \"on the left (respectively, on the right).\"\n\nThere are also numerous prior works around deep kernel processes and machines that would be worth discussing in the related work:\n* https://arxiv.org/abs/2010.01590\n* https://arxiv.org/abs/2108.13097\n\nwhile this work uses a very different theoretical approach, it ultimately addresses similar conceptual issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q8e4aCFk5Q", "reviewer": "Reviewer_H2W1", "summary": "The authors propose combining ideas from random matrix theory to evaluate limits of infinite width Bayesian neural networks (such as the classic work by Neal (1996)) when the number of observations also goes to infinity as well as the number of input dimensions. The paper provides novel claims of the behavior of the limit in sublinear regimes.", "questions": "1. For Figures 1 and 2, how many simulations are the authors using to obtain the error bars? Also, is there any uncertainty around the blue lines in the same figures? If so, it should be reported.\n2. Can the authors reconcile the differences between the simulations in Figure 2 between their proposed estimates and the blue lines? \n3. In Theorem 3.3, the authors bring in the notation: \n$\\mathbf{x}\\sim \\lim\\_{{N_0}\\to\\infty} \\mathbb{P}\\_{N_0},$ without having defined $\\mathbb{P}\\_{N_0}$. \nDoes $\\mathbb{P}\\_{N_0}$ correspond to $p\\_\\mathbf{x}$ for a specific dimension of $N\\_0$?\n4. The authors emphasize the rectified linear unit (ReLU) activation function (deservedly so). However I wonder if this approach can also work for other activation functions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Using Marchenko-Pastur to evaluate this type of limits, seems like a good idea. Combining these approaches with Mercer's theorem seems valuable. The ideas of random matrix theory in general seem underutilized in the ML community.", "weaknesses": "1. It is not immediately clear how the integrals in Theorem 3.4 can be explicitly evaluated. \n2. Notation for $\\Phi^*$ in Theorem 3.4 was never defined.\n3. Between lines 300 and 303, the authors claim that for __small__ datasets the computations would not be too computationally intensive. However, in footnote 3 (p. 6), they claim that the limit can be approximated using __large__ objects. It is not completely clear what has to be large or small for the evaluations to be feasible. \n4. Section 4 appears to be incomplete. It is not evident from the writing which of the figures correspond to the two examples they mention at the beginning of the section. \n5. In the first experiment they mention, with the \"linear teacher\", the authors do not specify how the $\\beta$ is defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)