Uncertainty Quantification with the Empirical Neural Tangent Kernel
Authors: Joseph Wilson
Chris van der Heide
Liam Hodgkinson
Fred Roosta
Venue: NeurIPS 2025 poster ()
Keywords: Neural Tangent Kernel
Bayesian Deep Learning
Uncertainty Quantification
Neural Network
Primary area: probabilistic_methods
TL;DR: Scalable, post-hoc, Bayesian uncertainty quantification method using an ensemble of linearised networks.

Abstract: While neural networks have demonstrated impressive performance across various tasks, accurately quantifying uncertainty in their predictions is essential to ensure their trustworthiness and enable widespread adoption in critical systems. Several Bayesian uncertainty quantification (UQ) methods exist that are either cheap or reliable, but not both. We propose a post-hoc, sampling-based UQ method for overparameterized networks at the end of training. Our approach constructs efficient and meaningful deep ensembles by employing a (stochastic) gradient-descent sampling process on appropriately linearized networks. We demonstrate that our method effectively approximates the posterior of a Gaussian Process using the empirical Neural Tangent Kernel. Through a series of numerical experiments, we show that our method not only outperforms competing approaches in computational efficiency--often reducing costs by multiple factors--but also maintains state-of-the-art performance across a variety of UQ metrics for both regression and classification tasks.

Uncertainty Quantification with the Empirical Neural Tangent Kernel
Joseph Wilson
Chris van der Heide
School of Mathematics And Physics
Dept. of Electrical and Electronic Engineering University of Melbourne chris. vdh@gmail.cmm com
University of Queensland
joseph. wilson1 Seroonnnngnnnnnttttcceeme edu.
Liam Hodgkinson
Fred Roosta CIRES and School of Mathematics And Physics University of Queensland fred. roostalqu edu au
School of Mathematics and Statistics
University of Melbourne
Ihodgkinsso@@nnimell edu.
Abstract
While neural networks have demonstrated impressive performance across various tasks, accurately quantifying uncertainty in their predictions is essential to ensure their trustworthiness and enable widespread adoption in critical systems. Several Bayesian uncertainty quantification (UQ) methods exist that are either cheap reliable, but not both. We propose a post-hoc, sampling- overparameterized networks at the end of training. Our approach constructs effi- cient and meaningful deep ensembles by employing a (stochastic) gradient-desee descent sampling process on appropriately linearized networks. We demonstrate that our method effectively approximates the posterior of a Gaussian Process using the empirical Neural Tangent Kerne. Through series of numerical experiments, we show that our method not only outperforms competing approaches in com- putational efficiency-often reducing costs by multiple nuttors-but also maintains state-of-the-arr performance across a variety of UQ metrics for both regression and classification tasks.
UQ method for
Introduction
Neural networks (NN) achieve impressive performance wide array of tasks, in areas such as speech recognition (Nassif et al. [019 Abdel-Hamid et al.) 2014: Deng et al., 2013),, image classification (LeCun et al. 1998, Krizhevsky et al. 2012 He et al. 2016), computer vision (Redmon
сt all, 2016 Redmon & Farhadi 2017), and language processing Vaswani et al. 2017 Ray 2023
Devlin et al. 2019), often significantly exceeding human performance. While the promising predictive and generative performance of modern NNs is evident, accurately quantifying uncertainty in their predictions remains an important and active research frontier (Abdar al. 202T). Models are often over-confident in predictions on diitriiitttiiiioo (OoD) inputs (Guo et al. (207)) and sensitive to determine when it
distribution-shift. (Ford al. 2019). By quantifying model's uncertainty, fails to provide well-calibrated predictions, indicating the need for additional training (possibly on more diverse data) or even human intervention. This is vital for deploying NNs in critical applications like diagnostic medicine and autonomous machine control (Nemani et al. 2023b).
An array of uncertainty quantification methods for NNs exist, each with benefits and drawbacks. Frequentist statistical methods, such as conformal prediction (Vovk et al.) 2005)] [Papadopoulos et all, 2002) Lei & Wasserman] (2014), create prediction intervals through parameter estimation and probability distributions on the data. While currently considered state-of-the-arr the main trawback
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

200
200
200
0
200
2000
200
-5.0
0.0
2.5
-5.0
-2.5
2.5
5.0
-2.5
5.0
LLA
NUQLS
200
200
200
0
-200
-200
-200
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
2 I-layer MLP, trained on data (red) lying on y = x² (black), with Gaussian noise added. The methods' mean predictors (blue) 3:3e (green) are shown, where o² is the variance estimated via each method. We see that NUQLS performs well on this task.
Figure I: Comparison of various Bayesian UQ methods (see Section
is that conformal prediction is data-hungry, requiring a large hold-out set. Several works Altieri et all (2024); Dadalto al., (2023); Granese al. (2021) employ the feature-space representations of the network to quantify the uncertainty of test predictions, through kernel densities (Kotelevski) all 22022) Gaussian Discriminant analysis (Mukhoti et al. 2023), non-constant mapping functions (Tagasovska & Lopez-Paz 2219), etc. The interpretation of uncertainty in these works is related to risk of misclassification; these methods generally perform very well detecting OoD points, While both conformal predictions and feature-space works are important, we limit our scope to Bayesian methods in this paper. Though Bayesian methods can suffer from prior misspecification (Masegosa 2020) and computational burdens, the predictive and posterior distributions of model are very natural frameworks for quantifying the uncertainty and spread of possible values of the model.
Unfortunately, largely due to the curse of dimensionality, existing Bayesian methods are very expensive compute, and provide poor approximations to the predictive distribution (Folgoc et al.. 2021). This results in a suite of methods that require excessive approximations to scale to large problems (Daxberger al. 2021), often the cost of theoretical underpinnings, or necessitating modifications to the network itself (He et al. 2020).
important tools in Bayesian machine learning (ML) that can directly capture the epistemic (model) uncertainty of predictions, and arise as large-width limits of NNs (Neal 1996). However, naïve GP training scales cubically in the number of training datapoints, necessitating approximations for modern applications. Neural Tangent Kernels (NTKs) (Jacot et al. 2018) describe the functional evolution of NN under gradient flow, and naturally arise in the analysis of model quality (Hodgkinson et al. 2023). Due to their deep connection to NNs, these covariance functions enticing potential tools for UQ of their corresponding NNs.
Gaussian Processes (GPs) (Rasmussen & Williams 2005
Motivated by this, we present a UQ method for a trained, over-parameterized NN model, wherein we approximate the predictive distribution through an ensemble of linearized models, trained using (stochastic) gradient-descee For certain loss functions, this ensemble samples from the posterior of GP with empirical NTK.
Contributions. Our contributions are as follows:
1. In Section 3.1
we present a Monte-Carlo sampling based UQ method to approximate the predictive distribution of NNs, called Neural Uncertainty Quantification by Linearized Sampling (NUQLS). Our method is lightweight, post-hoc, numerically stable, and embarrassingly parallel.
2. Under certain assumptions, Section 3.2 details the convergence of NUQLS to the predictive
distribution of a GP with an empirical NTK kernel, providing novel perspective on the connection between NNs, GPs, and the NTK.
3. On various ML problems in Section 4 we show that NUQLS performs as well as better than leading UQ methods, is less computationally expensive than deep ensemble, and scales to large image classification tasks.

---

4. In Section 4.4 we introduce a novel metric for evaluating the quality of UQ methods for classifi- cation tasks. This metric more directly measures the quality of UQ methods than existing UQ metrics.
Remark 1.1 (Necessity of Contribution 4.). Evaluating the quality of UQ methods is non-trivial, There exists in the literature lack of suitable framework for evaluating the quality of UQ methods in classification settings. Common metrics such as Negative Log-Likelihood (NLL), Expected Calibration Error (ECE) (Nciini al. 2015) and Area Under the Curve of the Receiver Operating Characteristic (AUCROC) are actually metrics for prediction quality, or are based on flawed surrogates for uncertainty. Further, it was shown in Abe et al. (2022) that the goal of Bayesian methods should not be to improve predictive ability; performance gains will be attained more economically by choosing a larger model class. Instead, Bayesian methods should seek to accurately quantify the uncertainty of a model, by computing the predictive variance. As uncertainty is not measurement that can easily be shown to be well-calibrated, one requires a more qualitative approach to evaluating the performance of UQ model. This is the motivation for the graphical technique we introduce in Section 4.4 for comparing the quality of UQ estimates for multi-class classification. For an in-depth discussion of these points, please see Appendix C
Notation. Throughout the paper, we denote scalars, vectors, and matrices as lower-case, bold lower-case, and bold upper-case letters, e.g. c, 0 and K, respectively. For two vectors v E TR? and E IRP, their Euclidean inner product is denoted supervised learning, which involves a function f: Id x IP + RS, assumed sufficiently smooth in its parameters E IR", a training dataset = {x, } = [x; y: C Rd x R and a loss function : x R - o0).
(v, w) = v'w. We primarily consider
The process of training amounts to finding solution, 0 the optimization problem ming En= (f((,, yi) where R()) is a regulariser. For a kernel function K R xxx Rexe, we define Kx,xx E RRnexne where the (i, (block) element is ((x;, (xx;,xx) E Box, Addition- K(Xn,x)) E Rexx with K}, Kx,x, For matrix J, its Moore-Penross pseudo-inverse denoted by Jt.
ally, we define Kx,x A [K(x1,x)
2 Background
Bayesian Framework. Parametric Bayesian methods admit access to distribution over predictions f(0, x²) for unseen test points x", through the posterior distribution p(0 x p(0))((())))))) and the predictive distribution p(y* Ixx,, D) = ply (f(O, x* ()(())
where p(0) is the prior, and p(D(0) is the likelihood function evaluated on the training data. Both the posterior and the predictive distributions are computationally intractable in all but the simplest cases. In our setting, their calculation involves very high-dimensional integrals, which we can approximate through a Monte Carlo (MC) approximation p(y"|x", D) x 1/S P(y approximation to the posterior distribution. The effectiveness of classical Markov Chain Monte Carlo (MCMC) methods in posterior sampling diminishes in this setting due to the curse of dimensionality, limiting the tractable techniques available with theoretical guarantees. This limitation necessitates coarser approximations for estimating the posterior (0), leading to the emergence of the following Bayesian methods for posterior approximation.
x")) for O, q()), where q()) an
Proposed in (Gal & Ghahramani 2016). Monte Carlo Dropout (MC-Dropout) takes a trained NN, and uses the dropout regularization technique at test time to sample 5 sub-networks, {f(O,, x= ==1:5 as an MC approximation of the predictive distribution. MC-Dropout is an inexpensive method, yet is unlikely to converge to the true posterior, and is erroncously multi-modal (Folgoc et al. (021].
In Variational Inference (VI) (Hinton & Van Camp, 1993 Graves 2011). tractable family of
approximating distributions for (() is chosen, denoted by qu (0), and parameterized by tb. The optimal distribution in this family is obtained by finding v6 that minimizes the Kullback-Leibler
divergence between qop (0) and p)) To be computationally viable, mean field and low-rank covariance structures often required for (0).
The Laplace Approximation (L-A) (MacKay 1992. Ritter et al. 2018 a tool from classical statistics which approximates the posterior distribution by Gaussian centered at the maximum posteriori
3

---

solution (MAP) with normalized inverse Fisher information covariance. This is justified by the Bernstein-von Mises Theorem (Van der Vaart [000 pp. 146), which guarantees that the posterior converges to this distribution in the large-data limit, for well-specified regular models. However, NNs are often weerpparameterieed, and the regime where n - oo with fixed p no longer valid or a reasonable reflection of modern deep learning models (De Bortoli & Desolneux 2022).
These limitations are acknowledged but seldom discussed by the Bayesian Deep Learning (BDL) community, which tends to view this as an additional layer of approximation rather than a modelling error, leading to the development of synonymous LA-inspired methods. However, we will show that such methods typically perform poorly compared to deep ensembles, which are often excluded from comparisons.
We can evaluate the posterior and predictive distribution in the LA using the linearization of f(x. 0) around the MAP solution. This approach is known the Linearized Laplace Approximation (LLA) and typically delivers better performance than LA (Immer et al., 2021). LLA generally requires reduction a subset of parameters or approximations of the covariance structure (Martens & Grosse 2015) to scale, at the cost of performance. Recent work (Antorán et al. 2022 Ortega et al. (023] has enabled LLA become more scalable with better performance for larger models and datasets.
Deep Ensembles (DE) [Lasshminarayanan et al., 2017) are comprised of S networks that are independently trained the same training data, with different initializations, leading to col- lection of parameters {0a;s = 1...,.1 S}. At test time, our predictive distribution becomes ply x. D)= = 1/5 P(y|f(0, x)). Despite their simple construction, DEs are able to obtain samples from different modes of the posterior, and are often considered state-of-the-aae art for BDL (Hoffmann & Elster 2021). However, due to the often large cost of training performant neural networks, deep ensembles of reasonable size can undesirably expensive to obtain.
Stochastic Weight Averaging Gaussian (SWAG) (Maddox et al. 2019) takes a trained network and undergoes further epochs of SGD training to generate a collection of parameter samples. A Gaussian distribution with sample mean and low-rank approximation of the sample covariance is then used to approximate a posterior mode.
Gaussian Processes. A GP is stochastic process that is defined by mean and a kernel function. GP models the output of a random function f Rd RR, at a finite collection of points x, as being jointly Gaussian distributed. Conditioning on training data D, it generates a posterior predictive distribution p(f(x.)[D) at a test point X.. For example, in regression settings where 1, with the mean and kernel functions μ Rd + I and K Rd Rd + E, as well as the observations y ~ (x((xx) o²), there is a closed form expression for the predictive distribution, (D) ~ N (H(x.),, (x.)). where
mxx)) x [ + I (y (a(x)) + p(x.)
o (x.) = n(x,, x.) kx. Kx.x + I²1 kx...
for y A [yi....,yy]] and (x() Jul (xxx)).... x T. GPs can yield impressive predictive results when suitable kernel is chosen (Rasmussen 1997). However, forming the kernel and solving linear systems makes GP computations intractable for large datasets. Approximations such as sparse variational inference (Titsias 2009). Nyström methods (Martinsson & Tropp 2020), and other subspace approximations (Gardner et al., (0188) can alleviate the computational burden; however, these approximations often result in a significant decline in predictive performance.
Neural Tangent Kernel. Under continuous time gradient flow, can be shown that a NN output f(.,0) 0) Rd + Re undergoes kernel gradient descent, namely 0,f(x, 0,) Ke, (x) (re(f(x, Oc).y.). where
Of 00 xx 0),
Of 00 (y, 0) E Rexc
y) A
the empirical NTK (Jacot et al. 2018). As the width of a network increases, the empirical NTK converges (in probability) to a deterministic limit, sometimes referred to as the analytic NTK, that
independent of the network's 's parameters. Lee et al. (2019) showed that in this limit, the network
acts according to its NTK lincarization during gradient descent (GD) training. This parameter

---

independence results in loss of feature learning in the limiting regime (Yang & Hu 2021). However, for mhite-width (Fot et al. (2020) empirically showed that the empirical NTK becomes "data- dependent" during training. Since we focus exclusively on the finite-widis wiith regime, we refer to the empirical NTK simply as the NTK.
Our method NUQLS shares notable similarities with, and exhibits distinct dif- ferences from, several prior works, such as Sampling-LLA. LLA, Bayesian Deep Ensembles and local ensembles. Due to the breadth and depth of this discussion, we relegate the related works discussion to Appendix
Related Works
3 NUQLS
We now present Neural Uncertainty Quantification by Linearized Sampling (NUQLS), our post-hoc sampling method for quantifying the uncertainty of trained NN. We begin by presenting the motivation and a high-level overview of our method. Subse- quently, provide theoretical justification, demonstrating that, under specific conditions, the NUQLS samples represent draws from the approximate posterior of the neural network, which is equivalent GP defined by the NTK.
Algorithm NUQLS
Input: number of realizations S, weights 0.
for = to S do
o,., 0 + Zy. where Ey (0, ?²1)
0 - Run (stochastic) GD from Oo.. to
(approximately) solve 3 and obtain 0
end for return {f(0z, .)
3.1 Motivation and High-level Overview
NNs are often over-parameteriizd, resulting in non-uniqueness of interpolating solutions, with sub- manifolds of parameter space able to perfectly predict the training data (Hodgkinson et al. (023). To generate a distribution over predictions, we adopt a Bayesian framework, where the uncertainty in a neural network's 's prediction can be interpreted as the spread of possible values the network might produce for a new test point, conditioned on the training data. To quantify this uncertainty, we can evaluate the test point other "nearby" models with high posterior probability and analyze their range of predictions. To identify such models, we propose using the linearized approximation of the simpler yet expressive surrogate. This approach
original network around its trained parameters can retain, to a great degree, the rich feature representation of the original network while enabling tractable exploration of the posterior distribution. In the same spirit DE, in the overparameterized setting, we can fit this linear model to the original training data, using (stochastic) gradient descent with different initializations, resulting in an ensemble of linear predictors. Not only does this ensemble explain the training data well, but it also provides practical way to estimate predictive uncertainty.
More precisely, let 0 be set of parameters obtained after training the original NN. Linearizing f around 0 gives
r(O. x)
f(0,x r(0.x) + ((,, -0) Ō).
where x(,,x) = (x)(0.))000) e Rexp is the Jacobian of f. Using the linear approximation 2 we consider
min Σ ((f((,xx,.x)
In overparameterized settings, may have infinitely many solutions. To identify these solutions and create ensemble of linear predictors, we employ (stochastie) gradient descent, initialized at zero-mean isotropic Gaussian perturbations of the trained parameter, ê. The pseudo-code for this algorithm is provided in Algorithm [] For given test point x", the mean prediction and uncertainty can be computed using {f(0;, x+) )
Note that while the training cost for linearised network is only slightly higher per epoch compared to standard NN training, each network in the NUQLS ensemble is initialized in neighborhood of a local minimum of the original NN. As a result, NUQLS often requires significantly fewer epochs to converge, leading to an order-of -in---ude computational speedup relative to DE (see Tables and for wall-clock time comparisons, and Appendix for a more in-depth analysis of the computation costs.).
5

---

3.2
Theoretical Analysis
We now establish the key property of Algorithm [ from the approximate posterior of the neural network, which in many cases corresponds to a Gaussian process defined by the NTK. Proofs are provided in Appendix A
under mild conditions. NUQLS generates samples
Suppose O is any solution to Using 0t, one can construct a family of solutions to as
3
0 0² + (1-J)dx) z, Vz ER,
where 0 is the parameters of the trained NN and J = [J x1) JT x,))" E RRcxp, Note that the second term in (4) consists of all vectors the null space of Jx. Since any such OF can be decomposed the direct sum of components in the null space of Jæx and its orthogonal complement, the family of solutions in 44) depends on the choice of 0F. However, under certain assumptions, we ensure that the representation (4) is uniquely determined, i.e., @! can be taken as the unique solution to (3) that is orthogonal to the null space of Jx. More precisely, we can show that, under these assumptions on the loss, o in (4) can be taken as the unique solution to
min Σ ((E(O, s.t. O E Range (J))
(5)
1=1
Lemma 3.1. Suppose the loss, V)) is either:
strongly convex in its first argument, or
strictly convex in its first argument, and a solution to exists.
The problem ( admits a unique solution.
As it turns out, any solution of the form 4 can be efficiently obtained using (stochastic) gradient descent.
Theorem 3.2. Consider the optimization problem and assume x is full row-rank.
(Gradient Descent) Suppose E(f.y) is strictly convex with locally Lipschitz continuous gradient, both with respect to f, and the problem ( admits solution. Gradient descent, initialized at z and with appropriate learning rate, converges to (4).
(Stochastic Gradient Descent) Suppose E(f, y) is strongly convex with Lipschitz continuous gradi- ent, both with respect to f, and any solution to the problem (3) is interpolating. Stochastic gradient descent, initialized at z and with small enough learning rate, converges to with probability one.
We note that the local smoothness requirement in the first part of Theorem 3.2 is relatively mild assumption: for example, it holds if we simply assume that l is twice continuously differentiable. Also, the full-row rank assumption on J in the second part of Theorem 3.2 is reasonable for highly over-parameterized networks; e.g., see Liu et al. (2022).
Now, suppose Jx is full row-rank and the assumption of Lemma 3.1 holds, ensuring the existence of
the unique solution o to 5 Noting Range (Jx) Range(J k),, we can write 4 as
0 - x + (I- J xx) z.
Vz E IR
where w is a vector for which 0 = J w, and Kx JzJJ = K q (x(X.)) E Rexxxe is the Gram
matrix of the empirical NTK I the training data X. Setting z 0 Zg for some Zg, get
0. K X w + (I (I- J @ -zo),
zo).
f(0r, x) f(0,x x) + Krr x (w - (w-Jx0) + (Kx.KK..- - x)) 2o,
(6)
where Kx,x & Kg(X,X)) Rexme Taking Zo to be random variable, we form ensemble of predictors {f(0", ,))2, where each 0* is formed from the projection of a random z onto NJ x)). We require z's distribution to be symmetric, isotropic, and centered 9, as we do not know priori

---

which directions contain more information. We take the maximum entropy distribution for given mean and variance, which is Gaussiar" Hence, we let Zg (O 2 for some hyper-parameter E R.. The expectation and variance of (6) and the distribution of the predictor f(0", x), are then
B(T(O', x)) 1(0,x) x) K x K X
xx
0) f(0,x),
Var Vfr"" x)) = (0, = (Kxx- - x x,
K
x xx
(8)
f(0", x) "P N (N x), 02
(9)
Remark 3.3 (Connections to GP: Regression). For scalar-valued f R2 x Rd + R with quadratic
loss (0, v) = (f(0,x) u)² we can explicitly write OF = Jx f(O,X) Jx where
f(0, X) - [f(e,x)) f(0, x2),
f(0,x,)) x..] and y - [e,.....nn! For za ~ (N we
thus have f(0", x) "RX"x N M(n(0, x), (0, x)) with
m(0,x) x) = kx,x K x 1 xy - f(0, + f(0,x). (², x) = (x(xx,x) kx,x x x kx,x
By the full-rank assumption on the Jacobian, this amounts to the conditional distribution of the following normal distribution, conditioned on interpolation f(O", x = y,
[() 231 [(0),)) -
x) [[(0.2)) x) ,72
K N K X
kx.x fix,x
(10)
Therefore, f(0", follows a GP with an NTK kernel. Conditioning on f(0", = y is reasonable, since by construction f(0', ,X) av f(0", X²), and under the full-rank assumption of Jx,, we have f(0" X) y.
Remark 3.4 (Connections GP: General Loss). Beyond quadratic loss, for general loss function satisfying the assumptions of Lemma 3.1 a clear GP posterior interpretation like that in (I0) may
not exist. Nevertheless, we can still derive related insights. If Ô is an interpolating solution from initial training, which is common for modern NNs, then as long as Jx full row-rank, solving 3
equivalent to finding 0 E IR" such that f(0, X) = X(), i... find O E R such that Jx
0 0.
r(O) x) and o² (0,x) = (Kx. KK xx x )2 as the conditional distribution of (0), conditioned on the event f(0", f(O, i.e., interpolation with E(f(O" x.).y.) for
So we obtain (9) with /=(0, x)
n.
The Punchline. Drawing samples from the posterior ( by explicitly calculating 7 and ( can be intractable in large-scale settings. Moreover, it can be numerically unstable due to the highly ill-conditioned nature of the NTK matrix However, by combining the above derivations with Theorem 3.2 we arrive at the key property of Algorithm it enables efficient sampling from the posterior
Corollary 3.5 (Key Property of NUQLS). With the assumptions of Theorem 3.2 the samples generated by Algorithm represent draws from the predictive distribution in (
Hence, we approximate (7) and (8) by computing the sample mean and covariance of (f(0;,
obtained from Algorithm I By the law of large numbers, the quality of these approximations improves S 30.
Remark 3.6. Loss functions that do not satisfy the assumption of Lemma 3.1 such as the cross- entropy loss, may fail to yield a unique representation of 4 so the above posterior analysis does not apply. However, our experiments demonstrate that Algorithm I can still generate samples that effectively capture the posterior variance (see Section (.44) and posterior mean (see Appendix I..). In cases where 3 lacks solution, Algorithm can still be executed by terminating the iterations of (stochastic) GD early. Investigating the distribution of the resulting ensemble and its connection to an explicit posterior remains a potential direction for future research.
I 'Heavier-talled distributions matching the above criteria, e.g. logistic distrihutions, may improve results. 2 Recall that the condition number of Kx is the square of that of J.x.
7

---

4 Experiments
We now empirically demonstrate the result of Theorem 3.2 as well as compare the performance of method with alternatives on various regression and classification tasks. Implementation details are given in Appendix [] The PyTorch implementation of our experiments is available heeee We have
also released our method package
For additional experimental results, please see Appendix H
Epochs
4.1 Empirical Convergence
10
10
10*
10
We demonstrate empirically the convergence of the predictive distribution of NUQLS to that of an NTK-GP. We take a series of ML.ss with NTK scaling (Jacot et al., 2018), and train these NNs on normalized Gaussian data (a regression task). Under this setting. the condition number of the re- sultant NTK is reasonable ( 10 - 100, compared to = 10 allows us to explicitly compute the predictive vari- ance for the NTK-GP GP for some normalized Gaus- sian test set. We then also compute the predictive variance using NUQLS, and take the l2 norm of the difference between the two variance sets, which term Squared Error of the Variance (SEV). We plot this against number of epochs of training for NUQLS, and the number of NUQLS realizations s. We also plot the average training loss of the linearised networks in the NUQLS ensemble. The values are averaged over 10 random realizations. The results are displayed in Figure 2 Note that the SEV cannot be zero, due to the condition number of the NTK; this affects the solution of the linear system K x = Kx in the computation of the variance for the NTK-GP. However, we see clear convergence of our method to the distribution of an NTK-GP as the ensemble members of NUQLS approach their minima, and as the number of ensemble members increases.
Single layer, width 20, Tanh (10%)
-.- Single-layer, layer, width 20, RelU (10%)
:30
Double-layer, wath 20, ReLU (10f)
loyelee width Tanh (109)
20
200
10
68:
some UCI regression datasets). This
40
SEV
20
10a
10a 10a
10
$
Figure 2: Plot of SEV and NUQLS train loss (A) against (top) number of epochs of training for NUQLS and (bottom) number of NUQLS realizations. Bracketed number is con- dition number of NTK Gram matrix. Mean and 95% confidence intervals are shown from 10 random realizations.
4.2 Toy Regression
We compare the performance of our method on a toy regression problem, taken from (Hernnndez
Lobato & Adams, 2015) and extended in (Park & Blei 2024). In Figure I we take 20 uniformly sampled points in the domain x E [-4,- -2 2 2, 4/. and let y x + e, c (0, 3²). A small MLP was trained on these data and used for prediction. We apply VI, SWAG, LA, LLA, DE and NUQLS to the network to find a predictive mean and uncertainty. Close to the training data, i.e. in [-4,--14 - U [2, we expect low uncertainty; outside of this region, the uncertainty should grow with distance from the training points. VI underestimates, while SWAG and LA overestimate the uncertainty. DE grows more uncertain with distance from the training points, however both NUQLS and the LLA contain the underlying target curve within their confidence intervals. Note that deep ensembles output a heteroskedastic variance term, and were trained on a Gaussian likelihood; in comparison, the variances for LLA and NUQLS were computed post-hoc.
4.3 UCI Regression
In Tables 1 and 99] we compare NUQLS with DE, LLA and SWAG on a series of UCI regression problems. Mean squared error (MSE) and expected calibration error (ECE) respectively evaluate the predictive and UQ performance, with Gaussian negative log likelihood (NLL) evaluating both. See (Nemani et al. 2023a, $4.11) for an explanation of ECE. We see that NUQLS consistently has the (equal) best ECE (except for the Song dataset, where it falls short of the LLA ECE by 0.1%). It has comparable or better NLL than other methods on all datasets, and often gives an improvement on RMSE. Finally, it is the quickest method, often by very significant margin, and it does not not fail
8

---

Table Comparing performance NUQLS, DE, LLA and SWAG on UCI regression tasks. NUQLS performs as well as or better than all other methods, while showing a speed up over other methods; this speed up increases with the size of the datasets. LLA-K denotes LLA with a KFAC covariance structure. Reported time for NUQLS, LLA and SWAG includes the training time for the original NN, with the run-time of the post-hoc method given in brackets.
Dataset
Method NUQLS
RMSE
NLL V
ECE 1
Time(s)
Energy
0.047
-2.400=t.200
0.002=to.o02
8.374 (0.151)
DE
0.218= -1.651=to.784
0.0044tv.ouz
102.244
LLA
0.048+to.008
-2.475
0.004+to.004
8.491 (0.269)
SWAG
0.058-to.016
-1.950
0.080+to.m1
45.306 (37.084)
Kin8nm
NUQLS
0.25221..0
-0.796 0205
0.000
26.570 (0.264)
DE
0. 0.252 0000
-0.914 028
0.002
73.967
LLA
0.260=to.o10
-0.783=0.054
0.001=t..00
38.272 (11.966)
SWAG
0.457=0.144
-0.06==0299
0.054
176.569
(150.263)
Protein
NUQLS
0.623 005
0. 0.209F0..047
0.002
81.264 (1.356)
DE
0.741 050052
0.203 203
0.011
1014.827
LLA-K
0.640
0.458
0.002
89.414 (9.506)
SWAG
0.730=0.044
0.187
0.002
548.88 (468.972)
ResNet9 FMNIST
ResNet50 CIFAR-10
ResNet50 CIFAR-100
0.30
ID Correct
ID Incorrect
00D
8 0.15
0.00
NUQLS DE SWAG MC LLA* BASE NUQLS SWAG MC LLA" BASE NUQLS DE SWAG MC
LLA BASE
Figure 3: Violin plot of VMSP, for correctly predicted ID test points, incorrectly predicted ID test points, and OoD test points. Median is shown, with violin width depicting density. Low variance is expected for ID correct points, and large variance for ID incorrect and OoD points.
on any datasets, like the other methods do. Note that for the two largest datasets, Protein and Song, we required approximations on the covariance structure of L.AA (see (Daxberger al. (02T)) ). A
detailed explanation of the hyper-parameter tuning method for NUQLS is given in Appendix F.1
4.4 Image Classification Uncertainty
We now compare the UQ performance of NUQLS, DE, SWAG, LLA* (LLA with a last-layer and KFAC approximation), and MC-Dropout (MC), on larger image classification tasks. We take variance of the maximum predicted softmax probability (VMSP), for a given test point, as the correct quantifier of uncertainty in this setting (see Appendix for justification).
Figure presents a violin plot of the VMSP for three test-groups: correctly predicted in-distribution (FaShionMNISST, CIFAR- 10) test points, incorrectly predicted in-distribution test points, and out-of distribution (MNIST, CIFAR-100) 100) test points. We would expect that there should be, on average, much smaller uncertainty for ID test points that have been correctly predicted, and larger uncertainty for incorrectly predicted ID and OoD test points. We compare against a completely randomized baseline method (BASE), where we sample 10 standard normal realizations of logits, passed through a softmax. In Table 10 in Appendix H.8. we display the corresponding median and sample skew values for each method in each test group, to quantify the distribution of VMSP for each test set. Ideally, a method should far outperform the baseline, so we can use the median and sample skew difference between method and the baseline as a way to compare different methods. We see that NUQLS outperforms all other methods, including the SOTA method DE. In Appendix HH.7 we provide additional experimental evaluation of NUQLS. ResNet50 trained on both SVHN and ImageNet, displaying the scalability of our method, as well as providing comparison with other competing methods, including Bayesian Deep Ensembles (BDE), Spectral- -oommllid Neural Gaussian Process (SNGP), BatchEnsemble (BE), and Stochastic
Figure 6 we evaluate NUQLS on

---

Gradient Langevin Dynamics (SGLD). Against these competing methods, NUQLS performs the strongest.
Conclusion
We have presented NUQLS, Bayesian, post-hoc UQ method that approximates the predictive distribution of an averpparameterize NN through GD/SGD training of linear networks, allowing scalability without sacrificing performance. Under assumptions distribution reduces to a GP using the NTK. We find that our method is competitive with, and often far outperforms, existing UQ methods regression and classification tasks, whilst providing a novel connection between NNs, GPs and the NTK.
the loss function, this predictive
Limitation. A theoretical limitation of this work is that its connection to the NTK-GP does not extend to loss functions that violate the assumptions of Theorem 3.2, such as the cross-entropy loss. The strong empirical performance of NUQLS classification tasks motivates future research to extend Theorem 3.2 to broader classes loss functions and alternative optimization algorithms. A further limitation of the method is the dependence of NUQLS the lincarization approximation. As can seen Appendix H when the target neural network poorly trained, and hence the loss-landscape is far from flat around the 'trained' parameters, the performance of NUQLS suffers. However, when the network is tranerra,,d, as is common in practical settings, the linear approximation holds and hence NUQLS performs well.
Acknowledgments and Disclosure of Funding
Liam Hodgkinson is supported by the Australian Research Council through a Discovery Early Career Researcher Award (DE240100144), Fred Roosta was partially supported by the Australian Research Council through an Industrial Transformation Training Centre for Information Resilience (IC200100022).
References
Abdar, M., Pourpanah, F., Hussain, Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, Cao, X., Khosravi, A., Acharya, U. R., Makarenkov, V., and Nahavandi, S. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76: 243-297, 2021. ISSN 1566-2535. doi: https://doi.org: 10tg//1..1011000 60016/..nmmn inffus..22000 1.0.0000.08..
Abdel-Hamid, O., Mohamed, A.-r., Jiang, H., Deng, I.., Penn, G., and Yu, D. Convolutional neural networks for speech recognition. JEEE/ACM Transactions on audio, speech, and language processing, 22(10):1533-155 2014.
Abe, T., Buchanan, F. K., Pleiss, G., Zemel, R., and Cunningham, Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35:33646-3366), 2022.
Altieri, A., Romanelli, M., Pichler, G., Alberge, F., and Piantanida. P. Beyond the nonms: Detecting prediction errors in regression models. Forty-first International Conference on Machine Learning. 2024.
Antorán, J., Padhy, S., Barbano, R., Nalisnick, E., Janz, D., and Hernández-Lobato. J. M. Sampling- based inference for large linear models, with application to linearised laplace. arXiv preprint arXiv:2210.04990 2022.
Bassily, R., Belkin, M., and Ma, S. On exponential convergence of SGD in non-convex parametrized learning. arXiv preprint www.:v://11.025500 2018.
Bitterwolf, J., Mueller, M., and Hein, M. In or out? fixing imagenet out-of-distributiin detection evaluation. arXiv preprint arXiv:2306.0003: 2023.
Blondel, M. and Roulet, V. The elements of differentiable programming. arXiv preprint 14660 2024.
10

---

Blundell, C., Cornebise, J.. Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. International Conference on Machine Learning, pp. 1613-1622, 2015.
Bubeck, S. et al. Convex optimization: Algorithms and complexity. Foundations and Trends& in Machine Learning, 8(3-4):231-3577 2015.
Chan, M., Molina, M., and Metzler, C. Estimating epistemic and aleatoric uncertainty with a single model. Advances in Neural Information Processing Systems, 17:109845-109870 2024.
Dadalto, E., Romanelli, M., Pichler, G., and Piantanida, P. A data-driven measure of relative uncertainty for misclassification detection. arXiv preprint arXiv:2306.01710, 2023.
Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34: 20089-20103, 2021.
De Bortoli, V. and Desolneux, A. On quantitative Laplace-type convergence results for some exponential probability measures with two applications. To appear in the Journal of Machine Learning Research, 2022.
Deng, L., Hinton, G., and Kingsbury, B. New types of deep neural network learning for speech recognition and related applications: An overview. International Conference on Acoustics, Speech and Signal Processing, pp. 8599-8603, 2013.
Deng, Z., Zhou, F., and Zhu, J. Accelerated linearized laplace approximation for bayesian deep learning. Advances in Neural Information Processing Systems, 35:2695- 2022.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional trans- formers for language understanding, pp. 4171-4186. Association for Computational Linguisties, 2019. doi: 10. 10.18653/V1/N19- 1423.
Eschenhagen, R., Daxberger, E., Hennig, P., and Kristiadi, A. Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:21 11:22///1,0577 2021.
Folgoe, I.. L.,, Baltatzis, V., Desai, S., Devaraj, A., Ellis, S., Manzanera, O. F. M., Nair, A., Qiu, H., Schnabel, J., and Glocker, B. Is mc dropout bayesian? arXiv preprint arXiv:2 / 2021.
Foong. A. Y., Li, Y., Hemáánezz-Lobtto J. M., and Turner, R. E. 'in-between' uncertainty in bayesian neural networks. arXiv preprint arXiv: 11377 2019.
Ford, N., Gilmer, J., Carlini, N., and Cubuk, D. Adversarial examples natural consequence of test error in noise. International Conference on Machine Learning. 97, 2019.
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution the neural tangent kernel. Advances in Neural Information Processing Systems, 34, 2020.
Franchi, G.. Bursuc, A., Aldea, E., Dubuisson, S., and Bloch, I. Encoding the latent posterior of bayesian neural networks for uncertainty quantification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2027-2040. 2023.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Balcan, M. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York. USA, 20-22 Jun 2016. PMLR. URL https \/proceedings. mlr press/v48/ wtt./////////.... htm
Gardner, J. R., Pleiss, G., Bindel, D.,. Weinberger, K. Q., and Wilson, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
Garrigos, and Gower, R. M. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint ar 2111:2230..1235,, 2023.

---

Granese, F., Romanelli, M., Gorla, D., Palamidessi. C., and Piantanida, P. Doctor: A simple method for detecting misclassification errors, Advances Neural Information Processing Systems, 34: 3669-5681, 2021.
Graves, A. Practical variational inference for neural networks. Advances Neural Information Processing Systems, 24, 2011.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. International Conference Machine Learning, 70:1321-1330 2017.
Havasi, M., Jenatton, R., Fort, S., Liu, J. Z., Snock, J., Lakshminarayanan, B., Dai, A. M., and Tran, D. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610, 2020.
He, B., Lakshminarayanan, B., and Teh, Y. W. Bayesian deep ensembles via the neural tangent kernel. Advances in Neural Information Processing Systems, 33:1010-1022, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021.
Hernández-Lobard J. M. and Adams, R. Probabilistic backpropagation for scalable learning of bayesian neural networks. International Conference on Machine Learning, pp. 1861-1869, 2015.
Hinton, G. E. and Van Camp. D. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5- 13, 1993.
Hodgkinson, L., van der Heide, C., Salomone, R., Roosta, F., and Mahoney, M. The interpolating information criterion for overparameterized models. arXiv preprint rxXiv:2307.07785vl 2023.
Hoffmann, L. and Elster, C. Deep ensembles from a bayesian perspective. arXiv preprint arXiv:2105.13288 132883 2021.
Huang. Z., Lam, H., and Zhang, H. Efficient uncertainty quantification and reduction for pwer-parameterized neural networks. Advances in neural information processing systems, 36: 2023.
Huseljic, D., Sick, B., Herde, M., and Kottke, D. Separation of aleatoric and epistemic uncertainty in deterministic deep neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9172 9179. IEEE, 2021.
Immer, A., Korzepa, M., and Bauer, M. Improving predictions of bayesian neural nets via local linearization. In International Conference Artificial Intelligence and Statistics, pp. 703-711. PMLR, 2021.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems. 31, 2018.
Johnson, R. and Zhang, Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
Karimi, H., Nutini, J., and Schmidt, M. Linear convergence of gradient and proximal-gradient condition. In Machine Learning and Knowledge Discovery
methods under the Folyak-Lojasiewea in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16, pp. 795-811. Springer, 2016.
Khan, M. F. E.,, Immer, A., Abedi, E.,, and Korzepa, M. Approximate inference turns deep networks into gaussian processes. Advances in Neural Information Processing Systems, 33, 2019.
12

---

Kotelevskii, N., Artemenkov, A.. Fedyanin, K., Noskov, F., Fishkov, A., Shelmanov, A., Vazhentsev, A., Petiushko, A., and Panov, M. Nonparametric uncertainty quantification for single deterministic neural network. Advances in Neural Information Processing Systems, 35:36308-36322, 2022.
Krishman, R., Esposito, P., and Subedar, M. Baycsian-torch: Bayesian neural network layers for uncer- tainty estimation, January 2022. /tt/ttts:://iithubbcc htt.://ww.....oeeaaaaaaa//aaaaee
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 2324, 1998.
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
Lehmann, N., Gottschling, N. M., Gawlikosski, J., Stewart, A. J., Depeweg, S., and Nalisnick, Lightning uq box: Uncertainty quantification for neural networks. Journal of Machine Learning Research, 26(54):1-7, 2025. /tttt:////mmml http://itrr...orppprrrrs///////s
2110 html
Lei, J. and Wasserman, L. Distribution- prediction bands for non- naramaraet regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-96, 2014.
Liu, C., Zhu, L., and Belkin, M. Loss landscapes and optimization in wver-paraeeterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116, 2022.
MacKay, D. J. Bayesian interpolation. Neural Computation, 4(3):415-447, 1992.
Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., and Wilson, A. G. A simple baseline for bayesian uncertainty in deep learning, 2019.
Madras, D., Atwood, J., and D' Amour, A. Detecting underspecification with local ensembles. arXiv preprint arxiv: 1910.09573, 2019.
Malitsky, Y. and Mishchenko, K. Adaptive gradient descent without descent. In International Conference on Machine Learning, pp. 6702-6712. PMLR, 2020.
Martens, and Grosse, Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408-2417. PMLR, 2015.
Martinsson, P.-G. and Tropp. J. A. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403-572, 2020.
Masegosa, A. Learning under model misspecification: Applications to variational and ensemble
methods. Advances in Neural Information Processing Systems, 13:5479-5491 2020.
Matthews, A. G. d. G., Hron, J., Turner, R. E.,, and Ghahramani, Z. Sample-then-optimizz posterior sampling for bayesian linear models. NeurIPS Workshop on Advances in Approximate Bayesian Inference, 2017.
Meurant, G. The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations. SIAM, 2006.
Miani, M., Beretta, L.,, and Hauberg, S. Sketched lanezos uncertainty score: low-llweeee low-memory summary of the fisher information. arXiv preprint 24X.: 2024a.
Miani, M., Roy, H., and Hauberg, S. Bayes without underfitting: Fully correlated deep learning posteriors via alternating projections. arXiv preprint arXiv:2410.1690) 2024b.
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P., and Gal, Y. Deterministic neural networks with inductive biases capture epistemic and aleatoric uncertainty, arXiv preprint arXiv:2 1582, 2021.
13

---

Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., and Gal, Y. Deep seterministic uncertainty: new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24394, 2023.
Nacini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning, In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Nassif, A. B., Shahin, I., Attili, L., Azzeh, M., and Shaalan, K. Speech recognition using deep neural networks. IEEE access, 7:19143- 119914-1916,, 2019.
Neal, R. M. Boyesian Learning for Neural Networks, Vol. 118 of Lecture Notes in Statistics. Springer- Verlag, 1996.
Nemani, V., Biggio, L., Huan, X.. Hu, Z., Fink, O., Tran, A., Wang. Y., Zhang, X., and Hu, C. Uncertainty quantification in machine learning for engineering design and health prognostics: tutorial. Mechanical Systems and Signal Processing, 205: 10796, 2023a.
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang. Y., Zhang. X., and Hu, C. Uncertainty quantification in machine learning for engineering design and health prognostics: tutorial. Mechanical Systems and Signal Processing, 205:1 10796, 2023b.
Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., and Tran, D. Measuring calibration in deep learning. CVPR workshops, volume 2, 2019.
Ortega, L. A., Santana, S. R., and Hermández-Lobato, D. Variational linearized laplace approximation for bayesian deep learning. arXiv preprint arXiv:2302. /2565, 2023.
Papadopoulos, H., Procdrou, K., Vovk, V., and Gammerman, A. Inductive confidence machines for regression. In 13th European Conference on Machine Learning, pp. 345-356. Springer, 2002.
Park, Y. and Blei, D. Density uncertainty layers for reliable uncertainty estimation. International Conference on Artificial Intelligence and Statistics,
2024.
Rasmussen, C. E. Evaluation of Gaussian processes and other methods for non-linear regression. PhD thesis, University of Toronto Toronto, Canada, 1997.
Rasmussen, C. E. and Williams, C. K. I. Gaussian Processes for Machine Learning. The MIT doi: 10.755 10.755 httss//mmmee////20000.....
Press, 11 2005. ISBN 9780262256834. 000.000000.0000
URL https
Ray, P. P. Chatgpt: A comprehensive review background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 2023.
Redmon, J. and Farhadi, A. Yolo9000: Better, faster, stronger. Conference on Computer Vision and Pattern Recognition, pp. 7263-7271, 2017.
Redmon, J., Divvala, S., Girshick, R., and Farhadi, You only look once: Unified, real-time object
detection. Conference on Computer Vision and Pattern Recognition, pp. 779-788, 2016.
Ritter, H., Botev, A., and Barber, D. A scalable Laplace approximation for neural networks. Interna- tional Conference on Learning Representations, 2018.
Roux, N., Schmidt, M., and Bach, F. stochastic gradient method with an exponential convergence rate for finite training sets. Advances in neural information processing systems, 25, 2012.
Shalev-Shwartz, S. and Zhang, T. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14(1), 2013.
Tagasovska, N. and Lopez-Paz, D. Single-model uncertainties for deep learning. Advances in neural information processing systems, 32, 2019.
Titsias, M. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics, pp. 567-574. PMLR, 2009.
Van der Vaart, A. W. Asymptotic Statistics, volume 3. Cambridge University Press, 2000.
14

---

Vaswani, A., Shazeer, N., Parmar, N.. Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. URL http////////prrcediingg neurips cc/ cc/pappr__.. f files/paperr/017 FiL.:3eeees77eeelaaaaaccccaaaaap
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in a Random World, volume 29.
Springer, 2005.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. pp. 681-688, 2011.
Wen, Y., Tran, D., and Ba, J. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715. 2020.
Xie, Z., Tang, Q-Y., Cai, Sun, M., and Li, P. On the power-law hessian spectrums in deep learning. arXiv preprint arXiv:2201. 13001, 2022.
Yang, G. and Hu, E. J. Feature learning in infinite-wine neural networks. International Conference on Machine Learning, 139, 2021.
15

---

A Proofs
Proof of Lemma 3.7 First, we note that by the assumption on f, a solution to (5) always exists.
Suppose to the contrary that (5) has two distinct solutions 0 and such that @ o 0. Since O e
[ange[[] (0, X)]1 and 0 E Range([J(O, (')]"") i.e., (0 0) T No X)). it follows that
J(O. x0 # J(0, ((0,X)6: 0 which in particular implies (Vf x,), 0) + (Vf(O, xi), 0) for all i
1...., n. Consider A = (0 + 0) By strict convexity on L, we have
0+0
((((0, x.). = Σ L f(0,x,) +
VF(0, x,),
0
.14
2
x(0,xx)+ +
x;;
0
f(0,x;) + (Vf(0, x,) 0 0
Σe
1/
2
2
8 (i)0,
(f(0xx)).1)
(f(0.x)).1)
which is contradiction.
Proof of Theorem 3.2
(Gradient Descent) Denoting
x) - [JT(0 _T)0,x.)] E RReexp
we can write
X) (1-J'00,)))0001 z.
where Ô represents the parameters around which the linear model f is defined in 2 The first iteration of gradient descent, initialized ((0) z. given by
or z x, V
0 z a
00
"
of
I - J' (0, x1) 2
Σ
(2, x, V((f(2,x))))
00
x1) 1 ve(f(2,x)),.)
I -
2+J' X'z
Σ
- J¹ (0,) z (1)
where y()) E Range
The next iteration is similarly given by
of
0(0) (
Σ
0 x V((F((()) xx))..) (x).))))))))).))))
d0
v( - Σ ((0,,i)) 'vetio)'vee(iouu (0((((00)))))))
I 1- (0, 2 (1)
I (. v1) (10) +2))
16

---

[_(0. Generalizing to the nth iteration,
where again v(2) E Range
@(m) J' 00
Hence, by the assumption L, as long adaptive learning rate is chosen appropriately according to Malitsky Mishchenko (2020). GD must converge to a solution of the form
0r (I J (0,
v
-11- (1- x) z vv
where v E Range ([J(0.x))')
In particular, for z = 0, by Lemma 3.1, we must have that v 01, where is the solution to 5 Therefore,
J (0, 0
(Stochastic Gradient Descent) Using similar argument above, it is easy to show that each iteration of the mini-batee SGD is of the form
+ Range (1110.x0))
(n)) = (I J JI
Hence, it suffices to show that SGD converges almost surely. Defining
xe(((0,x)):11)
((0)= - (f, and g(0,x) e
E R
we write
VL(0) = Σ J( xi) (e(T(0,x:).x)) [J x]] g)0
Let 0* be any solution to 33. The full row-rank assumption on x0 implies that we must have g(0", .2) 0, i.e., (e(
= 0, = -,..., By the j-stroon convexity assumption on y( with respect its first argument, is easy to see that L(0) satisfies the Polyak-L.ojsiewin inequality (Karimi et al. 2016) with constant 2μ1 where
- min
o min (J
1= I.....
and 0 miin (J(0,x) is the smallest non-zero singular value of (0, Indeed, let 0* be any solution 33. From p-strong convexity of {(.,)) with respect to its first argument, for any 0 have
(((0, - -( xx). <
(((((0xx,),ss)
2/
1
[1(0,xi]] v((((0,x)).x))
2 min (J(0.x,))
which implies
1
x30 8(0,
(( ((0)-)(0')+ <
2px |V
2nd
17

---

By the smoothness assumption on each ((f(,,;)),) ys) as well as the interpolating property of 0* Bassily et al.) (2018) Theorem 1) implies that the mini-batch SGD with small enough step size r) has an exponential convergence rate as
EL(O() - (1- p) C()
for some contact < <
This in particular implies that for any > 0,
EL(O(H) - (()(0)
E (1-p)* ((
P(((O()) > c) -
ep < oc.
Now, the BorelCantelli lemma gives ((O(H) + 0, almost surely.
B Related Works: Further Details and Discussions
NUQLS shares notable similarities with, and exhibits distinct differences from, several prior works, which are discussed in-depth below.
B.1 Linearized Laplace Approximation (LLA) Framework.
The popular LLA framework (Khan et al. 2019; Foong et al. 2019 Immer et al. [021]] Daxberger
ataly (02T) shares a close connection with NUQLS, as both methods fundamentally rely linearizing the network. However, a subtle yet significant distinction lies in their constructions: LLA begins by obtaining a proper distribution over the parameters and then draws parameter samples from it, while NUQLS bypasses this step and directly targets an approximation of the posterior distribution of the neural network.
As direct consequence of this, in overparameterized settings, where the Hessian (or its Generalized Gauss-Newton approximation) is not positive definite, the LLA framework necessitates imposing an appropriate prior over the parameters to avoid degeneracy. In sharp contrast, NUQLS directly generates samples from the predietive distribution without introducing any artificial prior, thereby avoiding potential biases that such priors might impose on the covariance structure of the outputs and eliminating the need for additional hyperparameters. Consequently, corresponds to a Bayesian generalized linear model (GLM), the weight-space vs. function-space duality in GLMs implies that its predictive distribution corresponds to noisy GP with an NTK kernel. On the other hand, NUQLS leads to a noise-free GP. In interpolating regimes, where the model perfectly fits the data, the noise-free setting of NUQLS appears be more suitable (Hodgkinson e all 2023).
the LLA framework
Another important consequence of this distinction arises in classification tasks. Due to the Laplace approximation, the LLA framework produces an independent GP for each output of the lincarized model. In contrast, NUQLS captures the covariance between outputs, offering more comprchensive representation of the predictive distribution.
Finally, for regression tasks, NUQLS offers additional flexibility by allowing the variance to be scaled post-hoc by a factor 7. This enables efficient hyperparameter tuning on a validation set without the need for retraining the model or optimizing marginal likelihood-a level of flexibility not available the LLA framework.
NUQLS can be seen as an extension of the "sample-then-optimizz"" framework for posterior sampling of large, finite-width NNs (Matthews et al., (2017). In this context, the work Antorrá al. (2022). henceforth referred to as Sampling-LLA, enables drawing samples from the posterior distribution of the LLA in a manner analogous Algorithm [ In this approach, series of regularized least-squares regression problems are constructed, and the collection of their solutions is shown to be distributed according the LLA posterior. An EM algorithm is then employed for hyperparameter tuning. In addition to the fundamental differences between NUQLS and LLA- LLA-inspired methods mentioned First, the objective functions in earlier, Sampling-LLA has notable distinctions from Algorithm|] [ Sampling-LLLA have non-trivial minimum values. As a result, the convergence of SGD for such problems necessitates either a diminishing learning rate (Bubeck et al. (015), which slows down convergence, or the adoption of variance reduction techniques (Roux al. 2012 Chalev-Shward &

---

Zhang, 2013 Johnson & Zhang 2013), which can introduce additional computational and memory overhead. By contrast, in overparameterized settings and under the assumptions of Theorem 3.2 the optimization problem in (3) allows interpolation. Consequently, SGD can employ a constant step size for convergence, improving optimization efficiency (Garrigos & Gower] 2023). Second, the inherent properties of the LLA framework, which require positive definite Hessian or its approximation, necessitate regularizing the least-squares term in the subproblem of Sampling-LLA. This results in strongly convex problem with unique solution. Consequently, to generate collection of solutions, SamplingLLLA constructs a random set of such subproblems, each involving fitting the linearized network to random outputs. These random outputs are sampled from zero- mean Gaussian, with covariance given by the Hessian of the loss function, evaluated on the data. In contrast, the subproblem of NUQLS, i.e., (3). involves directly fitting the training data, and the ensemble of solutions is constructed as a result of random initialization of the optimization algorithm. Hence, the uncertainty captured by NUQLS arises naturally from the variance of solutions in the overparameterized regime, without the need for additional regularization or artificially constructed subproblems.
While the Sampling-LLA method enhances the scalability of LLA, the competing method Variational LLA (VaLLA) (Ortega et al. 2023) offers comparable or superior UQ performance while significantly reducing computation time. VaLLA achieves this by computing the LLA predictive distribution using a variational sparse GP with an NTK. Another competing LLA extension is Accelerated LLA (ELLA), which uses a Nyström approximation of the functional LLA covariance matrix (Deng et al.) 2022), and seems to attain similar performance to VaLLA, again at reduced cost compared to Sampling-LLAA We compare the performance of NUQLS to Sampling-LLLA, VaL.LA and ELLA in Appendix H.3
Note that work by (Miani et al. 2024b) approximates an LLA by taking the covariance as the projection onto the null space of the GGN matrix, in order to compute a posterior that retains the same performance as the original network on training data. To compute samples, this work uses alternating projections.
B.2 SNGP
We briefly describe the Opectral-Normalliee Neural Gaussian Process (SNGP). SNGP lies at the intersection of GP with NN. Specifically, SNGP adds weight normalization step during training, and then replaces the output layer of NN with a GP that takes the feature extractor of the network as an input. While this method is also not post-hoc, and is not strictly a Bayesian method, we still compare SNGP with NUQLS Figure 6 and Table where we observe superior performance of NUQLS.
methods and Bayesian methods, and similarly to NUQLS combines
B.3 Ensemble Framework.
In Lee al. (2019), infinitely wide neural networks were shown follow a GP distribution; however. this GP did not correspond to a true predictive distribution. Building on this, He et al. (020) introduced random, untrainable function to an infinitely wide neural network, deriving GP predictive distribution using the infinite-width NTK. An ensemble of these modified NNs was then interpreted as samples from the GP's predictive posterior. In contrast, our method demonstrates that trained, finite-width, unmodified linearized networks are inherently samples from GP with an NTK kernel. While their method, Bayesian Deep Ensembles (BDE), shares some conceptual similarities with ours, we omit it from our main experiments for several reasons. Firstly, the posterior analysis of BDE is valid only in the limit of large model sizes. For smaller datasets and models, the infinite-widhh NTK differs greatly from the empirical NTK (see Fort et al. (2020)). We can see that in this regime, in both Figure and Table g that NUQLS outperforms BDE, Secondly, the method computationally more expensive than DE, due to the need to compute the untrainable function (8()) and tune the scaling hyperparameter for classification. This contradicts our goal to provide a UQ method that is computationally more efficient than the state-of-the art (SOTA) method DE while maintaining competitive performance. Finally, for larger model sizes, we provide comparison of NUQLS to BDE in Figure 6 and Table [1 where we observe that NUQLS outperforms BDE.
In Madras et a 2019), the authors propose a method called "local ensembles", which perturbs the parameters of trained network along directions of small loss curvature create an ensemble of
19

---

nearly loss-invariann networks. The uncertainty of the original network is then quantified as the standard deviation across predictions in the ensemble. Building on this approach, Miani al. (2024a).
in method called Sketched Lanczos Uncertainty (SLUU, use the GGN approximation of the Hessian of the loss to identify these directions and introduce a sketched Lanczos algorithm (Meurant (006) to efficiently compute them. We compare the performance of NUQLS to SLU in Appendix H.2
While similar, there are key differences between local ensembles and NUQLS. Local ensembles form a subspace of networks that attain similar loss values, allowing for directions with small but non-zero curvature, potentially encompassing more directions than those with exactly zero curvature. In contrast, when a solution to the linear optimization problem exists, NUQLS creates an ensemble of networks that all attain exactly the same loss. Additionally, NUQLS relies on first-order information to construct this ensemble, whereas local ensembles depend on second-order information.
While the zero-curvature directions of the GGN approximation correspond the Jacobian's null space, the local ensembles method also includes directions with small but non-zero curvature. This inclusion introduces a notable distinction between the ensembles generated by local ensembles and those formed by NUQLS. Finally, while local ensembles employ low-rank approximations to compute directions efficiently, such approximations may inadequately represent the Hessian and fail to accurately capture the true curvature structure, as highlighted in Xie el al. (2022).
We also give mention to methods that seek to make DE more efficient, such BatchEnsemble (BE) (Wen et al. 2020) and Snapshotnnsemble (Havasi et al. 2020). We note that these methods are yet to surpass DE as the SOTA in the literature. However, we compare NUQLS against BE in Figure and Table 1T, where we see that NUQLS shows superior performance, whilst remaining as scalable as BE. Further, BE requires modification the structure of the network, and hence is not post-hoc.
B.4 Neural Tangent Kernel Methods
The Procedural-Nooe Soatrral-Noi---Cooooe Correcting (PNC) method of (Huang et al. 2023) employs the limiting NTK for infinite-widhe networks to characterize the noise in the optimization process of training a neural network. It uses this provide frequentist confidence interval around test predictions of the neural network, and as it arises from the frequentist framework, PNC recovers statistical guaranetees. In comparison, NUQLS does not rely on the limiting regime of the NTK, but rather employs the feature extraction of the empirical NTK to quantify uncertainty using a Bayesian framework. In Appendix(H.. [..] PNC. Unsurprisingly, given their constructions,
compare the performance of NUQLS see that PNC outperforms NUQLS when the width of a network is extreme, while for finite-width networks that are fully trained, NUQLS outperforms PNC. This experiment illustrates that the methods are actually complementary, as they both excel in different settings.
C Uncertainty vs. Prediction
Here we discuss the use of common metrics to measure UQ ability in the BDL community. Generally, NLL,, ECE and AUCROC (or the entropic version OODAUC) are used to measure the ability of a method to correctly quantify the uncertainty in neural network. However, we argue that the use of these metrics for UQ is ill-advised, due to either misalignment between the metric and the goal of UQ or due to flawed measurements of uncertainty implicit in the metric. Specifically, we argue that variance over the softmax predictions is the uncertainty in the network, in the same way that the variance over probabilities is the uncertainty Dirichlet model. In contrast, NLL and AUCROC (OODAUC) metrics for prediction, while ECE is metric based on flawed surrogate for uncertainty, calibration.
Prediction While we will argue that uncertainty quantification must involve computed variance term, and that common UQ metrics are either actually prediction metries, or are based on a prediction of uncertainty, this discussion also begs the question: what about the purported benefits of BDL to prediction ability? As has been argued in Abe et al. (2022), it is often more economical to simply train a larger capacity model, than to use BDL to improve prediction quality. Hence, we restrict our main focus of NUQLS to its UQ ability. However, for completeness, we also evaluate the novel predictor of NUQLS; these results can be found in the Appendix G
20

---

Uncertainty Uncertainty can be divided into two categories: aleatoric uncertainty, or data uncer- tainty, and epistemic uncertainty, or model uncertainty. Aleatoric uncertainty arises from intrinsic randomness or noise in the data, and can often not be controlled. In contrast, epistemic uncertainty arises from having many potential models that may fit the data, and the uncertainty in not knowing which model is 'correct'. As this arises from lack of knowledge of the original data-mapping, uncertainty in this manner can often be reduced through increased training data, using a larger model, training for longer etc. There are numerous benefits to untangling the aleatoric and epistemic uncertainty (Mukhoti et al. 202T Huseljic et al.) 202T: Chan et al., 2024). We can use the framework of Dirichlet model, the conjugate prior of a Categorical distribution, understand how to evaluate these uncertainties in the context of K --class classification. Assume there is some subspace M of the parameter space + R', such that 0" E .M. If it is common across parameter values in .M that f(x; 0*) has high entropy for some x, i.e. f(x; (*) outputs a prediction close to the uniform distribution with high probability across 0' E EM, then we are sure that the outcome is unpredictable in x, i.e. that there is high aleatoric uncertainty. This equates to a Dirichlet model with parameters large magnitude. In contrast, assume that over all parameter values in M, the softmax outputs of f(x:0% for given have very large variance. This means that no model can agree on the 'correct' data-mapping, and hence there is very large epistemic uncertainty. This is analogous to when the parameters a of a Dirichlet model are small in magnitude. Our method, NUQLS, measures the variance over softmax outputs for ligh-performing parameters 0 E .M. Hence, we are measuring the epistemic uncertainty of the neural network.
network f(:; 0*) attains "good' test performance for any
(a ak) where is uniform and of
NLL Due to its probabilistic framework, NLL is often used as metric for UQ ability. For K-clas classification, up to a constant the NLL is just the cross-entropy loss
NL = K
YoRk log pi,k
where Vi.k corresponds to the k-th element of the one-hot encoding of the i-th label, and Mi, softmax [xx;;; 0))x. This loss minimised as softmax(f(x;; 0))s + Ie., where 63 is the correct label for input x,. While a lower NLL is generally considered to show better UQ ability, we can see that this metric is instead a smooth, probabilistic measurement of accuracy, where proximity of the predictor to the identity function is rewarded with lower NLL. This metric does not measure uncertainty quantification ability.
ECE A common surrogate for uncertainty in neural networks is the predicted softmax probabil- ities. While neural networks generally output over-confident probabilities, many methods, such as temperature scaling and many Bayesian methods, attempt to create well-calibrated predictors. A well-calibrated output is one where the predicted softmax probabilities accurately convey the forecasted probability of seeing each of the K-classes. For example, of all test points where a network has maximum predicted probability of 0.9, we would hope that the network is 90% accurate on these test points. Once a predictor is well-calibrated, the softmax probabilities are then seen to quantify uncertainty. However, there are issues with this:
1. Firstly, a point prediction from network cannot be interpreted as the uncertainty in the network: uncertainty quantities must accompany a network prediction, for example with a variance term. At best, network confidence can be seen to predict the uncertainty.
2. Secondly, if softmax probability does in fact accurately predict the uncertainty, is this uncertainty due to data uncertainty, or model uncertainty? If model accurately predicts that there is 90% chance that class c is correct, does that uncertainty arise due to noise in the data, or some misspecification in the labeling process? Or is it due to the fact that the input is OoD, and thus the model is unsure of itself? Hence, we still require a variance term to ascertain what the epistemic uncertainty is.
3. Finally, calibration is only well-defined for ID test points. To see this, take a network trained on MNIST, and then tested on FashionMNIST. Any network prediction is meaningless on this test set, as the model has been trained for integer labels. The prediction that will minimize the calibration error will be the uniform prediction i.e. the prediction is [0.1,..., points in the OoD test set. However, the accuracy on OoD test points should be Instead the predicted accuracy is 0,1, which is what arises from randomly picking one class for each OoD test
E R0 for all
21

---

point. Therefore, the predicted probabilites cannot be directly interpreted as forecasted probability of accuracy for OoD points.
In practice, calibration is tested using ECE. This metric bins predictions, and then computes the average confidence in each bin. This is compared with the average predicted accuracy, and l2 norm is taken over the differences, As was noted in Nixon et al. (2019). there are several issues with ECE. For examples, it reductive in a multi-class setting, where the bottom K probabilities do not contribute meaningfully to the error, and it suffers from the inherent sharpness of neural networks, where the majority of maximum probabilities tend to be very close to and thus this confidence range is highly over-represented. So we struggle to even tell when a method is well-calibrated.
This issue with ECE can be summarised as follows: confidence is flawed estimate of the total uncertainty: further, it is hard to tell when a method is well-calibrated. It is thus evident that it is much better to directly model the uncertainty in the model, by calculating the variance over the softmax predictions. Once equipped with this model uncertainty, entropy (or network confidence) can be better trusted inform of the aleatoric uncertainty.
The final metric commonly used for UQ is AUCROC (OODAUC). This metric uses the top softmax prediction (entropy) as a predictor, to attempt detect wether point is ID or OoD. While this is a very useful task, it is still a metric of prediction, as only the mean predictor is used to compute this value. As have argued, UQ requires variance term. Miani et al. (2024a) in fact use the variance over logits to compute the AUCROC score, and report this score for their method SLU in comparison to other leading UQ methods. In Appendix H.2 H.2 outperform SLU.
AUCROC
we show that in this metric,
VMSP To combat the issues of the previous metrics, we compare the performance of Bayesian methods using the variance of the maximum softmax predictor, or VMSP. We first provide an explicit definition of VMSP: for Bayesian method, generally have mean predictor μ R + R and covariance function Σ Rd + Rxxe for example the mean and covariance of the lincarized ensemble in the case of NUQLS, that output in the probit space. To compute VMSP for given test point x", we first find à = argmax, & /4xx)) ),, where μu(*)), ) denotes the k-th output of (xx*). ). That is, we find the class that the Bayesian method predicts, given x We then define VMSP = E(x* Exx")),,= o²(x*) )a that is, the variance of this prediction.
We also introduce a pictorial method for evaluating the performance of Bayesian methods, using VMSP as the correct uncertainty measure. For a given dataset, we want a UQ method to provide low uncertainty for correctly predicted test points, and high uncertainty for incorrectly predicted OOD test points. We then compare these distributions pictorially using a violin plot, and quantitatively using the median and skew values for the respective distributions. With the addition of poorly-penn porrl-pperorming bascline model, we are able to easily compare the ability of UQ models to quantify uncertainty.
D Guarantees Against Mode Collapse
Given that all linearized models are initialized locally around the trained parameters, one may consider whether it is possible that all linear networks will converge to the same parameter solution, or mode of the linear loss. Fortunately, see this, we note that the solution to (3) for each linearized model is given by a unique row-space component (given our assumptions), plus a projection of the initialization z; onto the null-space of the Jacobian. For cole-collapse, we would require the projection for two i.i.d initializations
are able to show that this will not occur, almost surely. To
21, 72 ~ (00,721 to be equal, or (I [I- - 22) = O, where 2 - Z2 * O with probability
one. Hence, Z1 Z2 Jx Zz), and thus 21 Z2 E RRanee i... 21, Zg E Range(J However, Range(Jx)) is a www-dimensional subspace of IR," and as
drawing i.i.d. from p- dimensional normal distribution (i.c. not degenerate), the probability of this event occurring is 0. Hence, we do not need to be concerned with all models converging to the same behaviour.
E Computational Cost
We now provide examination of the computational cost of NUQLS. We compare the computational complexity for epoch of training for the neural network fo(x), for batch æ ddd,, and an
22

---

epoch of training for a single linear network fo(x). We take approximations on the computational complexity of both forward-mode AD and backward-mode AD from Blondel & Roulet (2024 Chapter 8). Specifically, we take [fp] as the computational complexity for evaluating (o))). We note from Blondel & Roulet (2024, Chapter 8) that both JVP and VJP cost roughly 2 x a forward pass in computational complexity and memory. Further, both a JVP and a JVP return function evaluation. Now, a standard epoch of training for the neural network involves a forward-pass to compute the error, and then a backward-pass (VJP), hence the complexity is approx. 3[ The linearised network involves a JVP (which includes a function evaluation) to form the linear network, and a VJP to compute the gradient. Hence, the complexity for an epoch of training for the linearized network is approx. 4[ So we observe that each epoch for a linearized network is only 4/3 x as expensive as for neural network. In regards memory, see that the memory requirement for both the linearized network and the neural network are similar. However, we generally train all linearized networks in parallel. For computational complexity, this will incur some additional cost, thought it will not be linear in number of networks, that is dependent upon the specific software and hardware. However, memory cost will scale linearly by number of networks, i.e. 25 M(fp))) where ([)p)) is the memory cost for a forward-pass, and S is the number of linear networks. As an example, we employ batch size of 56 for ImageNet on ResNet50 with 10 ensemble members when using an 80GB H100 GPU, due to the large parameter count and large number of classes. Note that in the case where the training set small, is beneficial compute and save in memory the Jacobian of the NN evaluated on the entire training set. We can then train the linear networks very quickly. This contributes to the impressive run on-times seen in Tables and
F Hyper-parameter Tuning
NUQLS contains several apperpparametes: the number of linear networks to be trained, the number of epochs and learning rate of training, and the variance of initialisation, y. In this section, we discuss strategies to select optimal hyper-parameters
F.1 Regression
For regression, and with sufficiently small learning rate, NUQLS samples from the distribution given in Remark We can see that the variance of the predictions scales linearly with y². Hence, we use the following framework to tune y:
1. To obtain 9, Algorithm !
we initialise our parameters with very small gamma, e.g. 7 0.01. This enables (stochastic) gradient descent to converge quickly with small learning rate.
2. We compute As per Remark [.3] for each point x E Xeal- we compute the mean predic- tion as plxx)
(xa)) where Kval is the inputs from validation set.
and the variance as dz(z) SampleVariance (:: k.a)) (note the scaling by y for only the variance). We
SampleMean (f(0)xx))
then use these values to compute the ECE across the validation dataset D
3. As coverage of confidence interval scales linearly with the size of the given standard deviation, and our computed standard deviation scales lincarly with y, find that the ECE is convex in ? Hence, we employ the Ternary search method (see Algorithm 2 to find the value 9 that minimizes this validation ECE.
4. For a test point x", our mean prediction and variance is then (xx*) and (3
As can be seen in Table and Table 9 this framework means that for regression our method is incredibly fast and computes well-calibrated variance values.
F.2 Classification
For uncertainty quantification performance, as measured by VMSP Figure we find that as long as y is small, training SGD for a small amount of epochs generally gives small training loss, and hence provides good performance. This means that our method can computed quite quickly in larger data/model settings.
23

---

Algorithm 2 Ternary Search
ʃ: function to minimize, 1: left boundary of search space, r: right boundary of search space, 6: tolerance. iter: iterations.
i 0
while [ r| 2 8 and <iter do 11/3 = 1+ (r - -
11/3 r- (
if = 1/3
f((1/a) < f(m then
else
r1/3 r/33
end if end while return r1+1
Table 2: Image classification predictive performance, using LeNet5 on MNIST and FashionMNIST (FMNIST).. Experiment was run 5 times with different random MAP initialisations get standard deviation metrics.
Datasets
Method MAP NUQLS
NLL
ACCT
ECE
OOD-AUCT
AUC-ROCT
Time
0.034-t
0.990
0.008
0.888+tu.me
0.886+to.00s
257
th.000
0.989-to.mm 001
0.003 000
0.930 oze
0.928 tro 02n
106
MNIST
DE
0.991:to.om 1:o 1000000
0.011 oo1
0.932:t..om 009
0.928:to.cm uow
2845
MC-Dropout
0. t.0
0.989 -oo 0o0
0.017 01
0.873:tu.uu2
0.87 1-to.om
533
SWAG
0.029
0.991
0.004 002
0.902
0.900
489
LLA*
t.034=oo.0oz
0.990=to 001
0.008
0.888:5.000
0.88622..0m
VaLLA
0.034+tu.00a
0.990
0.008
0,889+tu.me
0.886=t..00
1583
MAP
0. 2298100.00
0.801 =0.5
0.006 001
0.8400-3.022
0.8041bo.021
158
NUQLS S
0.302 102-+ho.om DOG
0.891 tho oo0
0.005 002
0.904 to 007
0.870 oon
89
FMNIST
DE
0 0.288 Lo. 002
0.896 1096_Lo..0 001
0.013 000
0.876 Lo.003
0.836 Lo.00s
1587
MC-Dropout SWAG
0. 0.306=to.oor
0.892 DoXI
0.026 ona
0.856:tu.u21
0.813+t..o
291
0.283 fo
0.899 noa
0.018
0.817+0..02
0.783
264
LLA*
0.298.Lo.c
0.891 1Loo
0.006.1 1o
0.841-Lo.022
0.805
26
VaLLA
0. 0.298-tu.007
0.891 +oo
0.007
0.841+u.uzz
0.805=m..02
1583
G Evaluation of Predictive Mean
While the main focus of NUQLS is the variance term, to compute the epistemic uncertainty of NN, we would also like demonstrate the predictive ability of our novel predictive mean, for completeness.
G.1 Image Classification Predictive
We compare NUQLS against the MAP NN, DE, MC-Dropout, SWAG, LLA* and VaLLA, on the MNIST and FaShionMNIST datasets, using the LeNet5 network. We compare test cross-entropy (NLL), test accuracy (ACC), ECF, and the AUC-ROC measurement for maximum softmax probability (AUC-ROC) and entropy (OOD-AUC) as the detector. The two metrics evaluate methods ability to detect out-of distribution points. We display the results in Table 2
We see that NUQLS performs the best in ECE, AUC-ROC and OOD-AUC, and is competitive the other metrics, while having the second fastest wall-time.
H Further Experimental Results
H.1 Comparison PNC using Confidence Intervals
We use this section to compare the difference in performance of NUQLS vs PNC. To test the difference between the two methods, we tested NUQLS on the confidence intervals problem from Huang et al. (2023). In this experiment, an MLP with single hidden-layer is trained on datapoints
24

---

Table 3: Evalaution of coverage (CR) and width (IW) of computed confidence intervals from toy problem, for a neural network with extreme width and partial training. A computed coverage exceeding the expected coverage bolded. The mean prediction (MP) is also provided.
PNC
NUQLS
95%CI (CR/TW) 90%CC (CR/W)
95%CI (CR/WW
90%CI (CR/WW
128
0.98/0.0437
0.95/0.0323
0.1998
0.93/0.0357
0.92/0.0298
0.2047
4, 256
0..8/0.0411
0.95/0.0304
0.3991
0.92/0.0596
0.86 0.86/0.0320 ..6/0.05500
0.4084
Table 4: Evalaution of coverage (CR) and width (IW) of computed confidence intervals from toy problem, for neural network with finite width and full training. A computed coverage exceeding the expected coverage is bolded. The mean prediction (MP) is also provided.
PNC
NUQLS
95%CI (CR/TW)
90%CI (CR/WW)
MP
95%CI (CR/TW)
90%CI (CR/TW)
MP
2= 128
0.8/0.133
0.72/0.0105
0.2022
0.99/0.0135
0.96/0.0114
0.2012
d=4, 256
0.96/0.0437
0.90/0.0336
0.4045
0.97/0.0315
0.97 /0.0/3
0.4030
d ==88, 512
0.88/0.0740
0.88/0.0568
0.8078
1.00/0.0667
0.98/0.0555
0.8050
d= 16, 128
0.8/0.1443
0.8/0.1108
1.6265
1.00/0.1350
0.98/0.1133
1.6121
from ([0, 0.2] where the target is y Σ R== sin(2;) + and is a small noise term. The method then asked form confidence interval around the prediction for x = (0.1, 1,110.... o. 0.1). setup is repeated several times, i.c. the network and training data are randomly initialized, and the coverage and average width the confidence intervals are recorded, as well as the mean prediction. For more details, refer the reader to Huang al. (2023). When the width of the network is 32 x and the network is only partially trained with 80 epochs and a learning rate 0.01, NUQLS performs poorly compared to PNC, have reached or exceeded the expected coverage. Further, smaller width intervals are preferred.
This
be seen in Table 3 Note that bolded numbers indicate intervals that
We note that scaling the width of a network by 32 x is rare in practice and that such wide network difficult to train. we instead form an MI.P with width equal to the number of training points, and increase epochs to 100 and learning rate to 0.5, so that the network is properly trained, NUQLS far outperforms PNC, as can be seen in Table 4 We note that for these experiments we tuned the y myper-paramette for NUQLS in fact complementary methods. For infinite-width networks near initialization, PNC performs well while NUQLS struggles. Conversely, for finite-width networks trained to a minimum, NUQLS excels while PNC performs poorly. We believe the latter regime is more representative of practical scenarios, where NUQLS offers significantly better performance.
a small validation set. We conclude that NUQLS and PNC are
H.2 Comparison to SLU
Here we compare the performance of our method against the competing SLU method. See Appendix B for a discussion of the differences between NUQLS and SLU. Due to the extensive experimental
details given in (Miani et al. 2024a), performance NUQLS against the SLU results found in (Miani et al.) 2024a). In Table [5) we compare NUQLS against SLU on OoD detection using the AUC-ROC metric, on a smaller single- layer MLP and larger LeNet model. The MLP is trained on the MNIST dataset, while the LeNet model trained on the FashionMNIST dataset. For MNIST, the OoD datasets are Fashion KMNIST, and a Rotated MNIST dataset, where for each experiment run, we compute the average AUC-ROC score over range of rotation angles (15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180). For FashionMNIST, the OoD datasets are MNIST, and the average Rotated FashionMNIST dataset. The AUC-ROC metric was calculated using the variance of the logits, summed over the classes for each test point, as score. We observe that NUQLS has a better AUC-ROC value over all OoD datasets, often by significant margin.
certain experiments from this work and report the
H.3 Comparison to Sampling-LLA, VaLLA and ELLA
We now compare NUQLS against Sampling-LLLA, VaLLA and ELLA. Due to issues with convergence, performance, memory usage and package compatibility when either rumning source
25

---

Table Comparing performance of NUQLS against SLU method. Metric given is the AUC-ROC, computed using the variance of the logits, summed over the classes for each test point, as score. AUC-ROC measures ability of method to differentiate between ID and OoD points. We see that NUQLS out-performs SLU in these experiments.
Model
MLP p = 15k: MNIST vs KMNIST
LeNet 40k FashionMNIST
ID Data OoD Data MNSST 0.26#to.02 0.67#0.07
Rotation (avg)
MNIST 0.94
Rotation (avg)
SLU
0.42:to.04 0.79:t..2
0.59:
0.74
NUQLS
0. 0.74
0.95
0.91#t.01
Table 6: Comparing performance of NUQLS against Sampling-LLA, VaLLA and ELLA on MNIST, trained used 2-layer MLP with 200 hidden units in each layer, and tanh activation. Original results taken from OOtega et al. 2023).
Model ELLA
ACC
NLL
ECE
BRIER
OOD-AUC
97.6
0.076
0.008
0.036
0.905
Sampled LLA
97.6
0.087
0.026
0.040
0.954
VaLLA 100
97.7
0.076
0.010
0.036
0.916
VaLLA 200
97.7
0.075
0.010
0.035
0.921
NUQLS
98.0=to.1
0.065=t..00
0.005=to.ou1
0.031#t.0001
0.953:t0.000
code or implementing methods from instructions given in (Antorán et al. 2022 and (Ortega et al.. 2023), we instead compare NUQLS against the results for Sampling-LLA, VaLLA and ELLA taken verbatim from (Ortega et al., 2023, Figure 3). We train 2-layer MLP, with 200 hidden-units in each layer, the MNIST dataset, according to the experimental details given in (Ortega et al. 2023). We then compare the accuracy, NLL, ECE, Brier score and the OOD-AUC metric of NUQLS against those reported for Sampling LLA, VaLLA and ELLA. The results are shown in Table 6 We display the mean and standard deviation for NUQLS; as is quoted in (Ortega et al. 2023). the standard deviation for the other methods was below 10-4 in magnitude, and was thus omitted. We see that NUQLS outperforms Sampling-LLA, VaLLA and ELLA in accuracy, NLL., ECE, and Brier score, and is within one-sixth of standard deviation of the leading OOD-AUC value, attained by Sampling-LLAA While we cannot directly comment on differences in computation time between our method and these LLA extensions,
were able run the source code for VaLLA for the experiments in Table 2 where NUQLS was an order-of-maanitude faster than VaLLA in wall-time. In (Ortega et al. [023] Figure (right)),
also observe that ValLA is ofeer-f--maniitude faster than both Sampling-LLA and ELLA. We also compare with (Ortega al, [223] Table 1). where a ResNet20 and ResNet34 model is trained on CIFAR-IO. The re- sults are displayed in Table We see that NUQLS is competitive with all other methods in this setting.
Table 7: Comparing performance of NUQLS against Sampling-LLA, VaLLA and ELLA on CIFAR-
10, with ResNet20 and ResNet32. Original results taken from (Ortega et al. 2023). Purple figures correspond to the top result, while blue figures are the second-best result.
ResNer20
ResNet32
Model
ACC
NLL
ECE
ACC
NLL
ECE
ELLA
92.5
0.233
0.009
93.5
0.215
0.008
Sampled LLA
92.5
0.231
0.006
93.5
0.217
0.008
VaLLA
92.6
0.228
0.007
93.5
0.211
0.007
NUQLS
92.5
0.228
0.006
93.4
0.215
0.007
26

---

Table Comparing performance of NUQLS and BDE on UCI regression tasks. We see that NUQLS outperforms BDE on all tasks.
Dataset Energy
Method
RMSE
NLL
ECE
BDE
0.406=to.039
00.225t0.212
0.008=t0.005
NUQLS
0.047=t..00
2.4000
0.002
Concrete
BDE
0.714:to.054
1.563=t.449
0.063:to.012
NUQLS
0.000=t..047
0.316-to.00
0.003=t.000
Kin8nm
BDE
0.851
1.383
0.042:::.012
NUQLS
0.252:to.00s
-0.796=0.025
0.000=0.000
H.4
Comparison to BDE
Figure displays the performance of BDE on the toy regression problem from Figure 1 compare BDE against NUQLS several UCI regression tasks in Table g
We also
BDE
NUQLS
200
6-
0-
-200
2.5
0.0
2.5
5.0
2.5
0.0
2.5
5.0
5.0
2.5
2.5
s.0
Figure 4: Comparison of BDE, DE and NUQLS on the toy regression problem from Figure We can see that the uncertainty of the BDE method is quite small.
H.5 UCI Regression
We present the results for select UCI regression datasets in Table I in the main body; we present results for several more datasets in Table 9
H.6 eNUQLS
In Figure 5 we demonstrate the performance of ensembled version of NUQLS, eNQQLS, This method is similar to a Mixture of Laplace Approximations (Eschenhagen et al. 2021 We observe excellent separation of the variances between correct and incorrect/OoD test groups for eNUQLS, especially for CIFAR-10 10 on ResNet9, Note that there is significant cost to ensembling our method, and we provide this figure simply illustrate performance capacity.
H.7 Additional VMSP Results
We use this section to expand the empirical evaluation of NUQLS. In the top left Figure 6 we observe that ResNet50 model trained SVHN, with CIFAR-10 the OoD test data, NUQLS outperforms all other methods. In the top right of Figure 0 we evaluated NUQLS on the ImageNet dataset in order to display the scalability of our method to larger datasets. We employed the pre- trained weights for a ResNet50, as found on turc and used the ImageNet-o dataset as our OOD test set (Hendrycks et al. 2021). Due resource budget constraints, NUQLS against baseline method, though future research could include a comparison of other methods against NUQLS, using VMSP. ImageNet. As can be seen, we see excellent separation between correct and incorrect predictions, while only adequate separation between correct and OOD points. We note from Bitterwolf et al. (2023) that approximately 21% of the images in ImageNet-o are actually in i--ditribution;; hence, we see that NUQLS correctly quantifies the variance of these test sets. We also provide comparison NUQLS with additional competing methods. the bottom left of Figure 6 we compare NUQLS against the BE method. Note that BE requires modification to the structure of a neural network; hence, we compare BE on modified WideResNNet34-1 of which an implementation of the correct modifications existed (Franchi et al. 2023), trained on FashionMNIST.
were only able compare
27

---

Table 9: Comparing performance of NUQLS, DE, LLA and SWAG UCI regression tasks. NUQLS performs as well as or better than all other methods, while showing a speed up over other methods; this speed up increases with the size of the datasets. Note that for the Song dataset, the LLA method uses diagonal covariance structure due to memory constraints: this is denoted as LLA-D.
Dataset Concrete
Method NUQLS
RMSE
NLL
ECE
Time(s)
0.000
0.316=to.60
0.003
7.339 (0.185)
DE
0.379:00010 -0.574-0.008
0.002
29.047
LLA
t..00
-0.294-t04799
0.003
7.451 (0.297)
SWAG
0.004
-0.562-10.22
0.009
43.416 (36. 33.2622
Naval
NUQLS
0.049=t.012
2..446-0.134
0.002
11.360 (0. (0.295)
DE
0.006
-1.761+t0.200
0.000
96.570
LLA
0.070+0.022
25.292
0.192
140.724 (129.659)
SWAG
1.130
0.303++1.091
0.084
103.727 (92.662)
CCPP
NUQLS
0.244
-0.885+t0.020
0.000
6. 6.698 (0.174)
DE
0.227 -1.009=o.0a1
0.002 008
79.791
LLA
0.243 29.420=:4.665
0.163
38.572 (32.048)
SWAG
0.252+0.001
-0.849+t..0as
0.001 - 0002
73.357 (66.833)
Wine
NUQLS
0.789-0..m
0.284
0.001 +
1.164 (0.115)
DE
0. 0..89
0.320=to.10p
0.001
13 241
LLA
0.792
1.012:to.182
0.009
1.389 (0.340)
SWAG
0.798
0.367
0.005
12.856 (11.807)
Yacht
NUQLS
0.042
1.561122.19
0.012
3.390 (0.164)
DE
0.647
-2.032:t0.340
0.016
40.132
LLA
0.043
-.7333-0.466
0.01 : 008
3.403 (0.177)
SWAG
0.04440.014
-2.565-to.116
0.067 0.002
19.408 (15.822)
Song
NUQLS
0.646
0.001 0.000
295.058 (91.673)
DE
0.846
0.180
0.005
2562.789
LLA-D
0.851 0.029
0.456-
0.000
413.814
(210.429)
SWAG
0.845+00.002
0.680+o..ooz
0.003
3477.825
(3274.440)
LeNet5 FMNIST
ResNet9 CIFAR-10
0.30
1D Correct
0CD
% 0.15
0.00
NUQLS
DE
eNUQLS
SWAG
MC
LLA
BASE
NUQLS
DE
eNUQLS SWAG
MC
LLA
BASE
Figure 5: Violin plot of VMSP, with an ensembled version of NUQLS, eNUQLS, included.
We again observe that NUQLS shows greater separation than BE for VMSP across correct predictions versus incorrect or OOD predictions. Finally, in the bottom right of Figure 6 we compare NUQLS against further competing methods, specifically BDE, SGLD (Welling & Teh 2011) (which include to demonstrate the inferior performance of MCMC methods in this regime) and SNGP on ResNet9 trained on FMNIST (note SNGP requires modifications to the training procedure). We see that BDE performs similarly to DE, and SLGD performs similarly to SWAG, which is unsurprising considering their respective constructions. We also observe that NUQLS still performs the strongest across all methods using the VMSP violin plots. The respective median and skew values for Figure 6 are displayed in Table 10 and Table these values also evidence the strong performance of NUQLS.
H.8 Image Classification Skew and Median
The median and sample skew for the VMSP Figure and Figure found Table TO and Table II
28

---

Table 10: Sample median and sample skew of variance from Figure and Figure 6 IDC= IDD correct, IDIC = ID incorrect, FMNIST = MNsHionMNIST. The median and skew for a method compared against baseline method. We expect positive skew for the ID correct (IDC) variances, and negative skew for the ID incorrect (IDIC) and OoD variances. If the median or skew for method is worse than the corresponding baseline, then is written in gray.
Median
NUQLS
DE
SWAG
MC
LLA
ResNet9
DDC
4.76 10-15
1.31 x 10'5
4.64 x 10
5.05 x 10-7
9.02 10-1 I0
0.020
FMNIST
DICC
0.182
0.040
0.048
0.004
0.004
0.020
OoD
0.217
0.109
0.075
0.004
0.009
0.020
ResNet50
IDC
0.00
8.29 x 10-5
0.018
1.49 x 10-V
1.70 x 10
0.019
CIFAR-10
IDC
0,178
0.120
0.123
1.32 x 10
0.008
0.020
OoD
0.178
0.106
0.134
6.74
10-5
0.004
0.020
ResNet50
IDCJ
1.00 10
8.73 10
0.0449
1.12
3.25 10
0.020
CIFAR-100
IDC
0.0624
0.101
5.93
10-3
0.0225
OoD
0.214
0.0665
0.0956
5.46
10-3
0.0199
0.020
ResNet50
DCC
8.61 X 10
3.59 x 10
1.71 x 10
2.18 x 10
1.61 x 10
0.020
SVHN
IDI
0.217
0.0569
0.0302
3.96
10
0.0241
0.020
OoD
0.233
0.127
0.0789
7.51 x 10
0.0252
0.020
ResNet50
IDC
0.00
0.020
ImageNet
IDIC
0,232
0.020
OoD
0.100
0.020
Sample Skew
ResNet9
IDC
3.51
3.72
6.46
FMNIST
IDC
0.615
0.928
0.378
OoD
-1.7
0.321
0.076
0.926
ResNet50
IDC
2.68
3.14
CIFAR-10
IDI
-0.15
0.05
0.816
OoD
-0.01
0.77
0.96
ResNet50
IDC
1.48
0.508
2.87
1.07
CIFAR-100
IDIC
-115
0.241
-0.47
2.05
0.228
1.09
OoD
-1.7
0.269
-0.233
1.85
0.217
1.13
ResNet50
IDCT
2.2
2.42
3.52
6.91
4.15
0.989
SVHN
IDC
1.31
0.52
0.878
1.64
0.601
1.12
OoD
-1788
0..4
-0.0119
1.18
0.296
1.06
ResNet50
IDC
1.52
1.21
ImageNet
IDIC
0.88
1.10
OoD
0.179
0.985
Table 11: Sample median and sample skew of variance from Figure 6 IDC = ID correct, IDIC ID incorrect, FMNIST FasHionMNIST. The median and skew for method compared against baseline method. We expect positive skew for the ID correct (IDC) variances, and a negative skew for the ID incorrect (IDIC) and OoD variances. If the median or skew for a method is worse than the corresponding baseline, then it is written in gray.
Median
NUQLS
BE
BDE
SGLD
SNGP
BASE
WRN-34-1
IDC
0.00
8.31 10
0.020
FMNIST
IDIC 1
0.178
0.045
0.020
OoD
0.173
0.107
0.020
ResNet9 FMNIST
IDC L
4.76 x 10 10 15
3.35 10 10-20
6.79 x 10
9.19 10
0.020
IDIC 1
0.182
0.00915
0.023
0.000322
0.020
OoD
0.217
0.0504
0.0352
0.000342
0.020
Sample Skew
WRN-34-1
IDC
3.1
2.8
1.01
FMNIST
IDIC
-0.342
0.944
1.11
OoD
00.8755
-0.272
1.11
ResNet9
IDC
2.4
6.46
5.49
29.8
0.93
FMNIST
IDIC
-0.615
1.43
1.23
24.4
1.15
OoD
-1.7
0.515
1.1
1.21
1.05
29

---

ResNet50
ImageNet
0.30
% 0.15
0.00
NUQLS
SWAG
MC FMNIITT
LLA
BASE
NUQLS
BASE
ResNet9
0.30
Correct Incorrect
00D
0.15
0.00
NUQLS
BE
BASE
NUQLS
BDE
SGLD SNGP SWAG MC
LLA
BASE
Figure 6: Violin plot of VMSP, for (top left) ResNe150 SVHN, (top right) ResNet50 ImageNet,
(bottom left) WRN-34-1 FMNIST, and (bottom right) ResNe19 FMNIST.
I Experiment Details
All experiments were run either on an Intel i7-12700 CPU (toy regression), or on an H100 80GB GPU (UCI regression and image classification). Where multiple experiments were run, mean and standard deviation were presented.
L1 NTK Convergence
For this experiment, 2 randomly sampled sets of 100 Gaussian inputs of dimension 5, and 100 scalar Gaussian targets, were used training points and test points respectively. The weights of the MLP were initialised to N ((0 1); no bias terms were used. To train, GD was employed with a Nesterov momentum parameter 0.9, a learning rate of 0.1, and 5000 epochs. For NUQLS, learning rate of 0.1 was used, and 7 was set to 1. Error bars on Figure 2 are the 95% sample confidence interval.
1.2 Toy Regression
We use a 1-layer MLP, with a width of 50 and SiLU activation. For the maximum a posteriori (MAP) network, train for 10000 epochs, with learning of 0.001, using the Adam optimizer and the PyTorrch polynomial learning rate scheduler, with parameters total_iters epochs, power 0.5. For DE, each network in the ensemble outputs heteroskedastic variance, and is trained using Gaussian NLL,, with 2000 epochs and a learning rate of 0.05. We combine the predictions of the ensembles as per (Lakshminarayane anaa al. 2017). Both DE and NUQLS use 10 realizations. The ? prperpparameter in NUQLS is set to 5, and each linear realization is trained for 1000 epochs with learning rate of 0.001, using SGD with momentum parameter of 0.9. In SWAG, the MAP network is trained for further 10000 epochs, using the same learning rate, and the covariance is formed with rank- approximation. prior precision of 0.1 and is used for LLA and LA respectively. as well as the full covariance matrix. The variational inference method used is Bayes By Backprop (Blundell et all 2015). as deployed in the Bayesian Torch package (Krishnan al. 2022). The prior parameters are (μ1 = 0, 1), and the posterior is initialized at (μ1 = 0, = -3). For SWAG, LLA, LA and VI, 1000 MC sample were taken at test time. These design choices gave the best performance for this problem.
30

---

Table 12: Training procedure for UCI regression results in Table
and Table
NN
Energy
Conerete
Kin8om 10-2
Naval 10-2
CCPP
Wine 10-2
Yacht
Protein
Song
Learning Rate
10-2
10-2
10-2
10-2
10-2
10
Epochs
1500
1000
500
150
100
100
1000
50
Weight Decay
10-2
10-5
10-5
10
10-
10-4
10-5
10-
0
Optimizer
Adam
Adam
SGD
SGD
Adam
SGD
Adam
SGD
SGD
Scheduler
PolyLR
PolyLR [150]
None
None
PolyLR
None
PolyLR [100]
None
None
MLP Size
150]
[100, 100]
[150, 150]
[100, 100)
[100]
[150, 200, 150]
[1000 1000
.50,500
NUQLS
Learning Rate
10
10
10
Epochs
150
100
50
15
10
10
100
25
10
DE
Learning Rate
10"
10
10
10
10
10"
10
10
10
Epochs
1500
300
100
100
100
100
1000
250
50
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Scheduler
None
None
Cosine
Nome
None
None
Cosine
None
None
Experiment
No. experiments
10
10
10
10
10
10
10
3
1.3 UCI Regression
We now provide the experimental details for the UCI regression experiments (as seen Table I and number of experiments to get mean and standard deviation
Table 9). For each dataset, for performance metrics. each experiment, we took a random /15% 15% split of the dataset for training. testing, and validation. The training hyper-parameters for the MAP, DE and NUQLS networks, size of the MIP used, and the number of experiments conducted for each dataset can be found in Table 12 Reported time for NUQLS, LLA and SWAG includes the training time for the original NN, with the time to run the method given in brackets.
NN: For the PolyLR learning rate scheduler, PyTorch polynomial learning rate scheduler was used, with parameters total_ tters=10xepochs power= 0.5. The MLP used tanh activation, so as to have smooth gradients. MLP weights were initialized as Xavier normal, and bias as standard normal. The dataset was normalized, so that the inputs and the outputs each had zero mean and unit standard deviation.
NUQLS:
The linear networks were trained using (S)GD with Nesterov momentum parameter 0.9. For all datasets except for Song, the full training Jacobian could be stored in memory; this made training extremely fast. For the Song dataset, we trained all linear networks parallel, by explicitly computing the gradient using JVPs and VJPs. The number of linear networks used was 10 across all datasets, and the y myper-paramete was kept at 0.01.
DE: Each member of the ensemble output separate heteroskedastic variance, and was trained minimise the Guassian negative log likelihood. The ensemble weights were also initialized as Xavier normal, and bias as standard normal. The number of ensemble members was kept at 10.
LLA: LLA requires two parameters for regression: dataset noise parameter, and prior variance on the parameters (Foong et al., 2019). To find the noise parameter, a grid search over 10 values log-scale between Te and Te2 was used find the noise that minimized the Gaussian likelihood of the validation set, with the LLA mean predictor as the Gaussian mean. The same grid search was used find the prior variance. in order minimize the expected calibration error (ECE) of LLA on the validation set. For the Protein dataset, a Kronecker-Factors covariance structure was used (Immer al. 202T), I) and for the Song dataset diagonal covariance structure was used. For all other datasets, LLA used the full covariance structure. The predictive distribution was computed with 1000 MC samples.
Curvature (KFAC)
SWAG: SGD was used, and the learning rate and number of epochs was kept the same as the NN. We used grid-search for weight decay, over the values [0, 0.005, 0.00005), minimize the ECE on the validation set. The rank of the covariance matrix was 10 for all datasets, and the predictive distribution used 1000 MC samples.
31

---

Table 13: Training procedure for image classification results in Table
Figure and Figure 6
MAP
LeNet5 MINST
LenetS FMNIST
ResNett FMNIST
ResNet50 CIFARIO
ResNet50 CIFARI00
ResNet50 SVIIN
ResNet50 ImageNet
Learning Kate
5.x 107
x
10
10-4
10"
10
Epoocht
35
35
10
200
200
50
Weight Decay
10"4
10'
10-1
10-
6 10
5 10
Busch Size
182
152
100
128
125
128
Opeimizer
Adam
Adom Cosine
Adam
SGD
SGD
SGD
Scheduler
Cosine
Cosine
Cocine
Crsine
Coonne
Accuncc
9936
90SS
42.556
92.0%
7576
67%
7036
NUQLS
Lesernnn Rate
10
10
10
10
10
10
10
Epoccs
10
50
10
Bonch Size
152
50
138
126
756
56
0.7
0.1
0.01
0.05
0.05
0.7
1.4 Image Classification
For MNIST and FaSHionMNISTT took a 5 training/validatto split of the training data. For CIFAR-10 10, simply used the test data as validation set. For CIFAR-IO, 10, random horizontal crop and flip on the training images was used as regularization.
We display the training procedure in Table 13 for both Figure 3 and Table 2
NN: We chose the training procedures to provide the best MAP performance. All networks have weights initialized as Xavier uniform. For SGD, momentum parameter of 0.9 was used. For the Cosine Annealing learning rate scheduler, the maximum epochs was set to the training epochs.
NUQLS: The number of samples was kept to 10 for all datasets.
DE: Similary, 10 ensemble members were used for datasets.
MC-Dropout: Dropout was applied to the network before the last fully connected layer. The dropout probability was set to 0,1 (a larger probability 0.25 was also used, but it did not change the result). At test time, 100 MC-samples
taken.
SWAG: The network was trained for further 1x training epochs. For CIFAR-100 and SVHN, learning rate kept the same the original NN; for other datasets, larger learning rate of 1e2xNN learning rate was used. The covariance rank was set at 10. At test time, 100 MC-samples were taken.
LLA*; We used a last-layer KFAC approximation to the covariance. The prior precision was found through grid search over 20 values log scale from 1e 2 to 1e2, using the probit approximation to the predictive, and a validation set. This configuration for LLA is what is recommended in Daxberger al. (202T) We used 1000 samples, remedy the large amount of approximations used.
VaLLA: We used the implimentation found in (Ortega et 2023 We kept nearly all hyper-
parameters the same as in the MNIST and FashionMNIST experiments in (Ortega et al. (023); however, we reduced the number of iterations 5000, due time constraints. We were unable to run this implimentation for ResNet9 or ResNet50, due to an out-of-memory error.
1.5 Confidence Intervals
For the confidence intervals experiments in Table and Table we used 100 repeats for d 4 and 50 repeats for d 2 8. We used multiplier of 1.5 on all NUQLS variance values, as we note that the search strategy for for regression only employs ECE; this does not reward confidence intervals that exceed the expected confidence, and hence computed confidence intervals may be conservative in width.
I.6
Comparison Experiments
1661
BDE
For Figure 4 and Table & BDE was implimented following details given in (He et al. 2020). To make comparison with NUQLS fair, each ensemble member of BDE did not output a separate hetero- skedastic variance term. Instead, the variance over predictions was taken the variance. Further, the large computational cost of BDE relative to NUQLS meant that it unrealistic for comparison to allow BDE to tune hyper-pann parameers through grid-search, this would take the run time of BDE
32

---

(which is slightly greater than that of DE) severely out-of-propootion to NUQLS. For Figure 4 the noise-parameter was given as the noise the data, For Table standard Gaussian noise was taken priori. The ensemble members were trained with the same training scheme as the original NN in both Figure(Jand Table(8) X For both experiments, ensemble members were used. For classification, we again followed the implementation details given in He et al. (2020) to the best of our ability. We employed the same hyperparameters as DE/MAP, and used 10 ensemble members.
1.6.2 SLU
For the results in Table 5 we copied the training details for (Miani et al. 2024a Table 3.); we point the reader to (Miani et al. 2024a Appendix D.1) for the exact details, For NUQLS, we used the following hyper-parameters for both MNIST and FashionMNIST: Epochs 10, samples 10, learning rate 0.01, batch size 152, y 1.
1.63
Sampling-LLA, VaLLA and ELLA
For the results in Table 6 used the training details for (Antorán al. 2022 Figure 3. (left)); again, we point the reader to the details given in (Antorán et al.,2202 Appendix F. F.1). For NUQLS, used the following hyper-parametter
for MNIST: Epochs 10, samples 10, learning rate 0.01, batch size 152. y 0.25. For ResNet20 the following hyper- were used: Epochs 1, samples 100, learning rate 0.0001, batch size 152, y 0.01. For ResNet32 the following hyper-parameters were used: Epochs 1, samples 150, learning rate 0.0001, batch size 152, 7 0.01.
L...4 BE
We employed the WideResNet architecture from Franchi et al. (203)). We used depth of 10 and widening factor of 1. The MAP network was trained with: Epochs 20, learning rate 0.005, weight decay 0.0001, batch size 152, the Adam optimizer and the Cosine learning rate scheduler. BE employed the same hyperparameters, yet with 1.5x 1.5x Epochs, to aid with training. BE used 5 ensemble members, in order to match the computational budget of NUQLS. The NUQLS hyperparameters were: Epochs 10, samples 10, learning rate 0.01, batch size 152, 7 1.
1.6.5 SNGP
We employed the Lightning UQ Box (Lehmann al. 2025) implementation of SNGP for experiments. The same hyperparameters as the MAP network were used. Note that we also tested SNGP ResNet50 SVHN, however SNGP was unable to train successfully, even after trying learning rates and 2 optimizers. Hence, we did not include the results for ResNet50 SVHN.
I.6.6 SGLD
We have implemented the SGLD method from Welling & Teh (201T) using the Lightning UQ Box package (Lehmann et al.) 2025) as the basis of the code. We then amended this code to include the learning rate scheduler from Welling Teh 201T). We followed[Madoo et al. (201)). and initialized the SGLD trajectory from the weights of the trained network. We also copied the learning rate from Maddox et al. (2019), and used the same weight-decay as the original network for the SGLDD prior. However, in contrast to Maddox et al. (201)) we took the noise-factor scaling to be 10-3 instead of x 10-4, as we found that this gave better performance. We sampled 100 epochs from the posterior, using a batch size of 100.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that our method approximates the posterior of an NTK- GP,, and provide theoretical and empirical proof in Section and Section 4. respectively. We also claim that we outperform competing methods in performance and efficiency, which can be seen in Section and Appendix H
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Discussion of our limitations can be found in the "Limitations" subsection of the
Conclusion, Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specificatio, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
We contain two provable results, Lemma 3.1 and Theorem 3.2 The proofs can be found in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross roossreffeeeced..
All assumptions should be clearly stated or referenced in the statement any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All code for the experiments has been included with the submission. All experimental details have been described in Appendix !
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
1. If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
2. If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
3. If the contribution is new model (e.g.. large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
4. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Link to code, as well downloadable package of our method, can be found the top of Section
35

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.cc/pbbll))
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to repro- duce the results. See the NeurIPS code and data submission guidelines (https://nips... cc/ public/ gui wwweeeeeeeeeeeieeeeeeeeeee wwwoooooewiiiiiooooooooooooee ssionclicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details are found in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: UCI Regression experiments (Tables 1 and and the empirical convergence figure (Figure 2 have еrror bars included, and are defined Figure 2 and in Appendix I.3 Violin plots do not have error bars, as they are generally not reported with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (c.g. negative error rates).
36

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification:
Experimental resources are listed at the top of Appendix !
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///eerrisss.cc/bbll//tttie [Yes]
Answer:
Justification: After reading through the NeurIPS Code of Ethics, believe that have con- formed to this code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the benefits of improved UQ methods in our Introduction and Abstract. We are not aware of any direct negative impacts of our work (though secondary negative impacts may exist).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular upplications. let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, is legitimate to point out that improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
37

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We did not use any high risk models or datasets, and instead only use highly common models and datasets that are commonly used in the UQ literature.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Any prior code for related methods that has been used has been referenced in the linked GitHub repository for our code. Any related methods for comparison has been properly cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpppsapppaeeeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our method has been released as package, with examples given for how to correctly our method.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
38

---

The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable), You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/reviewv based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The development of our method did not involve the of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve I.LMs any important, original, or non-standard components.
Please refer to our LLM policy should or should not be described.
http://ppp...cotoeeeeeeeee////// for what
39


Reviews:
[{"review_id": "Ws594ywzsW", "reviewer": "Reviewer_j2t5", "summary": "The paper proposes NUQLS (Neural Uncertainty Quantification by Linearised Sampling), a post-hoc method that draws an ensemble of linearised networks around trained parameters and then runs (stochastic) gradient descent to (approximately) solve a convex surrogate problem. Under stated conditions, the resulting predictors are shown to be samples from the predictive posterior of a Gaussian process with the empirical neural-tangent-kernel (NTK). Experiments on toy regression, nine UCI regression sets, and image-classification benchmarks (FashionMNIST, CIFAR10/100) show that NUQLS matches or outperforms Deep Ensembles, Laplace/LLA variants, SWAG, and MC-Dropout while being up to 10 faster in wall-clock time on the larger tasks than Deep Ensembles. The paper also introduces a variance-of-mean-softmax-probabilities (VMSP) metric for classification uncertainty.", "questions": "1. Can you demonstrate scalability to larger, more modern architectures? And additional datasets with additional epistemic uncertainty metrics?\n2. How does the method perform when the linearization approximation quality degrades?\n3. Can you compare to the most important related work in the main paper?", "limitations": "The authors adequately discuss computational limitations and the restricted theoretical scope. However, they could better address the theory-practice disconnect and provide more analysis of when the linearization approximation might fail.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n**Computational Efficiency**: The method demonstrates significant computational speedups over deep ensembles (often order-of-magnitude improvements) while maintaining competitive performance on the tested datasets.\n\n**Clear Algorithmic Contribution**: Algorithm 1 is simple, interpretable, and appears to be readily implementable with minimal modifications to existing training pipelines.\n\n**Empirical Performance**: Within the tested domains (toy problems, UCI regression datasets, and small-scale image classification), the method shows competitive performance.\n\n### **Weaknesses**\n\n**Theoretical Foundation**: The core mathematical construction in equation (4) is presented as general but only holds under specific loss function assumptions (strong/strict convexity). For cross-entropy loss used in classification experiments, equation (4) may not even represent valid solutions to optimization problem (3). This makes the theoretical analysis somewhat irrelevant to the larger-scale experimental validation.\n\n**Theory-Practice Disconnect**: The meaningful deep learning experiments (classification on MNIST/CIFAR) operate entirely outside the theoretical framework. The remaining empirical evaluation consists only of nine UCI regression tasks and toy experiments, severely limiting the practical relevance of the work.\n\n**Evaluation Methodology**: The paper introduces VMSP (Variance of Maximum Softmax Probability) as a UQ metric without explicit definition or proper theoretical justification (even taking into account appendix C). \n\n**Limited Theoretical Novelty**: The core ideas of network linearization, NTK connections, and sample-then-optimize frameworks have been extensively explored. Particularly, Immer et al. (2021) already studied linearization, refinement through additional training, and NTK-GP connections. The contribution appears incremental rather than fundamental.\n\n**Inadequate Experimental Scope**:\n\n* Experiments limited to MNIST/CIFAR-scale datasets and outdated architectures rather than modern benchmarks and model architectures\n* Missing evaluation on uncertainty-specific benchmarks (e.g., diabetic retinopathy detection tasks by Band (2022) or others)  \n* No comparison with important recent UQ methods, e.g. Immer et al. (2021).  \n* Related work discussion relegated entirely to appendix\n\n### **Technical Concerns**\n\n**Minor Missing Details**:\n\n* Pseudo-inverse specification (left vs. right) not clarified  \n* Insufficient explanation of why objective perturbation (standard in sample-then-optimize?) is unnecessary  \n* Hyperparameter γ tuning may lead to validation overfitting without principled prior specification\n\n**Methodological Issues**: The approach forgoes having a principled prior and predictive variance formulation, potentially making it more prone to overfitting on validation data compared to proper Bayesian methods.\n\n**Scalability Limitations**: While computationally faster than deep ensembles, scalability to larger & more modern architectures remains undemonstrated, and the linearization approximation quality at scale is unclear.\n\n---\n\nImmer, Alexander, Maciej Korzepa, and Matthias Bauer. \"Improving predictions of Bayesian neural nets via local linearization.\" International conference on artificial intelligence and statistics. PMLR, 2021\\.\n\nBand, Neil, et al. \"Benchmarking bayesian deep learning on diabetic retinopathy detection tasks.\" arXiv preprint arXiv:2211.12717 (2022).", "was_revised": true, "final_justification": "The authors have provided a good rebuttal. I'm still split on the novelty, but I'm also not sure if I know the literature well enough. The point about sampling from the null space is not novel by itself: Miani, Marco, Hrittik Roy, and Søren Hauberg. \"Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections.\" AISTATS, 2025, ArXiv Oct 2024, which also projects onto the kernel, and which I'll add to my response. Unlike that paper, this paper remains really hard to understand, though.", "rebuttal": "We thank the reviewer for their feedback and constructive criticism. We hope that the following aids in alleviating your concerns. \n\n> The core mathematical construction in equation (4) is presented as general but only holds under specific loss function assumptions (strong/strict convexity)...The meaningful deep learning experiments (classification on MNIST/CIFAR) operate entirely outside the theoretical framework. The remaining empirical evaluation consists only of nine UCI regression tasks and toy experiments, severely limiting the practical relevance of the work...\n\nWhile our theory does not currently extend to cross-entropy loss, as is common in NTK-related work and many aspects of theory in machine learning more broadly, we would like to highlight the following two points. Firstly, it is practical extensions of the theory to settings where the assumptions fail that are often the most valuable to empirically test. As is seen in our results, we find that our framework also correctly quantifies the uncertainty in a range of image classification tasks. Secondly, we note that our theory does hold for classification using the Brier loss, and models trained using this loss have been shown to perform similarly to cross-entropy based models [1]. Testing NUQLS with the Brier loss is therefore expected to yield comparable performance and presents an interesting direction for future work. We have clarified these points in the updated version of our paper.\n\n> The authors...could better address the theory-practice disconnect and provide more analysis of when the linearization approximation might fail... How does the method perform when the linearization approximation quality degrades?\n\nWe kindly refer the review to the _confidence intervals_ experimental results we have provided in our response to Reviewer Z6xt. We observe from the first given table that, in line with its theoretical construction, our method does not perform very well for poorly trained networks. This is because far away from a local minima, the linearization approximation may be poor. Further, we rely on the 'data-dependent' NTK that is learned during training [2]. While this is a space where PNC becomes a good method to choose, we do not view it as a limitation of our work. This is because (useful) networks in practice are well-trained; we observe in the second table of results that the performance of NUQLS is excellent once the network has been trained.\n\n> The paper introduces VMSP (Variance of Maximum Softmax Probability) as a UQ metric without explicit definition or proper theoretical justification (even taking into account appendix C).\n\nWe are happy to provide a more explicit definition of VMSP: for a Bayesian method, we generally have a mean predictor $\\mu : \\mathbb{R}^d \\to \\mathbb{R}^c$ and a covariance function $\\Sigma : \\mathbb{R}^d \\to \\mathbb{R}^{c \\times c}$, for example the mean and covariance of the linearized ensemble in the case of NUQLS, that output in the probit space. To compute VMSP for a given test point $\\mathbf{x}^\\star$, we first find $\\hat{c} = \\text{argmax}\\_k ~\\mu(\\mathbf{x}^{*})\\_{k}$, where $\\mu(\\mathbf{x}^\\star)\\_k$ denotes the $k$-th output of $\\mu(\\mathbf{x}^\\star)$. That is, we find the class that the Bayesian method predicts, given $\\mathbf{x}^\\star$. We then define $\\text{VMSP} := \\Sigma(\\mathbf{x}^\\star)_{\\hat{c},\\hat{c}} = \\sigma^2(\\mathbf{x}^\\star)\\_{\\hat{c}}$, that is, the variance of this prediction. We will include this description in the paper, to aid comprehension of the metric. \n\nAs for theoretical justification, we can also be more explicit: we have detailed in Appendix C how the uncertainty of the predictions is captured by their variance. For a given dataset, we want a UQ method to provide low uncertainty for correctly predicted test points, and high uncertainty for incorrectly predicted or OOD test points. We then compare these distributions pictorially using a violin plot, and quantitatively using the median and skew values for the respective distributions. With the addition of a poorly-performing baseline model, we are able to easily compare the ability of UQ models to quantify uncertainty. \n\n> The core ideas of network linearization, NTK connections, and sample-then-optimize frameworks have been extensively explored. Particularly, Immer et al. (2021) already studied linearization, refinement through additional training, and NTK-GP connections. The contribution appears incremental rather than fundamental...No comparison with important recent UQ methods, e.g. Immer et al. (2021).\n\nTo be clear, we have in fact compared to the work of Immer et al. (2021) (i.e. LLA) in most of the experiments, and kindly refer the reviewer to Appendix B.1, where we describe in detail the connection to, and differences with, the LLA framework methods. To summarize, the work of Immer et. al. showed how forming a Laplace Approximation over the parameter posterior of a linearized model resulted in a noisy NTK-GP predictive posterior. This approach incurs several challenges: forming the posterior requires explicitly forming and inverting the NTK gram matrix, independent GP's are required for each class in classification, and approximations such as last-layer and KFAC are required for scaling. In comparison, our method skips the parameter posterior, and uses the randomness in the null-space of the Jacobian to form a noise-free GP. Due to our derivation, we are able to easily sample from this GP posterior using linearized networks, without requiring covariance structure approximations. This enables us to out-perform LLA in larger image classification tasks (see Figure 3 in main text).\n\n> Experiments limited to MNIST/CIFAR-scale datasets and outdated architectures rather than modern benchmarks and model architectures...While computationally faster than deep ensembles... the linearization approximation quality at scale is unclear...Missing evaluation on uncertainty-specific benchmarks (e.g., diabetic retinopathy detection tasks by Band (2022) or others)...Can you demonstrate scalability to larger, more modern architectures? And additional datasets with additional epistemic uncertainty metrics?\n\nWe thank the reviewer for raising this point, as the evaluation of our method on a larger problem will add to the value of our paper. We have now trained and evaluated NUQLS on ResNet50 trained on **ImageNet**, and using the ImageNet-o dataset for out-of-distribution testing [3]. We employed the pre-trained weights found on _torch.hub_ for ResNet50 / ImageNet. While we cannot provide an image of the violin plot of the VMSP, we can still provide the median values (where (>) denotes values which we want to be large, and (<) those we want to be small):\n\n|         | NUQLS   | BASE    |\n|---------|---------|---------|\n| IDC (<)      | $0.00$ | $0.0206$ |\n| IDIC (>)    | $0.232$ | $0.0197$ |\n| OOD (>)     | $0.100$ | $0.0197$ |\n\nand the skew values:\n\n|         | NUQLS   | BASE    |\n|---------|---------|---------|\n| IDC (>)    | $1.52$ | $1.21$ |\n| IDIC (<)   | $-0.88$ | $1.10$ |\n| OOD (<)    | $0.179$ | $0.985$ |\n\nFor the in-distribution test set, we see excellent separation between the correct and incorrect predictions. As was stated in [4], approximately $21$\\% of the images in ImageNet-o are in-distribution. Hence, we would expect that the OOD VMSP values to skew slightly more towards zero than the in-distribution incorrect predictions, as we observe. We note that ImageNet/ResNet50 is generally considered a large-scale setting for Bayesian methods.\n\n> Related work discussion relegated entirely to appendix\n\nWe have relegated this discussion due to the large amount of related works, and the detail the discussion required. However, we can include a condensed version in the main body, given the additional space allowed in the camera-ready version.\n\n> Pseudo-inverse specification (left vs. right) not clarified\n\nAs we assume $J_X$ to be full-row rank, the pseudo-inverse is computed as the right inverse. \n\n> Insufficient explanation of why objective perturbation (standard in sample-then-optimize?) is unnecessary\n\nWe appreciate the opportunity to clarify this point, as this is a contribution of the paper. Standard sample-then-optimize works rely on randomness in the loss function. Instead, NUQLS relies on the fact that we are in an underdetermined linear system, and hence we can form an ensemble of solutions simply through the variability of random projections onto the null-space of the Jacobian. Thus, we do not require any variability in the loss function to sample from the posterior, in contrast to other sample-then-optimize methods. \n\n> The approach forgoes having a principled prior and predictive variance formulation, potentially making it more prone to overfitting on validation data compared to proper Bayesian methods... Hyperparameter $\\gamma$ tuning may lead to validation overfitting without principled prior specification\n\nWe respectfully disagree with the reviewer that we do not have a principled predictive variance formulation, as the predictive distribution follows a Gaussian Process. In regards the prior, we do not find evidence of overfitting on validation data, and instead show very competitive results. Further, we find that it is common for Bayesian methods to choose a Guassian isotropic prior.\n\nPlease let us know if there are any other points that we can clarify, as we are eager to engage further to improve your assessment of our work.\n\n---\nReferences:\n[1] Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks, Hui et. al. 2020\n\n[2] Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel, Fort et. al. 2020\n\n[3] Natural Adversarial Examples, Hendrycks et. al. 2017\n\n[4] In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation, Bitterwolf et. al. 2023"}, {"review_id": "lccuh9mwNY", "reviewer": "Reviewer_Z6xt", "summary": "The paper proposes a post-hoc, sampling-based uncertainty quantificaiton method for overparameterized networks by approximating the predictive distribution of deep models using Gaussian Processes (GPs) by neural tanget kernels. The proposed method, NUQLS first update the learned parameters $\\hat{\\theta}$ with $S$ steps for the posterior (9). The method has lower computational complexity than full Bayesian neural networks (BNNs) and avoids costly ensemble-based inference.The authors demonstrate the performance of their method on regression and classification benchmarks, comparing against deep ensemble, SWAG, MC, and LLA.", "questions": "- Robustness to Representation Quality:\nHow sensitive is your method to the quality of the learned representations? Would randomly initialized or undertrained networks still yield calibrated uncertainty?\n\n-Non-Gaussian Outputs:\nHow do you handle classification tasks where predictive distributions are inherently non-Gaussian (e.g., multimodal posteriors)? Does your method underperform in such settings?", "limitations": "The method’s performance is evaluated primarily on small to medium-scale datasets, limiting the generalizability of its claims to larger or more diverse real-world scenarios.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tThe idea of using random features to approximate a GP on top of neural network embeddings is simple.\n2.\tThe paper evaluates expected calibration error (ECE), negative log-likelihood (NLL), and predictive intervals on standard datasets such as UCI benchmarks and CIFAR10.\nWeaknesses\n1.\tUnlike the NeurIPS 2023 PNC paper, the submission lacks rigorous statistical theory (e.g., asymptotic coverage guarantees, bias-variance decomposition) to back the empirical results. Additionally, compared to the SWAG paper, insufficient empirical evidence is provided to demonstrate how Equation (9) (the main result) relates to uncertainty quantification.\n2.\tThe method relies on approximate Gaussianity of the hidden-layer representations' output distribution and the strong convexity of the loss function. These assumptions may break down for complex real-world classification tasks (especially with multimodal outputs), yet this limitation is not critically analyzed.\n3.\tThere is little discussion on when or why the method might fail (e.g., poorly trained base networks, overfitting, high-dimensional noise in embeddings).", "was_revised": true, "final_justification": "While the empirical performance of the method is strong, the theoretical justification is somewhat unclear. In particular, Lemma 3.1 is derived using convexity, but its proof appears to be questionable. The use of convex loss to argue for a unique solution in a local region also seems unnecessary. Given these issues, I decided to keep the score.", "rebuttal": ">  Unlike the NeurIPS 2023 PNC paper, the submission lacks rigorous statistical theory (e.g., asymptotic coverage guarantees, bias-variance decomposition) to back the empirical results.\n\nWe appreciate the opportunity to highlight the differences between the PNC paper and ours. Firstly, PNC is formed from a frequentist framework, and hence, similar to conformal predictions, they are able to get statistical guarantees on confidence interval coverage. Further, PNC relies on the NTK theory, where for infinitely wide networks, the network behaves according to kernel regression. From this, the authors are able to estimate possible function values. Instead, NUQLS relies on the network being trained to a local minima, and then employs the feature-learning of finite-width networks to create meaningful ensembles. \n\nTo test the difference between the two methods, we tested NUQLS on the _confidence intervals_ problem from [3]. In this experiment, an MLP with a single hidden-layer is trained on $n$ datapoints from $U([0,0.2]^d)$, where the target is $y = \\sum_{i=1}^d \\sin(x_i) + \\epsilon$, and $\\epsilon$ is a small noise term. The method is then asked to form a confidence interval around the prediction for $x = (0.1, \\dots, 0.1)$. This setup is repeated several times, and the coverage and width of the confidence intervals are recorded, as well as the mean prediction. For more details, we refer the reviewer to [3]. When the width of the network is $32 \\times n$, and the network is only partially trained with $80$ epochs and a learning rate of $0.01$, NUQLS performs poorly compared to PNC, as can be seen in the following table. Note that bolded numbers indicate intervals that have reached or exceeded the expected coverage. Further, smaller width intervals are preferred.\n\n\n|                 |      PNC     |              |        |     NUQLS    |              |        |\n|:---------------:|:------------:|:------------:|:------:|:------------:|:------------:|:------:|\n|                 | 95\\%CI (CR/IW) | 90\\%CI (CR/IW) |   MP   | 95\\%CI (CR/IW) | 90\\%CI (CR/IW) |   MP   |\n| (d = 2) n = 128 |  **0.98**/0.0437 |  **0.95**/0.0323 | 0.1998 |  0.93/0.0357 |  **0.92**/0.0299 | 0.2047 |\n| (d = 4) n = 256 |  **0.98**/0.0411 |  **0.95**/0.0304 | 0.3991 |  0.92/0.0596 |  0.86/0.0500 | 0.4084 |\n\nWe note that scaling the width of a network by $32$ times the number of training points is rare in practice, and that such a wide-network is difficult to train. If we instead form an MLP with width equal to the number of training points, and increase epochs to $1000$ and learning rate to $0.5$, so that the network is properly trained, NUQLS far outperforms PNC, as can be seen in the following table. We note that for these experiments we tuned the $\\gamma$ hyper-parameter for NUQLS on a small validation set.\n\n|                  |      PNC     |              |        |     NUQLS    |              |        |\n|:----------------:|:------------:|:------------:|:------:|:------------:|:------------:|:------:|\n|                  | 95\\%CI (CR/IW) | 90\\%CI (CR/IW) |   MP   | 95\\%CI (CR/IW) | 90\\%CI (CR/IW) |   MP   |\n|  (d = 2) n = 128 |   0.8/0.136  |  0.72/0.0105 | 0.2022 |  **0.99**/0.0135 |  **0.96**/0.0114 | 0.2012 |\n|  (d = 4) n = 256 |  **0.96**/0.0437 |  **0.9**/0.0336  | 0.4045 |  **0.97**/0.0313  |  **0.92**/0.0263 | 0.4030 |\n|  (d = 8) n = 512 |  0.88/0.0740 |  0.88/0.0568 | 0.8078 |  **1.00**/0.0667 |  **0.98**/0.0559 | 0.8050 |\n| (d = 16) n = 128 |  0.8/0.1443  |  0.8/0.1108  | 1.6265 |  **1.00**/0.1350 |  **0.98**/0.1133 | 1.6121 |\n\nWe observe that NUQLS and PNC are in fact complementary methods. For infinite-width networks near initialization, PNC performs well while NUQLS struggles. Conversely, for finite-width networks trained to a minimum, NUQLS excels while PNC performs poorly. We believe the latter regime is more representative of practical scenarios, where NUQLS offers significantly better performance. \n\nFinally, we note that rigorous statistical guarantees that exist for PNC do not currently exist for Bayesian methods, a point that is raised in the PNC paper [3].\n\n> Compared to the SWAG paper, insufficient empirical evidence is provided to demonstrate how Equation (9) (the main result) relates to uncertainty quantification\n\nIn regards to the experiments in the submitted paper, we find that under a variety of regression and classification tasks, our method does in fact accurately quantify the uncertainty in the predictions of a neural network. Further, we have now also provided an extra regression experiment, i.e. the _confidence intervals_ experiment from [3], and a large-scale image classification experiment, i.e. the **ResNet50/ImageNet** result that has been provided in the response to Reviewer j2t5. From this, we hope that the reviewer finds adequate empirical evidence that the variance term derived in (9) does in fact accurately quantify the uncertainty in a neural networks predictions. \n\n> The method relies on approximate Gaussianity of the hidden-layer representations' output distribution and the strong convexity of the loss function. These assumptions may break down for complex real-world classification tasks (especially with multimodal outputs), yet this limitation is not critically analyzed...How do you handle classification tasks where predictive distributions are inherently non-Gaussian (e.g., multimodal posteriors)? Does your method underperform in such settings?\n\nWe would like to respectfully clarify a potential misunderstanding: we do not assume Gaussianity for the original neural network at any point. Rather, if the network is approximated by its linearization—a reasonable assumption near a local minimum—we show that the predictive distribution of the linearized model follows a Gaussian Process. Consequently, the predictive distribution of the original network is approximately Gaussian in this regime.\n\nEven in real-world classification tasks such as ImageNet with ResNet-50, where the posterior is likely multimodal, we observe excellent uncertainty quantification. While we have no guarantee that the predictive distribution is exactly Gaussian in these settings, we find that the linearized networks form meaningful ensembles that effectively explore the local function space and capture predictive uncertainty.\n\n\n> There is little discussion on when or why the method might fail (e.g., poorly trained base networks, overfitting, high-dimensional noise in embeddings)...How sensitive is your method to the quality of the learned representations? Would randomly initialized or undertrained networks still yield calibrated uncertainty?\n\nThis is an insightful question, and has prompted an important addition to our paper. As has been previously noted, we can see from the attached _confidence interval_  results that untrained or undertrained networks do not necessarily yield well-calibrated uncertainty estimates. Instead, if the network is very wide and poorly trained, a method like PNC would be more appropriate. However, we again note that this would be uncommon in practice, as most networks will be well-trained. In the case of well-trained networks, we find that NUQLS excels. \n\n> The method’s performance is evaluated primarily on small to medium-scale datasets, limiting the generalizability of its claims to larger or more diverse real-world scenarios.\n\nWe kindly refer the reviewer to our response to Reviewer j2t5, where we show results for ResNet50 trained on **ImageNet**. This setting is generally considered large-scale in the Bayesian UQ commmunity.\n\nPlease let us know if you have any questions or there is something else that needs clarification, as we are eager to increase your assessment of our work.\n\n---\nReferences:\n\n[1] Deep learning versus kernel learning: an empirical\nstudy of loss landscape geometry and the time\nevolution of the Neural Tangent Kernel, NeurIPS2020\n\n[2] Harnessing the Power of Infinitely Wide Deep Nets on Small-data\nTasks, Arora et. al. 2019. \n\n[3] Efficient Uncertainty Quantification and Reduction for\nOver-Parameterized Neural Networks, Huang et. al. 2023"}, {"review_id": "Asv4ShT8Up", "reviewer": "Reviewer_CkYf", "summary": "This paper investigates uncertainty quantification in neural networks. Specifically, it constructs an ensemble of linear models centered around the parameters of a trained deep neural network (DNN). To generate diverse initializations, isotropic Gaussian noise is added to the trained parameters. The paper presents a theoretical analysis showing that, under a regression loss, the proposed method samples models from an approximate posterior of the neural network, which corresponds to a Gaussian process defined by the neural tangent kernel. Empirically, the method demonstrates improved calibration of uncertainty estimates compared to baseline approaches on both regression and classification tasks.", "questions": "Since all linearized models are initialized around the same local optimum, is there a risk of mode collapse—i.e., all models converging to similar behavior? A discussion of this potential limitation and how it is mitigated would be helpful.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- Uncertainty quantification is a critical and largely unsolved challenge in deep learning. While Bayesian neural networks are principled, they often underperform in practice, and deep ensembles, though effective, are computationally expensive. The proposed method offers a compelling compromise between these two extremes.\n\n- The paper is clearly written and well-structured.\n\n- Experimental results are promising and demonstrate improved calibration over baselines on multiple tasks.\n\n**Weaknesses:**\n\n- The comparison to prior work is limited. Several efficient ensemble methods, such as BatchEnsemble [1], Snapshot Ensembles [2] were proposed. SNGP [3] are highly relevant but not included in the evaluation. Additionally, LLA is not a particularly strong Bayesian baseline.\n\n- Although the paper evaluates both regression and classification tasks, it would strengthen the contribution to demonstrate that improved uncertainty estimates lead to better downstream performance (e.g., in active learning or decision-making under uncertainty).\n\n- Prior work has studied the connection between uncertainty quantification and the neural tangent kernel (NTK). It would be helpful for the authors to clarify how their method differs from or extends these works, such as [4].\n\n**References:**\n\n[1] *BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning*, ICLR 2020  \n[2] *Training Independent Subnetworks for Robust Prediction*, ICLR 2021  \n[3] *Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness*, NeurIPS 2022  \n[4] *Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks*, NeurIPS 2023", "was_revised": true, "final_justification": "The author has provided additional experimental results to address my initial concerns during the rebuttal. So I think a weak accept is appropriate for this paper.", "rebuttal": "We thank the reviewer for their careful reading of our work, and constructive criticism. \n\n> The comparison to prior work is limited. Several efficient ensemble methods, such as BatchEnsemble [1], Snapshot Ensembles [2] were proposed. SNGP [3] are highly relevant but not included in the evaluation. Additionally, LLA is not a particularly strong Bayesian baseline.\n\nWe thank the reviewer for bringing attention to the importance of these methods. Unfortunately due to the limited time of the rebuttal period, we were only able to impliment one of the referenced methods, of which we chose SNGP, as there exists a Lightning Pytorch implementation of the method [5]. We tested SNGP using VMSP on both ResNet9 / FashionMNIST and ResNet50 / SVHN. The median results are presented in the following table. We write $(>)$ to denote values which we want to be large, and $(<)$ for values we want to be small (synonymous with the up and down arrows in the paper, respectively).\n\n|               | NUQLS                  | SNGP                  | BASE    |\n|---------------|------------------------|-----------------------|---------|\n| R9FM: IDC (<)      | $4.76 \\times 10^{-15}$ | $9.19 \\times 10^{-7}$ | $0.020$ |\n| R9FM: IDIC (>)    | $0.182$                | $3.22 \\times 10^{-4}$ | $0.020$ |\n| R9FM: OOD (>)     | $0.217$                | $3.42 \\times 10^{-4}$ | $0.020$ |\n| R50SVHN: IDC (<)   | $8.61 \\times 10^{-15}$ | $3.64 \\times 10^{-6}$ | $0.020$ |\n| R50SVHN: IDIC (>) | $0.217$                | $3.64 \\times 10^{-6}$ | $0.020$ |\n| R50SVHN: OOD (>)  | $0.233$                | $3.66 \\times 10^{-6}$  | $0.020$ |\n\nand the skew results:\n\n|               | NUQLS    | SNGP   | BASE    |\n|---------------|----------|--------|---------|\n| R9FM: IDC (>)      | $2.40$   | $29.8$ | $1.08$  |\n| R9FM: IDIC (<)    | $-0.615$ | $24.4$ | $1.09$  |\n| R9FM: OOD (<)     | $-1.70$  | $1.21$ | $1.06$  |\n| R50SVHN: IDC (>)  | $2.2$    | $0.309$ | $0.020$ |\n| R50SVHN: IDIC (<) | $-1.31$  | $0.284$ | $0.020$ |\n| R50SVHN: OOD (<)  | $-1.78$  | $0.304$ | $0.020$ |\n\nWe see that on ResNet9/FashionMNIST, NUQLS far out-performs SNGP. For ResNet50 / SVHN, $3$ learning rates and $2$ optimizers were tested; unfortunately SNGP did not properly train for the tested combinations, and hence the best results are very poor. \n\nNote that we focused our comparison on LLA variants as they have become popular recently, with the release of the Laplace package [5], and through scalable variants such as Sampling-LLA and VaLLA. \n\nWe believe that the efficient ensemble methods are also important methods to consider, and hence we are working toward including results comparing these methods to NUQLS before the end of the review period. \n\n> Although the paper evaluates both regression and classification tasks, it would strengthen the contribution to demonstrate that improved uncertainty estimates lead to better downstream performance (e.g., in active learning or decision-making under uncertainty).\n\nWe appreciate the suggestion of the reviewer to display the performance of NUQLS on a downstream task, as we believe the results would be interesting and would strengthen the value of the paper. Unfortunately we are unable to run a well-thought out, large scale experiment in the rebuttal period due to time constraints. However, we will include such an evaluation in future work. \n\n> Prior work has studied the connection between uncertainty quantification and the neural tangent kernel (NTK). It would be helpful for the authors to clarify how their method differs from or extends these works, such as [4].\n\nThe referenced work PNC is very interesting, and the connection of NUQLS to PNC will be included in the paper. We refer the reviewer to our response to Reviewer Z6xt, where we discuss the inherent differences between NUQLS and PNC, as well as empirically compare them on an experiment from [4]. To summarise, PNC is developed under the asymptotic NTK theory and performs well for very wide networks near initialization (i.e., poorly trained networks), where NUQLS struggles. In contrast, NUQLS does not rely on the asymptotic NTK regime and instead leverages the trained, data-dependent NTK to construct meaningful linear ensembles. For finite-width networks that are well trained—which we believe to be the practically relevant setting—NUQLS is more appropriate and clearly outperforms PNC.\n\n\n> Since all linearized models are initialized around the same local optimum, is there a risk of mode collapse—i.e., all models converging to similar behavior? A discussion of this potential limitation and how it is mitigated would be helpful.\n\nThis is an excellent question, and we're glad to offer a concrete answer: **mode collapse cannot occur—almost surely**.\n\nTo see this, note that the solution to equation (3) in our main paper for each linearized model consists of a unique row-space component (under our assumptions), plus a projection of the initialization $z_i$ onto the null space of the Jacobian. For mode collapse to occur, the projections for two i.i.d. initializations $z_1, z_2 \\sim \\mathcal{N}(0, \\gamma^2 I)$ would need to coincide—that is, we would require $(I - J^{\\dagger}J)(z_1 - z_2) = 0$, even though $z_1 - z_2 \\neq 0$ almost surely. This implies $z_1 - z_2 \\in \\text{Range}(J^T)$. However, $\\text{Range}(J^T)$ is a low-dimensional subspace of $\\mathbb{R}^p$, while $z_1$ and $z_2$ are drawn iid from full-rank (i.e., non-degenerate) $p$-dimensional Gaussian distribution. The probability that $z_1 - z_2$ lies in a fixed lower-dimensional subspace is zero. Therefore, mode collapse occurs with probability zero, and we can safely rule it out as a concern.\n\n\nWe hope that this has alleviated your concerns with our work. Please let us know if there are any further questions of points that require clarification, as we would like to engage further if it will improve your assessment of our paper. \n\n---\nReferences:\n\n[1] BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, ICLR 2020\n\n[2] Training Independent Subnetworks for Robust Prediction, ICLR 2021\n\n[3] Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness, NeurIPS 2022\n\n[4] Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks, NeurIPS 2023\n\n[5] Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning, Lehmann et. al. 2024\n\n[6] Laplace Redux – Effortless Bayesian Deep Learning, Daxberger 2021"}, {"review_id": "tlQxUk9HbX", "reviewer": "Reviewer_4xTc", "summary": "This paper proposes a post-hoc ensembling method that approximately samples from the NTK GP posterior under certain conditions, mostly related to the convexity of the loss in the predictions of the network. The method linearizes the neural network, and reformulates the objective for the linearized network. This objective is optimized for multiple times from multiple randomized starting points that are obtained by distorting the original solution with Gaussian noise. \n\nThe method is motivated theoretically by Theorem 3.2 and Corollary 3.5, which roughly state that the proposed algorithm samples from the NTK GP posterior when the loss is convex. \n\nThe method is also tested on a toy regression task and several regression and classification data sets. The latter being interesting because the assumptions of Theorem 3.2 do not work, even though practical performance is still high. The authors test with established metrics they deem flawed as well as a newly proposed metric that creates violin plots of the variance of predictions.", "questions": "- Could the authors elaborate on the new UQ metric and its novelty? Is it the idea of taking the variance of the softmax outputs, or the act of visually inspecting the resulting violin plots? \n\n- Can the authors elaborate on the computations described in appendix D, perhaps in mathematical forms?", "limitations": "Limitations are addressed adequately. The authors acknowledge that their theoretical motivation relies on assumptions that are often violated, which is common in NTK theory.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a good paper. It is well written and easy to read. The introduction and background gives a good overview of current literature, although I was missing perhaps gradient based MCMC methods which also have been applied to neural networks, see e.g. [1-3].\n\nThe theoretical justification is laid out well and not too difficult to follow. Such theoretical justifications are also especially important for uncertainty quantification which is inherently difficult to benchmark in cases where we do not have access to the true posterior. I resonate strongly with the messages in Appendix C about flaws in common benchmarking metrics, although I can't speak to the novelty of the proposed UQ metric.\n\nThe experiments are well designed and the algorithm shows good performance on established (but possibly flawed) metrics as well as the proposed UQ metric. I understand the reasoning to exclude BDE, but do not understand how the same reasoning would not also apply to DE. I think including BDE in the image classification experiments could benefit the clarity and transparency of the paper, and it would also be interesting to see how BDE compares to DE on the newly proposed metric. Finally, I would also have liked to see an SGMCMC method in the baselines.\n\nOverall, this is a very interesting work, and uncertainty quantification for neural networks is an important research topic. I believe the field would benefit from this paper at a high profile venue like NeurIPS.\n\n[1] Wenzel, Florian, et al. \"How good is the bayes posterior in deep neural networks really?.\" _arXiv preprint arXiv:2002.02405_ (2020).\n\n[2] Izmailov, Pavel, et al. \"What are Bayesian neural network posteriors really like?.\" _International conference on machine learning_. PMLR, 2021\n\n[3] Garriga-Alonso, Adrià, and Vincent Fortuin. \"Exact langevin dynamics with stochastic gradients.\" _arXiv preprint arXiv:2102.01691_ (2021).", "was_revised": true, "final_justification": "The authors have fully addressed my questions in their rebuttal to my review and in the rebuttal to j2t5. They have also added multiple baselines that convince me that the experimental results are sound and that the conclusions based on them are reliable. I feel that my original rating of 5 accurately reflects the quality of the paper.", "rebuttal": "> I was missing perhaps gradient based MCMC methods which also have been applied to neural networks, see e.g. [1-3]...I would also have liked to see an SGMCMC method in the baselines.\n\nWe thank the reviewer for bringing attention to the SGMCMC methods. We originally did not compare to any of these methods due to the often large cost associated with computing them in high-dimensions, and due to some poor performance in previous UQ benchmarks (albiet computed using flawed metrics). However, we appreciate the importance of SGMCMC to the Bayesian framework, in that they are able to compute approximate samples from the posterior. We are actively working toward addressing your comment and will endeavour to include an SGMCMC method in our comparisons before the end of the rebuttal period.\n\n\n>  I understand the reasoning to exclude BDE, but do not understand how the same reasoning would not also apply to DE. I think including BDE in the image classification experiments could benefit the clarity and transparency of the paper, and it would also be interesting to see how BDE compares to DE on the newly proposed metric.\n\nThis is certainly an interesting point, and we appreciate the opportunity to clarify. We find that the prevailing opinion in current Bayesian UQ works is that DE is still the SOTA, even with the addition of BDE. Further, we find it rare that works will compare with BDE, rather than simply comparing with DE. Hence, we currently rank DE as the highest performing method. A primary goal of our work was to achieve performance similar to DE, whilst reducing the computational cost. We then sought to compare with other methods that seek to achieve a similar goal. Hence, this is why we excluded BDE from the image classification results. However, we agree that a comparison of BDE to both DE and NUQLS on this new metric is enlightening, as BDE may in fact show better performance than DE. Further, the similarity / difference between BDE and NUQLS in this setting will be interesting to observe. Thus, we will aim to evaluate BDE on an image classification result and hope to provide the results before the end of the rebuttal author-reviewer period. \n\n> Could the authors elaborate on the new UQ metric and its novelty? Is it the idea of taking the variance of the softmax outputs, or the act of visually inspecting the resulting violin plots?\n\nThis is a great question, and we are happy to provide further details on this point that we may have skipped over in the main text. We firstly find that no previous works have employed the variance of the softmax outputs as the quantity used to compare UQ performance. For example, most works simply use the softmax outputs themselves to form evaluation metrics, either by taking the maximum class prediction as the confidence, or the entropy over the classes. The SLU work [4] used the variance of the logits to compute the AUCROC between ID and OOD points (see Appendix G.1 in our work for a small comparison to SLU on this metric). However, we are unaware of other works that have employed the variance of the softmax outputs. Secondly, previous works, such as SWAG (Appendix E.2), have plotted the entropic histograms of the softmax predictions for both ID and OOD points. To our knowledge though, we are the first work which employs violin plots, that compares the median and skew values, and that compares to a poorly performing baseline. Hence, we see that the metric is novel in both the quantity it uses for evaluation, and the manner in which it compares performance between methods. Note that we also provide further details on the VMSP metric in response to a question from reviewer j2t5. \n\n> Can the authors elaborate on the computations described in appendix D, perhaps in mathematical forms?\n\nWe are happy to provide further mathematical details on the computations in Appendix D. We compare the computational complexity for an epoch of training for the neural network $f_\\theta(x)$, for batch $x \\in \\mathbb{R}^{d \\times n}$, and an epoch of training for a single linear network $\\hat{f}\\_\\theta(x)$. We take approximations on the computational complexity of both forward-mode AD and backward-mode AD from the following text [5]. Specifically, we take $[fp]$ as the computational complexity for evaluating $f_\\theta(x)$.\n\nWe note from [5] that both a JVP and a VJP cost roughly $2$ $\\times$ a forward pass in computational complexity and memory. Further, both a JVP and a JVP return a function evaluation. Now, a standard epoch of training for the neural network involves a forward-pass to compute the error, and then a backward-pass (VJP), hence the complexity is approx. $3[fp]$. The linearised network involves a JVP (which includes a function evaluation) to form the linear network, and a VJP to compute the gradient. Hence, the complexity for an epoch of training for the linearized network is approx. $4[fp]$. So we observe that each epoch for a linearized network is only $4/3~\\times$ as expensive as for a neural network.  In regards memory, we see that the memory requirement for both the linearized network and the neural network are similar. However, we generally train all linearized networks in parallel. For computational complexity, this will incur some additional cost, thought it will not be linear in number of networks, that is dependent upon the specific software and hardware. However, memory cost will scale linearly by number of networks, i.e. $2S \\times M([fp])$, where $M([fp])$ is the memory cost for a forward-pass, and $S$ is the number of linear networks. As an example, we employ a batch size of $56$ for ImageNet on ResNet50 with $10$ ensemble members when using an 80GB H100 GPU, due to the large parameter count and large number of classes. \n\nThese improvements have been made to the working copy of our paper (pending additional experiments). Please let us know if you have any other questions or points that need clarification. \n\n---\nReferences:\n\n[1] Wenzel, Florian, et al. \"How good is the bayes posterior in deep neural networks really?.\" arXiv preprint arXiv:2002.02405 (2020).\n\n[2] Izmailov, Pavel, et al. \"What are Bayesian neural network posteriors really like?.\" International conference on machine learning. PMLR, 2021\n\n[3] Garriga-Alonso, Adrià, and Vincent Fortuin. \"Exact langevin dynamics with stochastic gradients.\" arXiv preprint arXiv:2102.01691 (2021).\n\n[4] Sketched Lanczos uncertainty score: a low-memory summary of the Fisher information, Miani et. al. 2024\n\n[5] The Elements of Differentiable Programming, Blondel et. al. 2024, Chapter 8"}]

Author rebuttal:

Meta-review: This paper proposes a post-hoc, sampling-based uncertainty quantification (UQ) method for overparameterized networks at the end of training. The approach constructs efficient and meaningful deep ensembles by employing a (stochastic) gradient-descent sampling process on appropriately linearized networks.  Through a series of numerical experiments, the authors show that their method not only outperforms competing approaches in computational efficiency, but also maintains state-of-the-art performance across a variety of UQ metrics for both regression and classification tasks. The reviewers have stated that the method demonstrates significant computational speedups over deep ensembles. Moreover, it is simple and easy to implement, and under the tested domains is very competitive. They also say that the paper is clearly written and well-structured. Overall, this is a very interesting paper, and uncertainty quantification for neural networks is a key research topic. Therefore, I believe the field would benefit from this paper.
Decision: Accept (poster)