Title: Uncertainty Quantification with the Empirical Neural Tangent Kernel
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural Tangent Kernel

Bayesian Deep Learning

Uncertainty Quantification

Neural Network
Abstract: While neural networks have demonstrated impressive performance across various tasks, accurately quantifying uncertainty in their predictions is essential to ensure their trustworthiness and enable widespread adoption in critical systems. Several Bayesian uncertainty quantification (UQ) methods exist that are either cheap or reliable, but not both. We propose a post-hoc, sampling-based UQ method for overparameterized networks at the end of training. Our approach constructs efficient and meaningful deep ensembles by employing a (stochastic) gradient-descent sampling process on appropriately linearized networks. We demonstrate that our method effectively approximates the posterior of a Gaussian Process using the empirical Neural Tangent Kernel. Through a series of numerical experiments, we show that our method not only outperforms competing approaches in computational efficiency--often reducing costs by multiple factors--but also maintains state-of-the-art performance across a variety of UQ metrics for both regression and classification tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post-hoc ensembling method that approximately samples from the NTK GP posterior under certain conditions, mostly related to the convexity of the loss in the predictions of the network. The method linearizes the neural network, and reformulates the objective for the linearized network. This objective is optimized for multiple times from multiple randomized starting points that are obtained by distorting the original solution with Gaussian noise. \n\nThe method is motivated theoretically by Theorem 3.2 and Corollary 3.5, which roughly state that the proposed algorithm samples from the NTK GP posterior when the loss is convex. \n\nThe method is also tested on a toy regression task and several regression and classification data sets. The latter being interesting because the assumptions of Theorem 3.2 do not work, even though practical performance is still high. The authors test with established metrics they deem flawed as well as a newly proposed metric that creates violin plots of the variance of predictions.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a good paper. It is well written and easy to read. The introduction and background gives a good overview of current literature, although I was missing perhaps gradient based MCMC methods which also have been applied to neural networks, see e.g. [1-3].\n\nThe theoretical justification is laid out well and not too difficult to follow. Such theoretical justifications are also especially important for uncertainty quantification which is inherently difficult to benchmark in cases where we do not have access to the true posterior. I resonate strongly with the messages in Appendix C about flaws in common benchmarking metrics, although I can't speak to the novelty of the proposed UQ metric.\n\nThe experiments are well designed and the algorithm shows good performance on established (but possibly flawed) metrics as well as the proposed UQ metric. I understand the reasoning to exclude BDE, but do not understand how the same reasoning would not also apply to DE. I think including BDE in the image classification experiments could benefit the clarity and transparency of the paper, and it would also be interesting to see how BDE compares to DE on the newly proposed metric. Finally, I would also have liked to see an SGMCMC method in the baselines.\n\nOverall, this is a very interesting work, and uncertainty quantification for neural networks is an important research topic. I believe the field would benefit from this paper at a high profile venue like NeurIPS.\n\n[1] Wenzel, Florian, et al. \"How good is the bayes posterior in deep neural networks really?.\" _arXiv preprint arXiv:2002.02405_ (2020).\n\n[2] Izmailov, Pavel, et al. \"What are Bayesian neural network posteriors really like?.\" _International conference on machine learning_. PMLR, 2021\n\n[3] Garriga-Alonso, Adrià, and Vincent Fortuin. \"Exact langevin dynamics with stochastic gradients.\" _arXiv preprint arXiv:2102.01691_ (2021).\n\nQUESTIONS:\n- Could the authors elaborate on the new UQ metric and its novelty? Is it the idea of taking the variance of the softmax outputs, or the act of visually inspecting the resulting violin plots? \n\n- Can the authors elaborate on the computations described in appendix D, perhaps in mathematical forms?\n\nLIMITATIONS:\nLimitations are addressed adequately. The authors acknowledge that their theoretical motivation relies on assumptions that are often violated, which is common in NTK theory.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates uncertainty quantification in neural networks. Specifically, it constructs an ensemble of linear models centered around the parameters of a trained deep neural network (DNN). To generate diverse initializations, isotropic Gaussian noise is added to the trained parameters. The paper presents a theoretical analysis showing that, under a regression loss, the proposed method samples models from an approximate posterior of the neural network, which corresponds to a Gaussian process defined by the neural tangent kernel. Empirically, the method demonstrates improved calibration of uncertainty estimates compared to baseline approaches on both regression and classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Uncertainty quantification is a critical and largely unsolved challenge in deep learning. While Bayesian neural networks are principled, they often underperform in practice, and deep ensembles, though effective, are computationally expensive. The proposed method offers a compelling compromise between these two extremes.\n\n- The paper is clearly written and well-structured.\n\n- Experimental results are promising and demonstrate improved calibration over baselines on multiple tasks.\n\n**Weaknesses:**\n\n- The comparison to prior work is limited. Several efficient ensemble methods, such as BatchEnsemble [1], Snapshot Ensembles [2] were proposed. SNGP [3] are highly relevant but not included in the evaluation. Additionally, LLA is not a particularly strong Bayesian baseline.\n\n- Although the paper evaluates both regression and classification tasks, it would strengthen the contribution to demonstrate that improved uncertainty estimates lead to better downstream performance (e.g., in active learning or decision-making under uncertainty).\n\n- Prior work has studied the connection between uncertainty quantification and the neural tangent kernel (NTK). It would be helpful for the authors to clarify how their method differs from or extends these works, such as [4].\n\n**References:**\n\n[1] *BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning*, ICLR 2020  \n[2] *Training Independent Subnetworks for Robust Prediction*, ICLR 2021  \n[3] *Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness*, NeurIPS 2022  \n[4] *Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks*, NeurIPS 2023\n\nQUESTIONS:\nSince all linearized models are initialized around the same local optimum, is there a risk of mode collapse—i.e., all models converging to similar behavior? A discussion of this potential limitation and how it is mitigated would be helpful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a post-hoc, sampling-based uncertainty quantificaiton method for overparameterized networks by approximating the predictive distribution of deep models using Gaussian Processes (GPs) by neural tanget kernels. The proposed method, NUQLS first update the learned parameters $\\hat{\\theta}$ with $S$ steps for the posterior (9). The method has lower computational complexity than full Bayesian neural networks (BNNs) and avoids costly ensemble-based inference.The authors demonstrate the performance of their method on regression and classification benchmarks, comparing against deep ensemble, SWAG, MC, and LLA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe idea of using random features to approximate a GP on top of neural network embeddings is simple.\n2.\tThe paper evaluates expected calibration error (ECE), negative log-likelihood (NLL), and predictive intervals on standard datasets such as UCI benchmarks and CIFAR10.\nWeaknesses\n1.\tUnlike the NeurIPS 2023 PNC paper, the submission lacks rigorous statistical theory (e.g., asymptotic coverage guarantees, bias-variance decomposition) to back the empirical results. Additionally, compared to the SWAG paper, insufficient empirical evidence is provided to demonstrate how Equation (9) (the main result) relates to uncertainty quantification.\n2.\tThe method relies on approximate Gaussianity of the hidden-layer representations' output distribution and the strong convexity of the loss function. These assumptions may break down for complex real-world classification tasks (especially with multimodal outputs), yet this limitation is not critically analyzed.\n3.\tThere is little discussion on when or why the method might fail (e.g., poorly trained base networks, overfitting, high-dimensional noise in embeddings).\n\nQUESTIONS:\n- Robustness to Representation Quality:\nHow sensitive is your method to the quality of the learned representations? Would randomly initialized or undertrained networks still yield calibrated uncertainty?\n\n-Non-Gaussian Outputs:\nHow do you handle classification tasks where predictive distributions are inherently non-Gaussian (e.g., multimodal posteriors)? Does your method underperform in such settings?\n\nLIMITATIONS:\nThe method’s performance is evaluated primarily on small to medium-scale datasets, limiting the generalizability of its claims to larger or more diverse real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NUQLS (Neural Uncertainty Quantification by Linearised Sampling), a post-hoc method that draws an ensemble of linearised networks around trained parameters and then runs (stochastic) gradient descent to (approximately) solve a convex surrogate problem. Under stated conditions, the resulting predictors are shown to be samples from the predictive posterior of a Gaussian process with the empirical neural-tangent-kernel (NTK). Experiments on toy regression, nine UCI regression sets, and image-classification benchmarks (FashionMNIST, CIFAR10/100) show that NUQLS matches or outperforms Deep Ensembles, Laplace/LLA variants, SWAG, and MC-Dropout while being up to 10 faster in wall-clock time on the larger tasks than Deep Ensembles. The paper also introduces a variance-of-mean-softmax-probabilities (VMSP) metric for classification uncertainty.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n**Computational Efficiency**: The method demonstrates significant computational speedups over deep ensembles (often order-of-magnitude improvements) while maintaining competitive performance on the tested datasets.\n\n**Clear Algorithmic Contribution**: Algorithm 1 is simple, interpretable, and appears to be readily implementable with minimal modifications to existing training pipelines.\n\n**Empirical Performance**: Within the tested domains (toy problems, UCI regression datasets, and small-scale image classification), the method shows competitive performance.\n\n### **Weaknesses**\n\n**Theoretical Foundation**: The core mathematical construction in equation (4) is presented as general but only holds under specific loss function assumptions (strong/strict convexity). For cross-entropy loss used in classification experiments, equation (4) may not even represent valid solutions to optimization problem (3). This makes the theoretical analysis somewhat irrelevant to the larger-scale experimental validation.\n\n**Theory-Practice Disconnect**: The meaningful deep learning experiments (classification on MNIST/CIFAR) operate entirely outside the theoretical framework. The remaining empirical evaluation consists only of nine UCI regression tasks and toy experiments, severely limiting the practical relevance of the work.\n\n**Evaluation Methodology**: The paper introduces VMSP (Variance of Maximum Softmax Probability) as a UQ metric without explicit definition or proper theoretical justification (even taking into account appendix C). \n\n**Limited Theoretical Novelty**: The core ideas of network linearization, NTK connections, and sample-then-optimize frameworks have been extensively explored. Particularly, Immer et al. (2021) already studied linearization, refinement through additional training, and NTK-GP connections. The contribution appears incremental rather than fundamental.\n\n**Inadequate Experimental Scope**:\n\n* Experiments limited to MNIST/CIFAR-scale datasets and outdated architectures rather than modern benchmarks and model architectures\n* Missing evaluation on uncertainty-specific benchmarks (e.g., diabetic retinopathy detection tasks by Band (2022) or others)  \n* No comparison with important recent UQ methods, e.g. Immer et al. (2021).  \n* Related work discussion relegated entirely to appendix\n\n### **Technical Concerns**\n\n**Minor Missing Details**:\n\n* Pseudo-inverse specification (left vs. right) not clarified  \n* Insufficient explanation of why objective perturbation (standard in sample-then-optimize?) is unnecessary  \n* Hyperparameter γ tuning may lead to validation overfitting without principled prior specification\n\n**Methodological Issues**: The approach forgoes having a principled prior and predictive variance formulation, potentially making it more prone to overfitting on validation data compared to proper Bayesian methods.\n\n**Scalability Limitations**: While computationally faster than deep ensembles, scalability to larger & more modern architectures remains undemonstrated, and the linearization approximation quality at scale is unclear.\n\n---\n\nImmer, Alexander, Maciej Korzepa, and Matthias Bauer. \"Improving predictions of Bayesian neural nets via local linearization.\" International conference on artificial intelligence and statistics. PMLR, 2021\\.\n\nBand, Neil, et al. \"Benchmarking bayesian deep learning on diabetic retinopathy detection tasks.\" arXiv preprint arXiv:2211.12717 (2022).\n\nQUESTIONS:\n1. Can you demonstrate scalability to larger, more modern architectures? And additional datasets with additional epistemic uncertainty metrics?\n2. How does the method perform when the linearization approximation quality degrades?\n3. Can you compare to the most important related work in the main paper?\n\nLIMITATIONS:\nThe authors adequately discuss computational limitations and the restricted theoretical scope. However, they could better address the theory-practice disconnect and provide more analysis of when the linearization approximation might fail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}