Title: Certifying Deep Network Risks and Individual Predictions with PAC-Bayes Loss via Localized Priors
Venue: NeurIPS 2025
Accepted: yes
Keywords: PAC-Bayes generalization

Data-dependent bounds

Bayesian neural networks

Adversarial robustness

Model calibration

Trustworthy AI
Abstract: As machine learning increasingly relies on large, opaque foundation models powering generative and agentic AI, deploying these systems in safety-critical settings demands rigorous guarantees on their generalization beyond training data. PAC-Bayes theory offers principled certificates linking training performance to generalization risk, yet existing approaches are rarely practical: simple theoretical priors yield vacuous bounds, while data-dependent priors trained separately are computationally costly or introduce bias. To bridge this fundamental gap, we propose a localized PAC-Bayes prior—a structured, computationally efficient prior softly concentrated near parameters favored during standard training, enabling effective exploration without costly data splits. By integrating this localized prior directly into standard training loss, we produce practically tight generalization certificates without workflow disruption. Theoretically, under standard neural tangent kernel assumptions, our bound shrinks as networks widen and datasets grow, becoming negligible in practical regimes. Empirically, we certify generalization across image classification, NLP fine-tuning, and semantic segmentation, typically within three percentage points of test errors at ImageNet scale, while providing rigorous guarantees for individual predictions, selective rejection, and robustness.


Peer reviews:
comments: SUMMARY:
The paper discusses the problem of certifying the links training performance to generalization risk. To solve this problem, the authors propose a localized PAC-Bayes prior, derive a non-vacuous bound for it, and show that integrating this prior into the training process yields practically tight generalization certificates. They validate their approach across diverse tasks.

STRENGTHS AND WEAKNESSES:
Strengths

1. The paper is well written. Its motivation and results are presented clearly. I enjoyed reading it, even though it lies outside my area of expertise.

2. The localized PAC-Bayes bound provided by the authors improves upon earlier work. Their claims are empirically verified across multiple datasets and architectures.

3. The proposed approach is interesting and improves existing solutions to the problem.

Weaknesses

The paper would benefit from including a proof sketch in the main text that outlines the key steps of the proofs of Theorems 3.1 and 3.2.

QUESTIONS:
1. Could the authors elaborate on the technical challenges they faced while proving the theoretical results?

2. Could the authors explain in more detail how their theoretical bounds compare with previous results?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tightens the PAC-Bayesian bound by introducing a localized prior into the computation. The proposed bound, along with three baseline PAC-Bayesian bounds, is evaluated on several benchmark datasets. The results show that the improved bound achieves better tightness compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n+ The tightness of the bound is evaluated across diverse benchmarks, including image classification, semantic segmentation, and NLP fine-tuning tasks.\n+ The paper is clearly written and easy to follow.\n+ The extension of the population risk bound to per-prediction guarantees offers intriguing potential applications in adversarial and out-of-distribution (OOD) input detection.\n\n**Weaknesses:**\n- Although the computational overhead is moderate, computing the localized bound still requires an additional forward pass during neural network training.\n- The tightness of the bound is data-dependent, which may limit its effectiveness when training data are not carefully curated.\n- The paper lacks a discussion on how to appropriately choose the $\\epsilon$ value to obtain meaningful per-prediction guarantees for adversarial or OOD detection use cases.\n\nQUESTIONS:\n1. Since the proposed bound is data-dependent, would it be possible to provide an error bar or confidence interval on the computed generalization error bound? This could offer additional insight into its reliability and variability across different datasets.\n2. The improvement in bound tightness appears to be more pronounced in image classification tasks compared to semantic segmentation and NLP fine-tuning. Could the authors elaborate on the reasons behind this discrepancy and the potential takeaways for practitioners?\n3. The application of per-prediction bounds for adversarial or out-of-distribution (OOD) input detection is intriguing. However, in Figure 2, there is considerable overlap between the adversarial and benign risk distributions in the range [0.625, 0.875]. How should the $\\epsilon$ threshold be selected to effectively distinguish between adversarial and benign inputs, while minimizing the false negative rate for a given false positive rate?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for computing tight PAC-Bayes generalization bounds for deep neural networks, aiming at addressing the issue of vacuous results  for over-parameterized models (e.g., deep neural networks) when we were trying to compute traditional PAC-Bayesian generalization bound built upon measuring the KL divergence between a learned posterior and a fixed prior.\n\nThe authors’ contribution is a data-dependent reference distribution called “localized prior”, which is constructed by softly tilting a base prior towards low-loss regions of the parameter space. This localization is controlled by a parameter $\\xi$. By integrating the optimization of the model's posterior and the localization hyper-parameters into a single training pass, the method avoids the second training stage required by many other data-dependent PAC-Bayes approaches, which is typically costly. The authors also provide a theoretical bound formalizing this approach and show empirically that it can yield non-vacuous generalization certificates on a range of tasks, including ImageNet classification, Cityscapes segmentation, and NLP fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper targets a critical problem in learning theory: how to provide rigorous generalization guarantees for large-scale neural networks, which is also a key requirement for their practical deployment in safety-critical domains beyond theoretical interests.\n2. The \"one-pass\" approach to learning a localized prior is a practical contribution that sidesteps the computational burdens of existing two-pass methods, making PAC-Bayes certification potentially much more accessible.\n3. The method is evaluated on a set of modern benchmarks with state-of-the-art reported tightness of the bounds.\n\n---\n\n### Weaknesses\n\nI am not working on ML theory, so the issues/questions identified below are primarily based on educational knowledge and experiences - please use them sparingly.\n\n1. The training objectives rely on a Monte Carlo estimator for log-partition functions with a very small sample size. As far as I know, in the high-dimensional setting of deep learning, such kind of estimator can be systematically biased, leading to an underestimation of the model’s complexity. I am concerned with whether this would lead to artifacts if estimation errors occur.\n2. The theoretical justification of the bound relies on a union bound over a discrete grid of hyperparameters. But the proposed algorithm appears to optimize these parameters via SGD continuously. I think there might be some concerns - are we allowed to optimize them during training while claiming PAC-Bayes guarantees? I think standard PAC-Bayes requires priors fixed before observing data, but these parameters are optimized on the training data?\n3. I think it would also be important to evaluate whether the number of Monte Carlo samples impacts the theoretical and empirical results, which were omitted currently.\n\nQUESTIONS:\nq1. Is it possible to provide an ablation study on high-dimensional tasks showing the final bound changes for different values of K, and further justifications where possible?\n\nq2. Regarding the optimization of algorithm 1. Is it valid to apply the union bound by optimizing $\\lambda$ and $\\xi$ continuously? Also, is it valid to optimize them “data-dependently” while claiming PAC-Bayes guarantees?\n\nLIMITATIONS:\nThe authors have noted the potential for underestimation from finite-sample importance sampling due to Jensen's inequality. However, I think this discussion should be extended - it frames the issue as a trade-off, which I believe is essential to analyze the severity of this bias.  The authors should have stated that their reported certificates are conditioned on the (potentially) questionable assumption that this biased, low-sample estimator is accurate.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to provide a PAC-Bayes bound on the generalization error (i.e the true risk) of a classifier. Pure PAC approaches yield vacuous bounds due to the expressiveness of neural networks. The approach of a traditional PAC-Bayes bounds compares a learned distribution of parameters to a prior parameter. This too can yield vacuous bounds due to the large movement in parameters needed to learn useful models. One could circumvent this issue with a data-splitting approach but this adds complications. Of course if we’re willing to split the data, we’ll seldom do better than just estimating the risk on the holdout data. There’s a reason why this is what we all do and will continue to do. \n\nThe authors propose a method that uses localized priors based on an idea from Catoni. The central problem here is how to use a data-dependent prior without sample splitting. The authors use an idea from Catoni that softens the data-dependent distribution with a parameter that is then accounted for in the bound. Assuming the results are correct (and sufficiently novel) the result is a new method for computing non-vacuous PAC-Bayes bounds. \n\nThe idea here involves an interpretation of training trajectories where checkpoints are taken as samples from a posterior distribution over the weights. A PAC-Bayes bound is calculated that accounts for the divergence between a  The finite set of checkpointed weights are taken as an approximation of the posterior and this approximation is taken into account in the bound. Assuming the analysis is correct, the authors offer a PAC-Bayes bound on the risk that accounts for some but not all approximations. Importantly the finiteness of the checkpoints but lack of independence of the checkpoints isn’t accounted for. \n\nThe authors present empirical results that show that the PAC-Bayes bounds that they produce on several academic datasets of interest are indeed nonvacuous and tighter than previous attempts although these results involve early stopping / the impacts from the modified objective are such that the results are far from state of the art. The test set is going no where fast :).  The experiments also include some results on usefulness for flagging adversarial / OOD examples but these are cases where the formal guarantees do not hold so the interest is purely empirical. \n\n\nI would like to see more empirical confirmation of the coherence of the results. Crucially, I would like to see what happens not just up to the point of early stopping but what happens when one continues training down to near-0 training error. If the method is valid, the PAC-Bayes bound should dip but then rise again and explode as the learned model overfits the data. The results in Figure 1 show only the descent but not the rise and given the complexity of the setup and presence of some violated assumptions doesn’t inspire confidence in me that the bounds. I’d like to see more empirical evidence that the bounds behave as expected when one trains into the interpolation regime.\n\nSTRENGTHS AND WEAKNESSES:\nI am not fully in position to verify the correctness of the paper but it offers an interesting take on an interesting problem, introducing a refined technique for calculating and optimizing PAC-Bayes bounds that can be applied on neural networks.\n\nWeaknesses: \n\nThe authors are a bit too coy in the introduction in suggesting that test set estimation of the generalization is somehow insufficient for providing assurance that the test error is indeed low. It’s far the best way to do this and in fact the only viable method. A test-set-free guarantee on the generalization error is primarily useful for the fundamental insights it sheds on generalization. No practitioner would take the massive hit in accuracy (here incurred due to the aggressive early stopping?) to get the elegance of a one-pass estimate of the risk using training data only. \n\nThe paper is harder to follow than it needs to be. The authors should more clearly lay out the shape of the strategy. What is the key insight from Catoni that doesn’t involve data splitting? How is validity ensured? What are the key pieces of the argument? The reader is left to go fishing and this makes the paper unapproachable for an otherwise competent reader somewhat unfamiliar with the terrain.  Moreover, it’s not sufficiently clear in the paper what is the key contribution over what’s already present in Catoni’s work. Is it the use of the samples to approximate the posterior and the ensuing analysis? The body of the paper lacks a proper prior work section but would benefit from one.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}