Title: Deep Actor-Critics with Tight Risk Certificates
Venue: NeurIPS 2025
Accepted: no
Keywords: Reinforcement learning

PAC Bayes

Risk Certificates
Abstract: After an period of research, deep actor-critic algorithms have reached a level where they influence our everyday lives. They serve as the driving force behind the continual improvement of large language models through user-collected feedback. However, their deployment in physical systems is not yet widely adopted, mainly because no validation scheme that quantifies their risk of malfunction.  We demonstrate that it is possible to develop tight risk certificates for deep actor-critic algorithms that predict generalization performance from validation-time observations.  Our key insight centers on the effectiveness of minimal evaluation data. Surprisingly, a small feasible of evaluation roll-outs collected from a pretrained policy suffices to produce accurate risk certificates when combined with a simple adaptation of PAC-Bayes theory. Specifically, we adopt a recently introduced recursive PAC-Bayes approach, which splits validation data into portions and recursively builds PAC-Bayes bounds on the excess loss of each portion's predictor, using the predictor from the previous portion as a data-informed prior.  Our empirical results across multiple locomotion tasks and policy expertise levels demonstrate risk certificates that are tight enough to be considered for practical use.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a critical problem for deploying deep actor-critic algorithms in physical systems: the absence of a validation scheme capable of quantifying their risk beforehand and this limits  their widespread adoption in physical world where hazards are real. The proposed solution involves developing tight risk certificates (in metric terms - a bound) for these algorithms (experiments are based on REDQ algorithm), which can accurately predict generalization performance based on validation-time observations. A key insight is the effectiveness of using minimal evaluation data, collected from a pretrained policy combined with a recursive PAC-Bayes approach. This method starts by fitting a\nPAC-Bayes bound using observations from the validation set and then they predict the discounted return of the policy on the test set. Empirical results across multiple locomotion tasks demonstrate that these risk certificates are tight enough for practical use as they show correlation with test error. They also show that  tightness of bound improves as recursion depth increases. Crucially, the recursive bounds also significantly improve sample efficiency, making them particularly valuable for costly data collection in physical robot systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n\n1. Paper is well-written and easy to understand. \n2. Addresses challenging problem of evaluating algorithms in RL in real world.\n3. Seems fairly novel. Not aware of applying recursive PAC-Bayes in Deep actor critic algorithms, although have seen application of PAC-bayes in RL for model selection. \n4. Good experimentation strategy. Specifically - a) Correlation between bound and test error for 3 levels of expertise. This can also serve as tool for convergence check, where noisy correlation between bound and true test error signifying lack of convergence. b) Results for different sizes of validation datasets.\n\nWeaknesses : \n1. Evaluated only off-policy type. Although authors do mention reason for choosing only one algorithm to delve deeper and perform more experiments rather shallow experiments on multiple ones but that said picking one from on-policy would have been great for e.g PPO which is right now go-to for LLMs. \n2. [Open question for my understanding] Argument that requirement of evaluation dataset is minimal is not clear. It is written that model is trained for 3e5 steps, eval dataset is of 100 episodes. I would like to know how this dataset compares to training dataset in size in similar scale i.e, steps or maybe transition tuples. \n3. Some good work in this track is missing in literature survey - \nPAC-Bayesian Model Selection for Reinforcement Learning (Fard & Pineau, 2010)\nPAC-Bayesian Policy Evaluation for Reinforcement Learning (Fard, Pineau & Szepesvári, 2012)\n\nQUESTIONS:\nI would like to know this for argument that requirement of  evaluation dataset is minimal. Can you tell validation dataset size as % of training dataset?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a deployment-time certification method for deep, continuous-action actor-critics. After training a policy, ≈100 fresh roll-outs are recursively split to build increasingly informative PAC-Bayes priors, giving a closed-form upper bound on the trajectory-level gap from a target return. Bayes-by-Backprop with the local-reparameterisation trick supplies a low-variance posterior, yielding certificates that correlate (ρ > 0.9) with true returns on five MuJoCo tasks, esp at intermediate and expert skill levels.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novel evaluation setting: First work to post-facto certify deep actor-critics with so little validation data.\n- Recursive prior refinement is a simple, theoretically sound trick that monotonically tightens bounds.\n- Practical implementation: Adds minimal overhead to a standard REDQ pipeline; code is easy to reuse.\n- Clear presentation: Figures link bound width to generalisation error; proofs are complete.\n\nWeaknesses:\n- Baseline availability is understated: Prior PAC-Bayes certificates exist for both continuous-action (PAC4SAC, PBAC, PAC) and discrete-action RL; at least one could be reported for fairness (discrete action ones could be trivially applied to this case by binning and checking for empirical tightness).\n- Return-only guarantee. Bounding expected return ignores tail-risk, per-step safety constraints and reward-shaping artefacts, which could limit practical assurance.\n- Single algorithm + simulator. No evidence on PPO/SAC or on real world robotics data or other control environments, only state space (no visual control) settings, so generality is unclear.\n\nQUESTIONS:\n- Add at least one prior certificate as a baseline (e.g., PAC-Bayes SAC evaluated post-training or a discretised bound).\n- Report sensitivity to deeper recursion and non-uniform split schedules; include wall-clock cost for posterior fitting + certificate computation.\n- Demonstrate generality on a different algorithm (e.g., SAC) or a small experiment to show the method survives sim-to-real noise.\n- Address safety tails: Either plot empirical return distributions or discuss how a CVaR-style loss could be substituted for the current return-gap term.\n\nLIMITATIONS:\nThe paper is forthright about simulation-only evidence and dependence on reward shaping, but should also note the bound’s blindness to rare catastrophic events and to online policy adaptation as it is done frequently in realistic settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduce a framwork for generating tight risk certificates for Deep AC RL using PAC-Bayes Bounds. auther demonstrates that even minimal evaluation data can accurate generalization guarantees.

STRENGTHS AND WEAKNESSES:
>Strength

1. paper is clear written.
2. auther did evaluate on varity of tasks.


>weakness

1. Figures are hard to see because of the dense information (Figure 2, 3,5 for some subfigures, the dense data make the result really hard to find). Espicially For figure 2, please have a legend on the figure. it especially hard to match the caption with the visual elements. For 3 and 5, in the appendix version, auther may plot it in other way to make it more clear. 

2. The PAC-Bayes introduces a sequence of posteriors and scaling factors which may increase computational complexity. 

3. Auther did study the quality of the learned policy. However the policy range is just from 100k to 300k which may be too limited espicially for complex task such as humanoid. Can auther report the evaluation quality of these learned policy?

QUESTIONS:
1. Please address the limitation 1 with better figure quality

2. From data efficency perspective, in order to achieve same quality of bound tightness, how much sample (in learning optimization process) the purposed method will save and how long clock time?  

3. Will deeper T keep improve performance? what are bottleneck?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for generating risk certificates for deep actor-critic architectures, aiming to provide high-confidence guarantees on generalization performance.\n\nThe method builds on the PAC-Bayes theoretical framework and introduces a recursive PAC-Bayes inference procedure. Traditional PAC-Bayes methods are difficult to apply directly to reinforcement learning due to their assumption of independent and identically distributed (i.i.d.) data, while reinforcement learning operates under a policy-driven Markov decision process. To address this challenge, the authors design a staged modeling and analysis process. The approach divides a small set of evaluation data into multiple subsets, sequentially trains a series of Bayesian neural networks, and recursively constructs upper bounds on generalization error at each stage. This strategy not only reduces the reliance on large validation datasets but also leverages prior training outcomes to improve the accuracy of risk estimation.\n\nIn the experimental section, the paper evaluates the method across five classic MuJoCo reinforcement learning environments—Ant, Hopper, Half-Cheetah, Humanoid, and Walker2d—covering different levels of policy quality (starter, intermediate, expert). The results show strong performance in multiple aspects: high confidence, tightness, sample efficiency and robustness.\n\nIn summary, the proposed method is theoretically sound and practically effective. It demonstrates strong applicability and scalability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper presents a carefully designed and well-executed experimental setup. It evaluates the proposed method across five diverse MuJoCo environments (Ant, Hopper, Half-Cheetah, Humanoid, and Walker2d), which are standard yet sufficiently complex benchmarks in continuous control. The experiments consider multiple policy quality levels (starter, intermediate, expert), account for randomness across multiple seeds and repetitions, and use appropriate statistical measures such as normalized error, correlation, and bound tightness. Additionally, the ablation studies assess the impact of recursion depth, validation data size, and local reparameterization tricks.  These experiments demonstrate not only the effectiveness but also the sample efficiency and robustness of the approach.\n2. The paper provides strong theoretical support. The method is built on a solid foundation of PAC-Bayes theory, particularly the recent advances in recursive PAC-Bayes bounds. The authors clearly describe key theoretical concepts such as KL divergence, excess loss decomposition, and recursive bound construction.  \n\nWeaknesses\n\n1. The Background section spans several pages and includes a deep dive into standard actor-critic reinforcement learning, PAC-Bayesian theory, and multiple versions of PAC-Bayes bounds. While these details are important, the extensive coverage overshadows the novel contributions of the paper. As a result, the structure feels unbalanced—the Background and Preliminaries occupy a disproportionate amount of space compared to the Methods and Results sections. \n2. Although the application of recursive PAC-Bayesian bounds to reinforcement learning is novel in execution, the core theoretical techniques are largely drawn from prior work. The recursive PAC-Bayes method is not developed in this paper, and many of the theoretical tools (e.g., KL bounds, data-informed priors) have been well established in prior literature. This gives the impression that the paper mainly applies existing ideas to a new domain rather than proposing fundamentally new theory. As such, while the work is well-executed and useful, its originality may be viewed as incremental rather than groundbreaking.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}