Title: Online Optimization for Offline Safe Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sequential Decision Making

Offline Safe RL

Minimax Optimization

No-Regret Algorithms
Abstract: We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose O3SRL, a training framework for offline safe RL. O3SRL builds on strong duality of the Lagrangian to solve the constraint optimization problem, by iteratively solving a cost penalized MDP and then updating the Lagrange multiplier with a no-regret “online” learning algorithm. The authors highlight an inherent limitation of offline policy evaluation that is required in their theoretical framework (used to update the Lagrange multiplier by the no-regret algorithm). To address this limitation practically, they discretize the Lagrange multiplier and place a multi-armed bandit to pick it, without directly evaluating policies offline.\nFinally, the authors demonstrate the performance of O3SRL on several offline safe RL tasks, and show that their method outperforms existing baselines, especially when the budget for the constraint is tight.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The empirical evaluation is comprehensive and adequate. The authors show that their method outperforms many existing baselines.\n* The use of EXP3 to circumvent offline policy evaluation is novel and interesting.\n\n## Weaknesses\n* The proposed framework and theory has been studied before in [1]. I couldn't find any reference for that.\n* While providing convergence guarantees (Theorem 1 & 2) adds value to the paper, I believe that showing the conditions for constraint satisfaction upon deployment, after training with offline data, would give a more interesting theoretical insight to readers, and would strengthen a lot the paper. This is more so given that the main theoretical argument that is used in the paper has already been shown by [1].\n* Some theoretical claims are not precise or not rigorous enough:\n  * L.128: the objective and constraints are generally linear only when there is no (non-linear) function approximation involved; this is usually not the case in many practical problems and even in your own experiments. The implication here is that, formally speaking, strong duality does not necessarily hold in many of these problems.\n  * L. 129: Slater's condition is typically assumed, i.e., Equation 1 not always satisfies it.\n  * Remark 1, L. 162-164, “notions of coverage depend very mildly on the reward functions, if at all”. I don't think that this statement is true. What about sparse reward tasks? Consider for example the sparse reward tasks in [2].\n* While the use of the bandit algorithm is novel, and seem to yield strong results, the practicality of it is somewhat limited.\n\n## Minor\n* Using only three seeds for evaluation should be improved.\n\n## Presentation\n* I believe that the empirical result is more significant in this paper and should receive more focus.\n* The significance of low budget constraints should be further motivated and highlighted.\n* Consider presenting your empirical results in a plot instead of a table---it is much easier the parse and could convey your empirical result better.\n\n[1] Le, Hoang, Cameron Voloshin, and Yisong Yue. \"Batch policy learning under constraints.\" In International Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.\n\n[2] Nair, A., Gupta, A., Dalal, M., and Levine, S. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv, June 2020.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nI believe that the main limitation of the proposed work is the practicality of the proposed algorithm:\n* Using a bandit to pick the Lagrange multiplier will require practitioners to implement an additional (non-trivial) subroutine.\n* The algorithm returns a “mixing” policy that combines several policies together and draw them randomly, as common in previous works that propose primal-dual algorithms to solve CMDPs. This is usually a limitation in practice, because such mixing policies are not easy to implement, and might execute severely unsafe actions.\n\nWhile the use of a bandit limits the practicality of O3SRL, I believe that this is still a useful contribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to solve the problem of offline safe RL by combining offline RL and online optimization. The incorporation of the cost into the reward enables the use of offline RL, while discretizing the value of the Lagrange multiplier enables the use of multi-armed bandit algorithms, eliminating the need for estimating off-policy value functions. Theoretical results are provided to show the optimality of the proposed framework, and modifications are made upon the framework to make it practically usable. Empirical evaluation on a set of benchmark tasks showcases the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The flow of this paper is clear and easy to follow.\n- The incorporation of the cost into the reward enables the use of well-developed offline RL algorithms, which is novel in offline safe RL as far as I know.\n- The discretization of the Lagrange multiplier is novel and empirically effective. Combined with the EXP3 algorithm, this eliminates the need for estimating the off-policy values which could be erroneous.\n- The empirical results are pretty promising, especially in terms of the safety metric, which is desirable for offline safe RL algorithms.\n\n\n\n### Weaknesses\n\n- The number and variety of the tasks evaluated on can still be improved. A more comprehensive evaluation like the one in the FISOR paper \\[1\\] would make the comparative results much more convincing.\n- Missing related work (and probably a baseline): TREBI \\[2\\], which serves as a baseline in the FISOR paper.\n\n#### Minors\n\n- Line 86: missing in-text reference.\n\n\n**References**\n\n\n\\[1\\]: Zheng et al., 2024, \"Safe offline reinforcement learning with feasibility-guided diffusion model\".\n\n\\[2\\]: Lin et al., 2023, \"Safe offline reinforcement learning with real-time budget constraints\".\n\nQUESTIONS:\n1. Line 286 concludes that higher values of K leads to diminishing returns, but from Table 1 the trend is the opposite.\n2. Why does discretizing the Lagrange multiplier help stabilize learning?\n3. Could you provide a more intuitive explanation of how the EXP3 algorithm updates the distribution over arms? Although this is not the contribution of this work, I think an intuition of how it works to avoid OPE can be insightful to readers. Maybe putting it in the Appendix would be a good choice.\n4. Table 1: Why does increasing the granularity reduce the safety of the agent? How can interpret this empirical result? I couldn't see an intuitive causal relationship between this hyperparameter and the safety performance.\n5. Table 3: \n   - Does the results mean that the agent becomes closer to the safety budget when the budget is higher? Shouldn't the normalized cost stay relatively the same regardless of the budget? \n   - Will it finally reach or even violate the constraint when the budget is high enough? \n   - Line 331 says approaching the budget can \"make better use of the available budget\", does this mean that the results of O3SRL in Table 2 can still be improved since the normalized cost is mostly close to 0?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline safe reinforcement learning (RL) with a constrained MDP notion of safety. The goal is to maximize the reward while keeping the associated cost of transitions under a certain safety budget. An algorithm is proposed that solves the Lagrangian objective using two approaches:\n\n- an offline RL oracle that optimizes over the given data using adjusted returns that include both reward and cost with current Lagrange multiplier values\n- a no-regret algorithm for optimizing the Lagrangian multiplier that is implemented interestingly as a discretization of the multiplier value solved using a bandit approach\n\nThe result shows that on safe RL benchmark, the proposed approach does well always leading to safe policies that get high returns. The algorithm is also robust to choice of offline RL oracle\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Safe RL is an important problem. \n- The proposed approach does make sense\n- Experiments are good\n\n**Weakness**\n\n- Parts of the paper, especially the theoretical part, do not make full sense to me. Firstly, there is no mention of data distribution coverage or use of pessimism, the latter of which is common in offline RL literature. The offline RL oracle assumption seems particularly strong. It is essentially assuming that given the data you have, you can learn an optimal policy with high probability. This is not true in general unless you assume concentrability or other notions of good data coverage. Alternatively, most offline RL approaches assume a data distribution that only covers the optimal policy and use pessimism. \n\n- Writing is unclear. The paper talks about how to approximate value function of the optimal policy without telling why is this needed. The oracle for Lagrange multiplier in Definition 3 does not seem to take that as input although later it is used in Line 4, Algorithm 2.\n\n- It is unclear how another approach will do that, simply runs the offline RL algorithm, treating lambda as a hyperparameter and tuning it on some dev set. It seems the core contribution is just to find the right value of this scalar.\n\nQUESTIONS:\n- Is data distribution assumed to have good coverage? What assumptions will the offline RL oracle need to guarantee the requirements in Definition 2.\n\n- Why is the value function need to be estimated? There is no mention of this in Definition 3. Also, it is stated in line 200 that \"approximation of the value function of the optimal policy\" but then Equation 6 shows it is estimation of the policy $\\pi$. Shouldn't you have $E[\\tilde{V}] \\approx V^\\star_f$.\n\n- What is a \"stochastic approximation\"?\n\n- It seems to me that the entire contribution of the paper is to set the value of the Lagrange multiplier and then call the offline RL algorithm. If yes, then how will a much simpler approach do which simply calls an offline RL algorithm with the Lagrange multiplier treated as a single scalar hyperparameter, and then setting this hyperparameter by doing runs on a held-out set of online runs. It is possible that iterative learning with averaging is needed, but this baseline should have been evaluated.\n\n- Related work: Provable Safe Reinforcement Learning with Binary Feedback, Bennett et al. AISTATS 2023.\n\n\nMinor:\n\n- Broken link on line 186\n- Citations should be in parentheses at several places such as Line 209\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}