Online Optimization for Offline Safe Reinforcement Learning
Authors: Yassine Chemingui
Aryan Deshwal
Alan Fern
Thanh Nguyen-Tang
Jana Doppa
Venue: NeurIPS 2025 poster ()
Keywords: Sequential Decision Making
Offline Safe RL
Minimax Optimization
No-Regret Algorithms
Primary area: reinforcement_learning
TL;DR: Creating safe and reward maximization policies from offline data via min-max optimization formulation and solving it using no-regret algorithms

Abstract: We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.

Online Optimization for Offline Safe Reinforcement Learning
Yassine Chemingui Washington State University yassine. chemingui @wsu. edu
Aryan Deshwal University of Minnesota adeshwal@umn. edu
Alan Fern Oregon State University
alan ......ern@orr.....e..eeeeaeee.e. www.@@rrgonsstaeeee....o edu
Thanh Nguyen-Tang
Janardhan Rao Doppa
New Jersey Institute of Technology thanh, nguyen@njit.enn
Washington State University
jana. doppa@wsu. edu
Abstract
We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn reward-mant maximiinn policy from fixed data under cumulative cost constraint. We propose novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available https:::/// //gi thub. com/ www.yyysssinehehhee
1 Introduction
Offline reinforcement learning (RL) [Levine et al., 2020] a powerful paradigm learn decision- making policies from logged datasets without the need for additional interaction with the environment. Offline RL has shown good success domains including autonomous driving, robotics, control systems, and black-box optimization [Fang et al. 2022 Lin et al. 2024 Li et al. 2024 Zhan et al. 2022] Cheminguu al.] 20244) where executing exploratory actions not practical. However, in safety-critical domains such healthcare and smart grid, the decision-maie maaiin agent needs to also satisfy some cost/safety constraints. Such problems are studied under the sub-area referred to as offline safe RL (OSRL) where the goal is to learn maxiaaiiiiiiiiing policies which satisfy cost constraints from offline datasets [Wachi et al. 2024
OSRL inherits the challenges from both offline RL and safe RL [Wachi et al. 2024 First, handling distributional shift when the learned policy encounters states and actions not seen in the offline dataset. Prior work in offline RL addresses this challenge by some form of penalization to realize the general principle of pessimism in the face of uncertainty [Levine et al. 2020). Second, ensuring that the learned policy meets the cost/safety constraints after deployment. Maintaining the cost constraints typically requires off-policy evaluation (OPE) procedures which are highly unstable and typically have estimation errors (Figuciredo Prudencio et al. 2024]. OSRL is an active research topic and is relatively less-studied than offline RL, Constrained policy optimization using Lagrangian relaxation
(Polosky et al. 2022 Lee al.) 2022 Xu al. 2022b is popular approach for OSRL, but due to the need to solve intertwined optimization problems they are unstable in practice, and are shown to have bad performance either causing oscillation/ddvergeen zero-reward) policies. Additionally, OSRL problem with stringent safety constraints (i.e., small cost
learning overly conservative (near
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

thresholds) is an important but severely under-studied problem [Zheng et al. 2024 and arises in many sufety-critical control applications such as industrial process and energy system control.
This paper develops a novel offline constrained RI. framework referred to as Online Optimization for minimax optimization problem for policy learning and solves it using an iterative approach that relies on two key components, namely, an offline RL oracle and a no-regret algorithm. We perform two main algorithmic steps in each iteration of O3SRL. First, we produce a distribution over policies from solving an offline RL problem by defining a new reward function that combines the original reward and cost values based on the values of the current Lagrange variable and the cost threshold. Second, we employ a no-regret algorithm (Hazan et al., 2016] to adaptively update the value of Lagrange variable based on the current distribution over policies. We prove that the O3SRL framework is guaranteed to converge to the minimax solution.
Offline Safe RL (O3SRL)).
Unfortunately, the general O3SRL. framework suffers from two drawbacks in practice. First, the of no-regret algorithms over a continuous Lagrange variable requires calling OPE procedures which are typically unstable [Figueiredo Prudencio et 2024] and can potentially lead error propagation over iterations. Calling an OPE procedure in each iteration also contributes to computational expense. Second, is compurationally--xpessy to run an offline RL algorithm to convergence in each iteration. To overcome these challenges, we develop a practical and effective algorithm with convergence guarantees. We employ К discrete values for Lagrange variable and apply multi- algorithm. Since most offline RL algorithms are based on stochastic gradient descent, we perform small number of gradient updates on the policy from the previous iteration for efficiency. One advantage of this approach is that we can leverage powerful offline RI. algorithms in plug-and-play manner to solve offline constrained RL problems.
bandit
We perform experimental evaluation of this practical approach on multiple DSRL benchmark tasks (Liu al.] 2024] and our main findings are as follows. First, even the simplest version of our approach with two arms (K=2) is quite effective and results in state-of-the-art performance. Second, our approach achieves excellent performance for the challenging setting of small cost constraint thresholds. Third, the performance improves with higher arms (K > 2) but shows diminishing returns beyond K=5, Finally, O3SRL performs effectively with different offline RL algorithms and higher cost limits demonstrating its robust performance.
Contributions. The key contribution of this paper is the development and evaluation of the O3SRL framework to solve offline safe RL. problems. Specific contributions include:
Formulating OSRL as a minimax optimization problem and providing an iterative framework based on no-regret algorithms with convergence guarantees the minimax solution.
Development of a practical algorithm with convergence guarantee by avoiding the usage of off- off-policy evaluation (can lead to error propagation) and running offline RL algorithm to convergence (high computational cost) inside each iteration of a multi-round method.
Empirical evaluation of the practical algorithm and its variants on DSRL benchmark tasks to demonstrate its effectiveness over state-of-the-art methods.
2 Problem Setup
RL problems with safety constraints are typically formulated using the Constrained Markov Decision Process (CMDP) framework [Altman| 2021). A CMDP is defined by the tuple (S, A, P,r, r, c, Y, p), where S and A denote the state and action spaces respectively, P S A - [0, 1] is the unknown stochastic state transition function, r S x A + R is the reward function, c S x A + Rgo is the cost function, 7 E (0, 1) is the discount factor, and μ is the initial state distribution.
Let π. : S - A denote a policy and let T = ((*,, ar Ft; C be trajectory of length T induced by under dynamics P. The cumulative discounted return and cost for a trajectory are given by R(() DLn 7'r, and C(7) = Σt= 7'c,, respectively. In offline safe reinforcement learning (OSRL) problems, the agent is given access only to a static dataset DOSRI. {(8;, 04 Tt, G, / of n examples collected from an unknown behavior policy, and the overall goal is to learn policy π to maximize the expected reward while satisfying a given constraint on the expected cumulative cost. Formally, the learning objective is as follows:
max E [R(T)] subject to (r S k,

---

where &. 2 is the maximum cost threshold/limit for safety constraint. This problem setting combines the challenges of offline RL (e.g., out-of-distribution states) with the added requirement of ensuring cost constraint satisfaction from static data alone, without online interaction.
Most of the prior work in OSRL cannot handle the setting where the cost threshold k is small (Zheng et al. 2024] that is important for safety-critical applications with a tight budget. One of the key goals of this paper is to handle this challenging setting in a principled manner while solving the general OSRL problem effectively for higher cost constraint thresholds.
3 Related Work
Offline RL. This setting differs from the standard online RL setup in the following way: the goal is to learn reward maximizing policies from a given static dataset without interacting with the environment. The key technical challenge is to effectively handle out- -i----i-dtiibution (OOD) states and actions
2024). Prior work has addressed this OOD challenge [Levine et al. 2020 Figueiredo Prudencio al.
using improvements in value function estimation [Fujimoto and Gu [202T] Kostrikov et al. 2022
Kumar et al. 2019 Lyu et al. 2022 Yang et al. 2022, Nguyen- lang and Arora| 2023b a sequential
modeling (Janner et al. [2021] Wang et al. 2022].
sicertainty-wwareeee
[An et al. [021]] Bai et al.
2022], constrainnng policy based on divergence [Wu et al. 2020 Jaques et al. 2020 Wu et al. 2022]
by extending methods for model based RL (Kidambi et al. 2002 Yu et al. 2020 Rigter et al. 2022
and imitation learning [Xu et al. 2022a]. and action selection based on filtering and reweighting
[Chen et al. 2023 Hatsen-Estruch et al. 2023).
Offline safe RL. This is generalization of the offline RL setting where we need to learn safe policy from a given offline dataset and a pre-specified cost constraint [Liu et al. [024]. Offline safe RL is a relatively less-studied problem. Prior work includes methods based on constrained
policy optimization (Polosky et al. 2022 Lee et al., 2022 and Lagrangian relaxation ([ee et al.
2019 Xu et al., 20226]. [Le et al. 2019 proposed one of the first batch-constrained frameworks that combines offline RL with online iegrange-multipee
updates. However, this approach faces practical challenges (error propagation and high computational effort) due to the need to call an OPE procedure multiple times in each iteration which we address in this paper. We elaborate on these challenges and differences the Appendix.
Extensions with convex MDP assumptions have also been explored Zhhagg et al. 2024 along with approaches based on adversarial cost penalties [Wei et al. [024 decision transformer [Liu ct al. 2023). and diffusion models to create safe policies [Lin ct al. 2023 Zheng et al. 2024 Yao
jet al. 2024). J. CAPS [Chemingui et al. 2025) addresses dynamic safety constraints at deployment
time by switching between a set of pre- pre-trained policies based on the test-time cost budget. [Guo
jtt al. 2025] introduced CCAC framework, which conditions the policy and value functions on cost thresholds to enable adaptation and generalization to unseen constraint levels. FISOR (Zheng et al., state-wise cost constraint satisfaction. Rather than optimizing toward given cost limit, FISOR focuses on minimizing cost directly by learning policies that select actions only within the feasible region.
2024) addresses safety by enforcing hard constraints-ensuue
This paper aims to address the following two drawbacks of prior work: 1) Lagrangian based con- strained policy optimization methods solve inter-twined optimization problems and are typically unstable in practice resulting in poor performance; and 2) While FISOR produces safe policies under tight cost constraints, it achieves low reward performance. The proposed O3SRL framework provides a theoretically groundddd and yet practically -effectilyee solution that can be wrapped around existing offline RL methods and does not require an OPE procedure.
4 03SRL Framework for Offline Constrained RL
In this section, we first provide a minimax optimization formulation of the offline constrained RL problem. Next, we describe the general O3SRL framework based on no-regret algorithms to solve this optimization problem along with theoretical guarantees for convergence to the minimax solution. Finally, we provide approximate algorithm for practical purposes.
3

---

4.1 Minimax Optimization Formulation
The discounted value functions of policy ι for reward and cost objectives are defined as follows.
0e V = Ex Σ (rr(s), oz)
and Ve == Ex Σ > 7'((se, ar)
where y is the discount factor, r(st, at)=r and c(st, ae)= are the reward and cost values corre- sponding to state-action pair (st, 0z) respectively.
Let II be the set of policies and ATI be the set of all distributions over II. We aim to solve the following constrained optimization problem:
max E [V,"] subject to EE-D [V S k.
DEAll
Let A(D, be the Lagrangian of Equation ① where À is the Lagrange multiplier.
L(D, == E
[
X [V]] - st) EAED V (T-M--a-)D))
(2)
The dual problem of Equation 1 is given by:
min max L(D, A). X20 DEAIl
(3)
Note that the objective and the constraint in Equation (1) are linear, thus convex and differentiable. Additionally, the constraint in Equation ( satisfies Slater's condition when II is expressive enough that there exists at least one policy E II that is strictly safe (V. < k), thus strong duality holds. This implies that solving the primal problem in Equation (T) is equivalent to solving the dual problem in Equation
3 It is also equivalent to constraining the domain of À to any bounded domain A
[0, C| for some > O: if (D", >*) is solution the dual problem and if X* > 0, then (D", 1) is also a solution to the dual problem. This is because under strong duality, any solution (D", X*) of the
dual must satisfies the KKT conditions: A(E [V]] K) O.
Definition (e-approximate equilibrium), (D, E All x A is e-approximate minimax equilibrium of Equation 3 if - & A(D,A) +,,((D,)) A A.
4.2 General O3SRL Framework via No-Regret Algorithms
Overview of the 03SRL Framework. O3SRL relies on two key elements: 1) an offline RL oracle which takes training dataset of (state, action, next-state, and reward) tuples to produce policy that maximizes reward, and 2) a no-regret algorithm which adaptively selects the value of Lagrange variable À from given search space A. We initialize with Åo and the distribution over policies D to Dn respectively. In each iteration I, we perform two algorithmic steps. First, we employ the given offline safe RL training dataset DOSRL = {(s,, ag T4, C2 +))) of n examples and convert it into a corresponding offline RL training dataset DoRL = [(s,, a;, r. s() by defining new reward r' that combines original reward r and cost e values using de based on the Equation [2] r
ri A (c. (1 - x)) where is the discount factor and x is the cost constraint threshold. We call an offline RL oracle on the dataset Rs. to get the distribution over policies D, Second, we call a no-regret algorithm |azan et al. 2016) by passing d and D, to get the updated Lagrange variable Aç. At the end of 7 iterations, we return the average distribution of policies D and the average Lagrangian value À as the solution. Algorithm provides a pseudo-code code of the O3SRL framework.
In what follows, we first formally define offline RL oracle and no-regret algorithm. Next, we provide a theoretical result that guarantees that the O3SRL framework converges to the minimax solution.
Definition 2 (Offline RL oracle). Let Doffine Rt. be an offline RL oracle that takes the offline data 'Di of size from an MDP with the expected reward functions f (and state transition function P) and produces a policy distribution D O Doffine-Re RL (Dy) E AlI the estimation error Colline- (n), i.e.
E E D ne (Dp) [V] V7] < Cuftiiee Rt. (T.).
(4)

---

Algorithm General O3SRL Framework for Offline Constrained RL via No-Regret Algorithms
Require: Offline training data DOSRL = {si, Oi, Ti G₁ s Offline RL oracle Rot no- regret update oracle NO RORREEERE _UPDATE, and search space of Lagrange variable A Initialize Xo A, Do E ATI
2: for = .,... do
3:
Dr < Doffine.RL RI. (DORL = ((8, Ci, 04, = - (1 7)8,, 8))))1)
4: be < NO_REGRETTTTEE LOC A)
5: end for
Ensure: D == Ei= Dr and E/= d
Parameter averaging
Remark The estimation error of an offline RL algorithm ColllineRLL also depends on the coverage of the offline data for the optimal policy. This coverage in turn depends on the underlying MDP class, i... the stochastic transition function and reward function. In Algorithm[]] we employ an offline RL algorithm repeatedly for multiple MDPs with different reward functions. However, the same order rate Coffline RI. (n) can be applied to multiple such offline RI. guarantees, with perhaps at the cost of a constant factor. The main reason is that, most notions of coverage depend very mildly on the reward functions, if at all. For example, the concentrability coefficients (Liu et al. 20200] [Rashidinejad et al., 2021] do not depend on the reward function at all. Similarly, the data diversity measure in [Nguyen and Arora 2024 do not
Tang and Arora, 2023a] and the policy transfer coefficient [Nguyen-Tanee depend on the specific reward function but rather class of reward functions. In particular, if use Fo approximate V. and G to approximate V then -- A-G := (f ff f F.g E G, E A} suffices to control the same policy transfer coefficient of class of MDPs defined by any newly created reward functions used in Algorithm !
Additionally, under good data coverage, we typically
have cottine (n) O (
Definition (No-regret algorithm). Let NO REGRET_UUDATE is no-regret algorithm for X E A with regret bound (A), i.e.,
At+1 NO REERRREGEEUUEE BEBRET_LUDDATEMA..D.......
where
L(D..A))
min Σ
(5)
2m11
and limi 100 (A) 0.
We consider a generic framework for solving offline constrained RL as shown in Algorithm relies offline RL oracle as per Definition 2 and no-regret update oracle as per Definition 3
From the given offline safe RI. training dataset Dosns = [(s;, as Ti; Gr K)) of examples, we can create a corresponding dataset that simulates the experiences of any MDP with the same unknown state transition probability function and an arbitrary reward function r' = f(r. c). by simply computing r f(r,, c.). Specifically, in our algorithm we define the modified reward function based on the Equation 2 as: r' = Ae(c- 11--1111-0 (1-7)k) where X E A is the Lagrange variable, y is the discount factor, and the cost constraint threshold.
Remark 2.
The below theorem provides guarantee that Algorithm wwll converge to the minimax solution.
Theorem 1. Let (D, X) be the output of Algorithm where e Cofllime-RL RL (n) Rr(A).
Then, (D,X) is e-approximate equilibrium,
T
See the detailed proof in Appendix A
5 Practical Algorithms
In this section, we introduce approximation to Algorithm that addresses two key practical challenges. Below we outline these challenges and the corresponding modifications, The resulting Algorithm 2 with minor approximation forms the basis of our primary empirical contribution.
5

---

The first practical challenge is that Algorithm 1 assumes access to an offline RL oracle. which is computationally expensive approximate by running offline RL algorithms to convergence. This is especially problematic because the oracle must be invoked in every round of the algorithm. To mitigate this cost, we assume weaker oracle: a stochastic oracle for offline RL in Assumption [ This assumption says that we have access to an oracle that returns a near-optimal policy and stochastic approximation of the value function of this near-optimal policy.
Definition Let $D SD offine-RL be a stochastic oracle for offline RL that takes the offline data D' of size nu from an MDP with the expected reward function f (and state transition function P across all MDP instances we have considered so far) and produces policy distribution and stochastic approximation of the value of the optimal policy: (D,V) S0oflineRt (D33), where
EDy B [V] VJ] S Cofliie- (n). and [V[] END [V]]
(6)
This significantly relaxes the need return exact value of the value function of near-optimal policy by an offline RL oracle, and, more importantly, leaves room for further practical approximation (as we elaborate in Section 5.I).
The second practical challenge arises from the use of no-regret algorithms over a continuous range of X. The online optimization over continuous domain of À requires an off-policy estimate of target policy under a mixed reward of the form r Ac for every value of A. This often requires an OPE procedure (to compute V c-(1->)> where De is the near-optimal policy distribution returned at line in Algorithm|] D at every iteration. The application of OPE at every iteration is problematic in two ways: 1) Can compound the OPE estimation error exponentially over the length of the iterations, causing instability and large error; and 2) Employing OPE for every iteration is computationally expensive. To address this, we discretize the range of À into K discrete values (A(1), A {(K) and optimize over this finite set. This transforms the problem into multi-armed bandit (MAB) setting, where each of the arms corresponds to a particular À value. Crucially, many MAB algorithms do require estimates of the underlying value functions, thus avoiding the need for OPE. In this work,
we adopt the EXP3 algorithm Auer et al.] 2002a]. which maintains a distribution over arms that is updated and sampled from in each round of Algorithm
Algorithm 2 O3SRL Approximation via Stochastic Oracle and EXP3 Multi-Arm Bandit Strategy
Require: Stochastic oracle for offline RL SOofflnn-RL. Offline safe RL training data DOSRL = {(8;, a4s Tis C2 } and search space = (X(), ,..,,, (k)
Initialize (A( K č= KKHH, where is cost limit and H I-7 is the horizon
2: for 1...., do
3:
(D,, Vi) S0 Sttttic-RL (D. = {(st, ai =r; A
4: Sample de- P. where P(()) x P2-1 exp (-1 B11(A=A--1)
Pe-1())
5: end for
Ensure: D = Projan (+ D.). x = ProjA (+
Theorem 2. Let A (2(1), ...., XK)3, where X and ((+1)) x = &- Then, the solution
(D, X) returned by Algorithm is e-approximate minimax equilibrium of Equation 3 where
E e= 0
K
CofflineMu(1)
VT
+
K
error rate of offline
regret of EXP3
discresization ernor
Remark 3: If we set K 133) and assume the minimax rate v of offline RL algorithm, then the error rate in Theorem 2 will be in the order of /
1 + 87. This rate is a bit slower than the rate
had we have applied a no-regret algorithm such as Follow- Dollow-The-Regullrized--LL Regularized-Leade (Hazan over continuous values of A.
Vn
VT
al.
2016

---

5.1 Final Approximate Algorithm for Empirical Evaluation
We consider an even more practical version of Algorithm 2 to form the final algorithm for our empirical evaluation. Specifically, we make two changes.
We further approximate the stochastic oracle by leveraging the fact that most offline RL. algorithms are based on stochastic gradient descent. In our approximation (Algorithm 2 rather than running the offline RL algorithm to convergence at round E, we perform only M gradient updates, after initializing from the result at the end of round t - 1. We find this truncated update scheme to effective even for small values of M.
Algorithm 2 (and Algorithm D maintains an averaged distribution over policies across iterations that returned at the end of I' rounds. In our case, this would involve storing the policy produced on each round, which would be prohibitively memory intensive. Instead, we simply return the last-iterate policy Tr.
In our experiments, we have found that this set of approximations leads to state-of- of-te-----e-rttttt performance even for as few as K=2 arms and small number of Offline RL iterations M per round.
Experiments and Results
In this section, we experimentally evaluate the performance of the approximate O3SRL algorithm described in Section 5.1 compare with state-of-the-art baselines, and discuss the results.
6.1 Experimental Research Questions
Our experiments are designed to answer the following key questions:
QI: What is the influence of the number of arms in our EXP-3 based approximate O3SRL approach on both reward and safety objectives?
Q2: How effectively does D3SRL_satisfy safety/cost constraints and achieve high reward performance for small cost constraint thresholds?
Q3: How does the safety and reward trade-off of O3SRL vary as the cost threshold increases?
Q4: Is coSpaicoepatible with different offline RL algorithms, demonstrating its generality?
Experimental Setup
Benchmarks. We evaluate O3SRL and baseline methods using the DSRL Bullet benchmark [Liu
et al. 2024], which provides standardized offline datasets for safe RI. research and evaluation. We consider eight continuous control tasks, comprising four "Run" and four "Circle" environments. Each task involves a different agent interacting with either the Run or Circle objective, following the common Agent-Task naming convention (e.g., Car-Run, Drone-Circle).
In the Run tasks, agents are rewarded for moving quickly between two boundaries while adhering to safety constraints, such as velocity limits and positional bounds. In the Circle tasks, agents are encouraged to follow a circular trajectory within a restricted safe region. These are challenging environments which require agents balance performance with constraint satisfaction.
Evaluation metrics. Following the DSRL evaluation protocol [Liu et 2024), we evaluate all methods using normalized cumulative reward and normalized sumulative cost. Let Tmax (M) and Tmin (M) denote the maximum and minimum empirical reward returns achievable for task M. The -T (M)
normalized reward is computed as Rnormalized Tioox (M) Tain (M) where Rx is the cumulative reward of policy म. The normalized cost is given by Cnormalized = Cz with λ being the cost budget.
where C₂ is the cumulative cost of the policy x and x is the cost threshold. A policy is considered safe Cnommalieed S 1. Our 03SRL method general-purpose and can be trained for any given cost threshold k. However, we report our main results under stringent low cost threshold of k=5 to
7

---

highlight the method's 's effectiveness in highly constrained settings. We report both mean and standard deviations over three random seeds, with each seed's policy evaluated across twenty episodes.
Baseline methods. We compare O3SRL with several state-of-the-an offline safe RL methods. (1) BC-Safe is a behavior cloning baseline trained exclusively on safe trajectories that satisfy the given cost limit. (2) BEAR-Lag is a Lagrangian- based extension of BEAR [Kumar et al. 2019]. which incorporates safety constraints during training. (3) CPQ (Xu al. 2022b penalizes unsafe actions by treating out-of-distribution actions as inherently risky and updates its Q-function using only safe
transitions. (4) COptiDICE [ee et al. 2022] builds on OptiDICE [Lee et al., 2021] by applying stationary distribution correction under cost constraints. (5) CDT [Liu et al. (023) is transformer-
sequence modeling. (6) CCAC [Guo et al.) 2025) learns a policy that adapts to changing safety budgets by jointly modeling constraints and environment dynamics. This approach conditions both the policy and value functions on constraint information. (7) CAPS [Chemingui et al. 2025), similar to CDT and CCAC, supports test-time adaptation to varying cost constraints by switching between a set of pre-trained policies with different cost and reward trade-offs.
based method that conditions policies on returns and costs, allowing constraint-awar
(8) FISOR [Zheng al. 20224] is a diffusion -bsed method that explicitly optimizes for feasibility, aims to produce zero-violation policies, and is the state-of-the-ar method for stringent cost constraints. All baselines are evaluated under the same cost constraint threshold k = 5. All results are averaged over three random seeds, with each policy evaluated across 20 episodes. Additional details are in the Appendix.
Configuration of O3SRL. We implement the approximate O3SRL algorithm described in Section 5.1 using BC (Fujimoto and Gu [021] as the underlying offiin RI. algorithm for our main results. We set the search space of Lagrange variable A= [0. C=5] and use K=5 arms for the main results but we also present ablation results for K=2, 5, and 10. We observe that increasing K beyond 5 (say to 10) yields only marginal improvements compared to the gains observed when increasing from K-2 to K=5. The algorithm is trained for T = 100,000 iterations, with arm probabilities updated every M = 10 steps of the base offline RL algorithm (i.e., M=10 stochastic gradient updates of the policy). For baselines, we employ the DSRL (Liu et al.] |024 implementation wherever available. For CCAC, CAPS and FISOR, we use the publicly released code provided by their respective authors. We employ IQL based instantiation of CAPS in our evaluation. Full architectural and hyperparameter details are provided in the Appendix.
6.3 Results and Discussion
Performance of O3SRL vs. the number of arms K. To evaluate the impact of the granularity of discretization of X, we test O3SRL using K E f2, 5, 10} arms. Table I shows the corresponding results. Across all six benchmark tasks, O3SRL satisfies the cost constraint under every configuration and maintains consistently high reward, demonstrating that the EXP3 strategy is effective even with a coarse discretization of X. With just two arms, the method keeps average cost at or near zero, but tends to underperform slightly in terms of rewards. Increasing to ten arms improves rewards marginally. The five-arm configuration achieves a good trade-off: it either matches or exceeds the highest feasible reward across all tasks while keeping costs safely below the threshold. For this reason, we adopted K 5 configuration of the approximate O3SRL approach for all main results.
Table 1: 03SRLresult for different values of K: the number of arms. Each value shows the average reward (t) and cost (L). Higher reward is better, and lower cost (up to threshold 1) is better.
Arms
Rull
CarRun
AntRun
BallCirele
CarCircle
AntCircle
reward
cost
reward
cost J
reward
cost
reward 1 cost
reward
cost
reward 1 cost. J
K =2
0.20
0.00
0.96
0.01
0.24
0.03
0.62
0.00
0.64
0.00
0.44
0.04
K = 5
0.25
0.00
0.96
0.02
0.33
0.14
0.62
0.06
0.66
0.11
0.48
0.00
K
0.27
0.00
0.96
0.00
0.29
0.17
0.64
0.28
0.67
0.15
0.49
0.00
O3SRL vs. Baselines. Table[]reeeents the comparative evaluation of O3SRL against baselines across
eight DSRL tasks. The table reports both normalized reward (higher is better) and normalized cost (lower is better, with values less than I indicating constraint satisfaction). O3SRL is the only method that consistently satisfies the cost constraint across all eight tasks. This is particularly noteworthy given the stringent cost budget, which poses significant challenge for all bascline algorithms.
8

---

Table 2: O3SRL normalized rewards and costs results. The cost threshold is 1. The 1 symbol denotes that higher rewards are better. The 1 symbol denotes that lower costs (up to threshold 1) are better. Each value is averaged over 20 evaluation episodes, and three random seeds. Bold: Safe agents whose normalized cost I 1. Gray: Unsafe agents. Blue: Safe agents with the highest reward.
Tasks
Sale
BEAR-lag
CPQ
COPIDICE
CIT
CCAC
CAPS
FISOR
038RI
HallKuun
Cost 4
Reward
0,16:0011
4.50
18.601635
0.53#004
0.09=026
2.20
10.83
0.53
0.27
2,57g323
0.00:000
0.31=000
0.00:000
0,07=uu
0.09=0007
1.28
0.25::005
0.00::000
CarRun
Reward
0.92::4:1
0.25
0,93-mm
0.91::M4
0.99:um
1.82a484
0.97=um
0.74
0.96::901
Cost 1
0.26::9:33
30.17::0.44
0.20=0018
0.00:0.00
0.30
24.57=3009
0.11=0117
0.00
0.02:0003
DroneRan
Cost &
Reward
0.41
1.62
14.85
-0.21=0112
2.35
0.29
15.02::..0
0.68#001
0,58=000
0.07:amm
0.5 0.50
16.29
0.112006
5.70
0.31=004
2.52
0.68::1.1
0.32::045
AntRun
Reward
0.566900
0.02:442
0,03
0.6120011
0,70gum
0.03
0,53g
0.43
0.33
Cost
1.15:047
0.00:8..1
0,05=000
3.26::1.9
1.66
0.00:0000
2.03:22.17
0.27::0115
0.14::0.1
BallCircle
Reward
0.481001
0.85
0.56 in
0.71
0.47
0.33
0.32:
0.62:
Cost L
1.21
10.9242.8
1.11
9,41600
2,03
10.19
0.01
0.00
0.066:9487
CarCrelle
Cost b
Reward
0.31
1.57
7.39
0.73
0.71=au
0.00
10.821:128
0.49:001
a.71
1,58
0.70
1.61
0.03__u
0.40:001
0.000:9.00
0.37:00.02
0.11:
0.66::8883
DroneCirclo
Cost L
Reward
0.501001
1.22a0.31
15.404143
0.84
.0.11 19:4
0.70:=0.1
0.26:1062
3.68
0.55
1.14gu0s
0.06
0.43:044
0.36
0.00-u0
0.48:0001
0.17::015
0.49:6687
0.23
AntCirele
Cost L
0.40s002
4.72
19.001144
0.67s00s
0.00
17.4010.1
0,18:8022
6.
0.45
0.02:19.68
0.00-umm
0.33=
0.04:
0.24::9/2
0.48::806
0.00
O3SRL's 's consistent satisfaction of the cost constraint stands in contrast to methods that only occasionally achieve strong reward, often by violating safety. For instance, CDT achieves high rewards on tasks such as CarRun and DroneRun, but exceeds the safety budget in the other six maskinnaking it unreliable. Similarly, CCAC achieves the best rewards on BallRun and AntCircle, yet violates the cost constraint in four out of eight tasks, further underscoring the challenge of maintaining both high performance and safety across environments. CPQ, on the other hand, stays within the budget more often, but its performance is inconsistent: it achieves a competitive reward on CarCircle, while falling short rewards when it does remain safe, and it exceeds the cost threshold in three tasks.
While CAPS and FISOR maintain safe behavior in six out of eight tasks, their reward performance consistently trails behind O3SRL. For example, in BallRun, O3SRL is one of the only three safe methods (along with CAPS and CCAC), yet it achieves reward of 0.25 = 0.03, which is over three times higher than CAPS (0.07 = 0.05). A similar pattern holds in BallCircle, where O3SRL maintains safety and attains 0.65 = 0.01 reward- more than double that of CAPS (0.23 = 0.17) and FISOR (0.27 + 0.15), both of which are also safe in that task. Note that FISOR, which aims for zero cost policies, fails meet the cost threshold in the DroneRun and BallRuun environments.
Overall, the results in Table 2 demonstrate that O3SRL strikes a uniquely favorable balance between maximizing rewards and satisfying safety constraints. It consistently ensures safety where many other methods fail, especially under a tight cost budget. Crucially, this safcty does not come at severe expense to the reward performance; O3SRL is the pest-performing method (in terms of reward) among safe agents in two out of eight tasks, and consistently ranks second in the rest. This suggests that O3SRL's 's iterative, no-regret approach to optimizing the reward and safety trade-off is more robust and effective than direct Lagrangian optimization or specialized mourisiccbbsse safety mechanisms in these challenging offline safe RL settings.
Performance of O3SRL vs. increasing cost limits. To assess the flexibility of O3SRL in handling different safety constraints, we perform an ablation study across two extra cost limits beyond the default setting of 6 = used in the main experiments. Table shows the corresponding results. We find that O3SRL continues to perform effectively under higher cost limits (i.c., less stringent safety constraints). adjusting its behavior to make better use of the available budget. As the constraint becomes more permissive, the learned policy shifts toward higher-reward strategies, while still respecting the specified cost limit. These results demonstrate that the O3SRL method does not rely on tuning for particular budget.
O3SRL with an alternative Offfine RL algorithm. To evaluate the generality of O3SRL with respect to the underlying offline RI. algorithm, we conduct an ablation using Implicit Q-Learning (IQL.) [Kostrikov et al. 2022) in place of TD3+BC, which is used in our main experiments, on the Circle tasks. Table 4 shows the corresponding results. We observe that O3SRL combined with

---

Table 3: O3SRL results for different values of k: cost budget. Each value shows the average reward (f) and cost (L). Higher reward is better, and lower cost (up to threshold 1) is better.
Cost Limit
Rall
DroneRun
BallCirele
DioneCircle
reward
cost 4
reward
cost
reward 1
cost
reward 1 cost
0.25
0.00
0.32
0.68
0.62
0.06
0.49
0.23
20
0.26
0.06
0.38
0.41
0.69
0.58
0.53
0.36
40
0,53
0.71
0.38
0.44
0.76
0.64
0.65
0.85
IQL. achieves similar performance to the TD3+BC variant, both in satisfying safety constraints and attaining competitive reward. These results demonstrate that O3SRL is not tied to a specific offline RL method and can effectively incorporate different algorithms, highlighting its flexibility general framework for offline safe reinforcement learning.
Table O3SRL for different base offline RL methods. Each value shows the average reward (1) and cost (4). Higher reward is better, and lower cost (up to threshold 1) is better.
BallCirele
CarCirele
DroneCirele
AntCircle
Offiine RL.
reward
cost 4
reward
cost 4
reward f
cost
reward f cost
TD3BC
0.62
0.06
0.66
0.11
0.49
0.23
0.48
0.00
IQL
0.54
0.06
0.63
0.10
0.42
0.03
0.41
0.06
In Appendix, we provide extended experimental results including evaluation on more benchmark tasks; treating fixed byper-parameen and ablation results for O3SRL over search space of and discretization levels, and the number of stochastic gradient updates of offline RL (M).
7 Summary and Future Work
This paper developed and evaluated a novel online optimization framework called O3SRL to create safe and reward-maxine maximiziixiiiggggg decision policies from offline datasets. O3SRL solves minimax opti- mization problem by employing an iterative approach that adaptively updates the policy distribution using an offline RL oracle and Lagrange variables in each iteration. To handle the practical chal- lenges, we studied an approximate algorithm that works over discrete values Lagrange variables (to avoid offline policy evaluation) and performs few gradient updates of offline RI. algorithm (for computational efficiency). Empirical results show that our approximate O3SRL algorithm outper- forms state-of-the-art methods, especially for stringent safety/cost constraints. Future work includes applying O3SRL to real-world applications and extending it to the offline-to-onliin safe RL setting.
Acknowledgments
The authors gratefully acknowledge the in part support by USDA-NIFA funded AgAID Institute award 2021-67021- 35344. The views expressed are those of the authors and do not reflect the official policy or position of the USDA-NIFA.
10

---

References
E. Altman, Constrained Markov decision processes. Routledge, 2021.
G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-based offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021. 2021.
P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing. 32(1):48-777 2002a. doi: 10.1137/50097553901388877
Auer, N. Cesa-Bianchi, Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal computing, 32(1):48-7) 2002b.
C. Bai, L. Wang, Z. Yang. Z.-H. Deng, A. Garg, P. Liu, and Z. Wang. Pessimistic bootstrapping for uncertainty-drrive offline reinforcement learning. In International Conference on Learning Representations, 2022. URL https:://operreviex hett/forum'dd-Y4cs122333
Y. Chemingui, A. Deshwal, T. N. Hoang, and J. R. Doppa. Offline model- optimization via policy-guided guided gradient search. In Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, pages 1 11230-12 1239. AAAI Press, 2024.
Y. Chemingui, A. Deshwal, H. Wei, A. Fern, and J. Doppa. Constraint-adaptive policy switching for offline safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 15722 15730, 2025.
H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023.
X. Fang, Q. Zhang. Y. Gao, and D. Zhao. Offine reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference Intelligent Transportation Systems (FTSC), pages 3417-342 IEEE, 2022.
R. Figueiredo Prudencio, M. R. O. A. Maximo, and E. L. Colombini. A survey on offline reinforce- ment learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 35(8): 2024. doi: 10. S.0/TNNN.S.20233252222222
S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Z. Guo. W. Zhou, S. Wang, and W. Li. Constraint-conditioee actor-critic for offline safe rein- forcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Hansen Hansen-Estruch. I. Kostrikov, M. Janner, J. G. Kuba, and Levine. Idql: Implicit q-learnnng as an actor-critic method with diffusion policies. arXiv preprint 1rXiv:2304.10573, 2023.
E. Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimization, 233-4):1573325, 2016.
M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems,
2021.
N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones. S. Gu, and R. Picard. Way off oof-policy batch deep reinforcement learning of human preferences in dialog. 2020. URL ftttpp:///opereviiie wtt.:eeee/ooummm.dd-r1111
R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline rein- forcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 21810-21823, Curran As- sociates, Inc., 2020. URL htRl/ittpps:/prrccedinge neurips http://cwwaan.aeeeeeeeccccpppeee flee/Foeeass60aaa0add55557770272
pdf
I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL htt.::ttps://// /ttpttpee//peeeeviiee.... forum/id-66n22297.Jvy

---

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off- q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference Machine Learning, pages 6120-6133 PMLR, 2021.
J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022. URL https://openrrevie net/ Forum2id=FLA55meeeeG]
S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint wxXiv:2005.01643 2020.
J. Li, X. Liu, B. Zhu, J. Jiao, M. Tomizuka, C. Tang, and W. Zhan. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. In 2024 IEEE International Conference on Robolics and Automation (ICRA), pages 7447-7454. IEEE, 2024.
H. Lin, W. Ding, Z. Liu, Y. Niu, J. Zhu, Y. Niu, and D. Zhao. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters, 2024.
Q. Lin, B. Tang, Z. Wu, Yu, Mao, Q. Xie, X. Wang, and Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning, pages 21127-21 152. PMLR, 2023.
Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Off-policy policy gradient with stationary distribution correction. In Uncertainty in artificial intelligence, pages 1180-1190. PMLR, 2020.
Z. Liu, Z. Guo, Y. Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 21611-21633 PMLR, 2023.
Z. Liu, Z. Guo, H. Lin, Y. Yao. J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan, and D. Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024.
J. Lyu, Ma, Li, and Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724 2022.
T. Nguyen-Tange and R. Arora. On sample-efficient offline reinforcement learning: Data diversity, posterior sampling and beyond. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL httppp////ppppeeeeeeeeaeee/eeeaa
T. Nguyen-Tann and R. Arora. Viper: Provably efficient algorithm for offline rl with neural function approximation. arXiv preprint arXiv:2332.12278) 2023b.
Nguyen-Tann and R. Arora. On the statistical complexity of offline decision-making, In Proceed- ings of the s/st International Conference on Machine Learning, pages 37900-37928, 2024.
N. Polosky, B. C. Da Silva, M. Fiterau, and Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning, pages 17801-17810, PMLR, 2022.
P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell. Bridging offline reinforcement learning and imitation learning: tale of pessimism. Advances in Neural Information Processing Systems, 34: 11702-11716, 2021.
M. Rigter, B. I Lacerda, and N. Hawes. Rambo-rl: Robust adversarial model-based offline reinforce- ment learning. Advances in neural information processing systems, 35:16082-16097, 2022.
12

---

A. Wachi, X. Shen, and Y. Sui. A survey of constraint formulations in safe reinforcement learningg arXiv preprint 242:2240..02025, 2024.
K. Wang, H. Zhao, Luo, K. Ren, Zhang, and D. Li. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:34748-34761, 2022.
H. Wei, X. Peng, A. Ghosh, and X. Liu. Adversarially trained weighted actor-critic for safe offline reinforcement learning. Advances in Neural Information Processing Systems, 37:52806-52835, 2024.
J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning, Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
Y. Wu, Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, 2020. URL httts://opperreiiet http://www....torrmmr.a....gggaa
H. Xu, L. Jiang, L. Jianxiong, and X. Zhan. A policy-guided imitation approach for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 35:4085-4099 2022a.
H. Xu, X. Zhan, and X. Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 8753-8760, 2022b.
R. Yang, C. Bai, X. Ma, Z. Wang, C. Zhang, and L. Han. RORL: Robust offline reinforcement learning via conservative smoothing. A. H. Oh, Agarwal, D. Belgrave, and K. Cho, editors. Advances in Neural Information Processing Systems, 2022. URL https::/ net/ Darumrid=QqzJJH.. KE
Y. Yao, Z. Cen, W. Ding, H. Lin, S. Liu, T. Zhang, W. Yu, and D. Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 37:78451-78478 2024.
T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14129-1414 Cur- ran Associates, Inc., 2020. URL http:httts//poceeddiiii htwpr/eeeeddiineeeeeeeeeeeeeeeee newrips. htsppaper_filesse paper/
pdff
X. Zhan, H. Xu, Y. Zhang. X. Zhu, H. Yin, and Y. Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1680-4688. 2022.
H. Zhang, X. Peng. H. Wei, and X. Liu. Safe and efficient: A primal-dual method for offline convex cmdps under partial data coverage. Advances Neural Information Processing Systems, 37: 34239-34269, 2024.
Y. Zheng. J. Li. D. Yu, Y. Yang. S. E. Li, X. Zhan, and Liu. Safe offline reinforcement learning with feasibility-guider diffusion model. In The Twelfth International Conference on Learning Representations, 2024. URL holpitttps:/pppeeeeeevv.eee http:///wwwe..eeeeoooor////////a
13

---

A Proofs for Theorem and Theorem 2
Proof of Thearem !
For any D ATT, have
1 M(),
L(D,))
(L(D, X) is linear in A)
T
1
$
(L(Di, Xc) Cottine-RL (n))
(due to Equation
(T(A) T
$ T L(De. X) + Colline (1) +
(due to Equation
= LD,, + Editiine RL (n) + (r(A)
(L(D, A) is linear in D).
T
For any À E A, we have
T
I(D,))
A(D.A)
T
(T(A)
2
Ad)
(due to Equation (
T
T
t=1
T
Colttine RL(()) (r((A)
Ae)
(due to Equation
4
T
T
t=1
Coffline (n) RT(AA)
= L(D, X)
T
Proof of Theorem 2 The result directly follows from the assumption of the stochastic oracle in Definition 4 the regret bound of EXP3 [Auer et al. 20026), and the approximation error for the projection from [0, CJ into the discrete domain A with resolution 1/K is 1/K.
In particular, by [Auer et al. 2002b), we have
M(D,,)) HV2VKKIK. Σ L(D,, Ax) min AEA
Applying the result in Theorem
1 we have
max L(D,)) +e < L(D,A) < min L(D, A) é1 DEAMI
XEA
where
T
fottine-RL (n) + H
2K log K
and = T
1
Σ
A
T
1=1
Note that the running average Î does not guarantee to be in A. Thus, with À Proja (³) projectioo of Å onto A, we have that,
I(DD,A) -(D,)) H K
Overall, we have
H K'
L(D,)) +1 K
LL(D,T) min L(D, +G+ xEa
DEAn Thus, (D,X) e-approximate equilibrium, where
2K log K
H K
Coffine-RLL (ni) H.
14

---

B Extended Discussion of Related Work
Lagrangian formulation is quite natural for offline safe RL. that both work and [Le et al. 2019 share. (Le et af.] 2019] and our work also share the general idea that we can use online optimization algorithms to tune the Lagrangian multiplier. The critical difference is in the design of efficient and effective algorithmic frameworks: our framework requires one call to an offfine RL oracle per loop and does not rely on any OPE oracle. On the other hand, [Le al. 2019 requires offtine RL oracle and four calls to an OPE oracle per loop: four calls to OPE oracle in each iteration can lead error propagation and exponential error over the number of iterations posing stability challenges; and six calls to oracles of different nature in each iteration leads to huge computational expense posing scalability challenges. Our approximate O3SRI algorithmic framework overcomes these challenges in principled manner.
calls to an
C Ablation Results for O3SRL
We provide additional ablation results for O3SRL beyond those presented in the main paper.
C.1 Extended Evaluation of O3SRL
To further evaluate the generalization capability of O3SRL, we extended our experiments to additional environments from the Gymnasium Safety Suite, following the FISOR [Zheng al. 2024 setup with a tight cost limit of 10. The results are summarized in Table 5
Table 5: O3SRL normalized rewards and costs results across SafetyGym environments. 1 denotes that higher rewards are better; J denotes that lower costs (up to threshold 1) are better. Bold: Safe agents whose normalized cost < 1. Gray: Unsafe agents. Blue: Safe agents with the highest reward.
Environment
CAPS
FISOR
CCAC
O3SRL
Reward
0.31
0.43+:0.05
0.57=0.08
0,53+00.22
PointCirclel
Cost 4
0.94
14.93
6.65
1.31:00.2
Reward
0.44:10.06
0.76:10.05
0.03
0.52:10.04
PointCircle2
Cost J
0.10:10.09
18.02:+4.17
3.99
0.55+0.41
Reward
0.804
0.8910.01
11.01:1.00
0.45
AntVelocity
Cost 4
0.19.00.6
0.00
0.00
0.05
Reward
0.86:10001 0.89
0.91=2004
0.93:10.02
HalfCheetah
Cost J
0.27+0.07
0.00++0.00
0.97=1012
0.00++0.00
Reward
0.29
0.12:10.03
0.51 -0.01
HopperVelocity
Cost +
1.61
0.90:11.07
0.00
0.20
Reward
0.38
-0.02:+0.05
0.06=1022
0.05++0.02
SwimmerVelocity
Cost +
2.1 2.1113001
0.23
0.91
0.00-10.01
Reward
0.78-10.01
0.51
0.40
0.77-10.02
Walker2dVelocity
Cost 4
0.08:10.07
1.92:10.57
5.20:11.16
0.02::0.02
O3SRL satisfies the cost constraint in 6 out of the 7 evaluated tasks, demonstrating strong general- ization across environments with varying dynamics and cost structures. The next performin method, CAPS, remains safe on 5 out of 7 tasks. In terms of reward, O3SRL achieves the highest normalized return on three environments and remains competitive on the remaining tasks. For example, O3SRL attains performance close to the best methods on SwimmerVelocity (0.05 vs. 0.06) and Walker2dVelocity (0.77 vs. 0.78) while maintaining safety.
Collectively, these results show that O3SRL maintains safety under strict cost constraints while achieving competitive or superior performance across diverse safety-critical environments.
C.2 Alternative Approach: Offline RL with Fixed À Hyperparameter
To assess the effect of treating the Lagrange multiplier as fixed hyperparameter, we conducted an additional experiment in which À was held constant throughout training, and the resulting policy
15

---

was evaluated at the end. This setup departs from the intended offline RL formulation, choosing an appropriate would typically require access online validation data or environment rollouts. Moreover, an exhaustive search over possible À values is computationally impractical and inconsistent with the offline setting.
In our proposed approach (O3SRL), we instead define discrete set of candidate À values, each corresponding to an "arm" in a multi-armed bandit framework. During training, a value is sampled from a learned distribution over these arms and used to shape the reward. The sampling distribution updated at every batch following the EXP3 algorithm, enabling adaptive reweighting based on the observed performance of each arm. This mechanism allows the policy to dynamically adjust the effective penalty strength, balancing constraint satisfaction and reward maximization throughout training rather than committing to single fixed X.
Empirically, as shown in Table fixed-A policies tend either ignore the cost constraints when À is too small or over-penalizz and sacrifice reward when X is too large. By contrast, O3SRL adapts its choice of during offline training, achieving more favorable trade-off between reward and cost across different constraint budgets.
Table 6: DroneCircle results with fixed arms under varying cost limits. The 1 symbol denotes that higher rewards are better; denotes that lower costs (up to threshold 1) are better. Bold: Safe agents whose normalized cost < 1. Gray: Unsafe agents. Blue: Safe agents with the highest reward.
Cost Limit
Arm
Arm
Arm 2
Arm
Arm
O3SRL
Reward f
0.90
0.86
0.55
0.47
0.39
0.49
5
Cost $
18.18
16.52
2.47
0.00
0.00
0.23
Reward f
0.90
0.87
0.75
0.49
0.40
0.53
20
Cost 4
4.54
4.33
2.80
0.04
0.00
0.36
Reward f
0.90
0.87
0.82
0.52
0.42
0.65
40
Cost $
2.27
2.21
1.88
0.08
0.00
0,85
C,3
Search space of Lagrange variable À
Recall that our search space for À is [0, CJ. Table 7 presents an ablation study on the impact of varying C, the maximum allowable value for the Lagrange variable X, which determines how strongly constraint violations are penalized in O3SRL. Results are shown for C = 2, 5, and 10.
The ablation results highlight that small values of C (e.g.. C = 2) often lead to insufficient constraint enforcement, resulting in catastrophic safety violations in some environments. For example, in CarCircle and DroneCircle, the average cost exceeds the threshold by factor of over 8 and 13 respectively, completely defeating the purpose of safe learning. These cases show that when is capped too low, the algorithm lacks the leverage penalize unsafe behavior adequately.
In contrast, increasing C' to 5 or 10 significantly improves safety across all tasks. For instance, in DroneCircle, the cost drops from 13.32 at C = 2 to 0.23 at C' = 5, and further to 0.00 at C = 10. However, in tasks with already low costs (e.g.. AntCircle), large values can leads to reduced reward, due to over-penalization.
Overall, C 5 emerges as robust choice, offering strong safety guarantees while maintaining high reward across environments. This ablation underscores the necessity of a sufficiently expressive À range enable O3SRL satisfy constraints in practice.
C.4 Discretization of X
Table & presents an ablation comparing two À set choices used in O3SRL: uniformly spaced set (e.g.. 1.25, 2.5, 3.75, 5)) and an adaptive set that retains the extreme values 0 and 5 but distributes intermediate values more densely near lower penalty regions (e.g., [0, 0.5. 1.12, 2.5, 5)). The purpose of the adaptive grid is to modulate constraint penalization based on the cost threshold k, allowing the algorithm to apply smaller penalties in more permissive cost budget scenarios. The results show that
while both grids perform similarly under striet cost threshold constraints (N = 5), the adaptive grid provides clear advantages as the constraint becomes more relaxed. For instance, under 40, the
16

---

Table 7: O3SRL results for different values of C: À maximum value. Each value shows the average reward (f) and cost (4). Higher reward is better, and lower cost (up threshold 1) is better. Bold: Safe agents whose normalized cost I 1. Gray: Unsafe agents.
Tasks
C 2
C == 5
C 10
Reward t
0.28 +0.01
0.25
0.25 :0001
BallRun
Cost
0.00
0.00
0.00
Reward f
0.97
0.96
0.95
CarRun
Cost +
0.06
0.02
0.00
Reward
0.43::0.10
0.32
0.33
DroneRun
Cost +
1.93
0.68 1.18
0.68
Reward
0.34++0.09
0.33
0.206
AntRun
Cost 4
1,16:1995
0.14
0.27:10.47
Reward t
0.82++0.6
0.62:10.01
0.58
BallCirele
Cost +
7.20
0.06
0.01
Reward
0.75+0.111
0.66:10.03
0.57
CarCircle
Cost +
8.07+7.53
0.11
0.00
Reward
0.79:+0.06
0.49:10.07
0.00
DroneCircle
Cost 13.32+2262
0.23
0.00
Reward
0.58.10.02
0.48:10.06
0.002
AntCircle
Cost 4
0.25:t0.26
0.00
0.04
adaptive grid allows significantly higher reward in tasks such as DroneCircle (reward improves from 0.45 to 0.65), by leveraging a smoother trade-off between reward maximization and cost constraint satisfaction. These findings suggest that finer resolution at low penalty levels enables better policy selection for higher cost constraint thresholds, making the adaptive grid more effective for balancing performance and safety.
Table results for different values the set of As and cost limits K. Each value shows the average reward (f) and cost ()). Higher reward is better, and lower cost (up to threshold 1) is better. Bold: Safe agents whose normalized cost < 1. Gray: Unsafe agents.
Tasks
20
40
uniform
adaptive
uniform
adaptive
uniform
adaptive
Reward
0.25
0.25.10.03
0.25 10.01
0.26.1..01
0.25 10.01
0.53
BallRun
Cost J
0.00
0.00
0.00
0.06.10.11
0.00
0.71
Reward
0.39
0.32+
0.31
0.38
0.30
0.38-10.00
DroneRun
Cost
1.95
0.68:+1.18
0.22++0.3
0.41
0.60
0.44-1044
Reward Cost +
0.63
0.62:t0.01
0.63
0.69
0.62+
0.76.10.04
BallCirele
0.07:10.12
0.06
0.20 0.300
0.58
0.10.10.10
0.64.10.18
Reward J
0.46-20.22
0.49
0.46-10.01
0.53
0.45
0.65
DroneCircle
0.00
0.23
0.00
0.36
0.00
0.85
C.5 Number of update steps MM
Recall that our approximate algorithm for empirical evaluation performs M stochastic gradient updates a given offline RL algorithm each iteration. Table@ppreeents an ablation on the number of offline RI. update steps M. This parameter controls how long the policy is trained before updating the value balance reward and constraint satisfaction. The results demonstrate that M = 10 provides stable trade-off between learning progress and responsiveness to constraint violations.
17

---

With M = 1, the frequent updates to À overall increases costs adjusting the policy aggressively. Conversely, setting M 100 delays updates, which can result policies drifting away from feasible regions, as seen in the sharp increase in costs for DroneRun and DroneCircle. Overall, the ablation results validate that safe and effective offline learning in O3SRL requires coordinated progress on both the policy and the constraint via well-timed À updates. The choice of M = 10 strikes this balance, supporting its use for all the results presented in the main paper.
Table 9: results for different values of M: Offline RL update steps. Each value shows the average reward (f) and cost (L). Higher reward is better, and lower cost (up threshold 1) is better. Bold: Safe agents whose normalized cost s 1. Gray: Unsafe agents.
Tasks
M = 1
M = 10
M 100
Reward f
0.25++0.03
0.25
0.26=0003
BallRun
Cost 4
0.37
0.00++0.00
0.00
Reward f
0.96:10000
0.96110.01
0.96=0.00
CarRun
Cost +
0.00::0.01
0.02:+0.03
0.01
Reward
0.34:10.06
0.32.10.05
0.27
DroneRun
Cost +
1.10:10.90
0.68
3.72:14.44
Reward f
0.34
0.33
0.39
AntRun
Cost 4
0.20::0.21
0.14+:0.10
0.97
Reward f
0.67:10.00
0.62
0.67 1: 0.05
BallCirele
Cost 4
0.25:10.19
0.06:10.07
0.61 0.88
Reward
0.66.10.01
0.66.10.03
0.68 0.01
CarCircle
Cost |
0.10.10.17
0.11
0.17 1: 0.29
Reward
0.44:10.06
0.49:+0.07
0.58=0020
DroneCircle
Cost 4
0.11:10.10
0.23
4.87=17.77
Reward
0.41::0.06
0.48
0.49
AntCirele
Cost $
0.02.10.03
0.00
0.00
C.6 Evaluation Across Multiple Random Seeds
To assess the stability of O3SRL with respect to random initialization, we extend our evaluation to include additional random seeds. Specifically, we compare results averaged over the original three seeds (10, 20, and 30) with those averaged over all six seeds (10, 20, 30, 40, 50, and 60). Table [T reports the mean and standard deviation of rewards and costs under both settings.
The results across confirm the stability of O3SRL, showing consistent rewards and safe cost levels across all evaluated environments. The performance trends observed with three seeds largely persist when extended to six, indicating robust and reliable behavior under different random seeds.
D Experimental Details
This section provides additional details about the experimental setup and pyper-parameters.
D.1 Description of environments
We evaluate O3SRL performane in a of environments from the Bullet-Safety-Gym suite. Each environment is defined by combination of robot type and task, resulting in scenarios such as BallRun, CarCircle, DroneRun, and so on. The agent types include Ball, Car, Drone, and Ant, and each is evaluated under either the Run or Circle task. Figure illlustrates the Buller-Safety-Gin environments, demonstrating various agent types (Ball, Car, Drone, Ant) performing the Run and Circle tasks.
18

---

Table 10: Comparison between 3-seed and 6-seed experiments. Reward (1) indicates higher is better, and Cost (L) indicates lower better. Bold: Safe agents whose normalized cost 1.
Environment
seeds
seeds
Reward f
0.25=0.03
0.26:::0.02
BallRun
Cost
0.00:00.00
0.00:10.00
Reward f Cost
0.96
0.96110.01
CarRun
0.02
0.00:10.00
Reward f
0.32:=0.05
0.37
DroneRun
Cost +
1.68
0.74
Reward f
0.13
0.22:10.10
AntRun
Cost +
0.14
0.17:100119
Reward f
0.62:10.01
0.63:1:0004
BallCircle
Cost +
0.06=0.07
0.26:10.25
Reward f Cost
0.66
0.66.10.03
CarCirele
0.11
0.04
Reward f
0.49
0.47=+0.03
DroneCircle
Cost +
0.23
0.02.10.04
Reward f
1..66
0.44.10.05
AntCircle
Cost
0.00==0.00
0.02:++0.03
Run Environments: In the Run task, agents are required move quickly along straight corridor. They receive rewards based on forward progress, encouraging high-speed locomotion. However, they are penalized for erossing lateral safety boundaries or exceeding predefined velocity threshold. These safety constraints are not enforced through physical barriers but through cost signal that tracks violations.
In the Circle task, agents are trained to move in a clockwise direction around a circular track, Rewards are higher when agents maintain fast, smooth motion near the boundary of the circle. Deviating from the intended path or exiting the safety zone results in penalties.
Circle Environments:
Agant
Agant
Agant
Ant Agent
Circle Task
Run Task
Figure I: Visualization of the Bullet-Safety-Gyy environments featuring different agents and tasks.
D.2 Training details
Our implementation available at
/
Adaptive À set: We construct the set using geometric spacing scheme that adapts to the cost limit. The grid always includes and maximum value Amux = C, and places l intermediate points between them. These intermediate values spaced geometrically and scaled by a factor
19

---

shrink = (reference /mmit/cost_um casss_lmittaaaaaa which compresses the grid as the cost limit increases, where reference limit and Oshrink 0.3. This design ensures finer resolution among low- low-penalty values when constraints are loose, and broader coverage when strict cost constraints require stronger penalization.
Rewards scale:
We observe that reward distributions in offline datasets can be heavy-tailed, with occasional extreme values that disproportionately affect learning stability, as illustrated in Figure
Reward distribution: BallRun
23900
2000
0000
Coumt
6000
2000
10
Devoord
Figure 2: An example of a heavy-tailed reward distribution from the BallRun dataset.
To mitigate this, we clip rewards symmetrically at the 7-th percentile of their absolute values:
Telip percentile(Ir).7))
clip(r, -Flip Telip)
where T = 99. We then scale down the clipped rewards multiplying by 0.9/reli-
We employ a batch size of 1024 for the tasks CarCircle, DroneRun, AntCircle, and BallCircle, while batch size of 512 used for the remaining tasks: CarRun, DroneCircle, AntRun, and BallRun. The other hyperparameters used in our experiments are listed in Table 11 For TD3+BC, we adopt the same hyperparameter settings in the original paper.
Hyperparrameter:
Table 11: O3SRL Hyperprrmtes
EXP3 parameters
Number of arms k
Maximum C
X update frequency M
10
À learning rate
2e-3
TD3BC parameters
Discount y
0.99
Policy noise
0.2
Policy noise clip
(0.5, 0.5)
Policy update frequency
O
2.5
Optimizer
Adam
Actor, Critic learning rate
3e-4
Actor, Critic hidden size
256
Training steps
100000
Seed
[10, 20, 30]
Training Curves of O3SRL: Figure shows the training curves of O3SRL the different tasks, plotting both the average reward and constraint cost over training iterations. The curves
20

---

demonstrate that O3SRL is able to quickly stabilize and achieve favorable trade-off between reward maximization and cost minimization. In most tasks, the cost decreases steadily and remains below the constraint threshold, while the reward improves progressively.
BallRun
CarRun
DroneRun
AntRun
Reownd
Powaad
3000
199.0
20090
COOE
BallCircle
CarCrele
DroneCircle
AntCircle
2080
120.0
2000
20000
2802
2008
3:2
Dng
Figure 3: Training curves of O3SRL across different tasks, showing average reward (top) and cost (bottom) over training iterations.
Computational Resources and Training Time: All experiments were conducted using NVIDIA A40 GPU with 48GB of memory. Training single task for one random seed takes approximately 20 minutes when the server is not under load.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provvided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used").
general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction claim to propose practical OSRL framework based on a min-max game solved with no-regret algorithms, which reliably enforces safety constraints while maintaining strong reward performance. The paper develops this frame- work (Algorithm and 2), provides theoretical convergence guarantees (Theorem and 2). and empirically demonstrates its effectiveness on the DSRL benchmark (Table 1), aligning with these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
22

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper explicitly discusses several limitations of its proposed method, particularly Section 5. It identifies two practical challenges: (1) the instability and compu- tational cost associated with using off-polic evaluation (OPE) procedures and continuous no-regret optimization over the Lagrange multiplier, and (2) the inefficiency of running offline RL algorithms to convergence during each iteration. To address these, the paper introduces a practical approximation using discrete values for the Lagrange variable and truncated gradient updates. These discussions acknowledge trade-off's in approximation quality, computational cost, and potential performance limitations, satisfying the guideline for reflecting practical constraints and assumptions of the approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper presents two main theoretical results: Theorem 1 concerning the convergence of the general framework (Algorithm 1) and Theorem 2 for the practical algorithm (Algorithm 2). Assumptions for these theorems (e.g., access to an offline RL oracle with defined error, properties of the no-regret algorithm, discretization of A) are stated. A proof sketch for Theorem 1 is provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
23

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper details the experimental setup in Section 6. including the bench- mark used (DSRL Bullet benchmark), specific tasks, evaluation metrics (normalized reward and cost with formulas), and the cost budget (x=5). It lists the configuration of O3SRL and all baseline methods. The full stepwise pseudocode of O3SRL (Algorithm 2) is described, including the use of EXP3. This level of detail should allow for reproduction of the main experimental comparisons.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer:[Yes]
24

---

Justification:
Our code and data will be made available on public GitHub repository. We also provide the anonymous code repository as supplementary for reviewing purposes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
aublicc/uudee//Codeuubnii wwt.ibmmissiooooocycocoo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
//nips http://///pb.ciguddee/es//////s/ htt////oooSSiisseeeeeeee irriiisssinpPeieee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides sufficient detail understand and reproduce the training and test setup. In Section 6.2 ("Experimental Setup"), "), it specifies the benchmark used (DSRL Bullet), describes the tasks, and outlines the evaluation metrics including how normalized rewards and costs are computed. It mentions key parameters such as the cost threshold (= 5), the number of evaluation episodes (20), the number training iterations (T = 100,000), the number of gradient steps per round (M = 10), and the number of random seeds (3). Additionally, the base offline RL algorithm (TD3+BC) is specified. For further transparency, the paper states that additional architectural and hyperparameter details are provided in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table reports mean values along with standard deviation for both normalized reward and cost evaluated over three random seeds which allows for assessment of variability and statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Details the computational resources used including hardware specifi- cations (GPU/CPU type, memory), and approximate training time - are provided in the appendix to support reproducibilityy
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the http://wwwrpppecccpppiii/eeeeeea
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research fully conforms to the NeurIPS Code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
26

---

The research focuses on methodological development for offline safe reinforce- ment learning, and we do not see any direct societal impacts of the work that would require specific discussion.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-party assets used in this paper are properly credited and used in accordance with their licenses. The DSRL Bullet benchmark datasets and associated imple- mentation are released under the Apache License 2.0, which permits use and modification with attribution. For the CAPS and FISOR baseline methods, we use the publicly available implementations provided by the original authors on GitHub. We cite all relevant works in the paper and comply with their usage terms outlined in their respective repositories
27

---

Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The primary new asset introduced in this paper is the O3SRL algorithm. The paper provide its implementation in the appendix, along with sufficient documentation to reproduce the main results. This includes code and configuration details needed to run the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
28

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The research does not use LLMs as part of the method development contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non nta-standard components.
Please refer to our LLM policy //neur cc. cc/Connferencee/22255////// for what should or should not be described.
29


Reviews:
[{"review_id": "qcaaybUbvw", "reviewer": "Reviewer_cDHW", "summary": "This paper studies offline safe reinforcement learning (RL) with a constrained MDP notion of safety. The goal is to maximize the reward while keeping the associated cost of transitions under a certain safety budget. An algorithm is proposed that solves the Lagrangian objective using two approaches:\n\n- an offline RL oracle that optimizes over the given data using adjusted returns that include both reward and cost with current Lagrange multiplier values\n- a no-regret algorithm for optimizing the Lagrangian multiplier that is implemented interestingly as a discretization of the multiplier value solved using a bandit approach\n\nThe result shows that on safe RL benchmark, the proposed approach does well always leading to safe policies that get high returns. The algorithm is also robust to choice of offline RL oracle", "questions": "- Is data distribution assumed to have good coverage? What assumptions will the offline RL oracle need to guarantee the requirements in Definition 2.\n\n- Why is the value function need to be estimated? There is no mention of this in Definition 3. Also, it is stated in line 200 that \"approximation of the value function of the optimal policy\" but then Equation 6 shows it is estimation of the policy $\\pi$. Shouldn't you have $E[\\tilde{V}] \\approx V^\\star_f$.\n\n- What is a \"stochastic approximation\"?\n\n- It seems to me that the entire contribution of the paper is to set the value of the Lagrange multiplier and then call the offline RL algorithm. If yes, then how will a much simpler approach do which simply calls an offline RL algorithm with the Lagrange multiplier treated as a single scalar hyperparameter, and then setting this hyperparameter by doing runs on a held-out set of online runs. It is possible that iterative learning with averaging is needed, but this baseline should have been evaluated.\n\n- Related work: Provable Safe Reinforcement Learning with Binary Feedback, Bennett et al. AISTATS 2023.\n\n\nMinor:\n\n- Broken link on line 186\n- Citations should be in parentheses at several places such as Line 209", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Safe RL is an important problem. \n- The proposed approach does make sense\n- Experiments are good\n\n**Weakness**\n\n- Parts of the paper, especially the theoretical part, do not make full sense to me. Firstly, there is no mention of data distribution coverage or use of pessimism, the latter of which is common in offline RL literature. The offline RL oracle assumption seems particularly strong. It is essentially assuming that given the data you have, you can learn an optimal policy with high probability. This is not true in general unless you assume concentrability or other notions of good data coverage. Alternatively, most offline RL approaches assume a data distribution that only covers the optimal policy and use pessimism. \n\n- Writing is unclear. The paper talks about how to approximate value function of the optimal policy without telling why is this needed. The oracle for Lagrange multiplier in Definition 3 does not seem to take that as input although later it is used in Line 4, Algorithm 2.\n\n- It is unclear how another approach will do that, simply runs the offline RL algorithm, treating lambda as a hyperparameter and tuning it on some dev set. It seems the core contribution is just to find the right value of this scalar.", "was_revised": true, "final_justification": "The authors have provided clarification, and so I have increased the score.", "rebuttal": "Thanks for your constructive feedback and review comments. We address them below.\n\n**Re: “clarifications on theoretical part: offline RL oracle assumption and data coverage”**\n\nRemark 1 (Lines 158-170) clarifies that offline RL oracle absorbs the data coverage condition.\n\nPessimism is an algorithmic approach that an offline RL oracle might employ to have a good estimation error rate.\n\nThe assumption of  offline RL oracle is to abstract away many state-of-the-art off the shelf offline RL algorithms. Any offline RL algorithm that returns a near-optimal policy, either in theory or practice, belongs to the oracle by definition. In the rich offline RL literature for both theory and practice, many choices for good offline RL algorithms exist (e.g., TD3BC and IQL). More broadly the idea of assuming oracles in theoretical work is widely used as it allows for the theoretical problems to be decomposed into pieces where independent progress can be made. In summary, we provide a reduction approach to leverage powerful offline RL algorithms to effectively solve offline safe RL problems.\n\n**Re: “unclear writing: approximate value function and parts of Algorithm 2”**\n\nThe approximate value function (Line 3, Algorithm 2) is a function of $\\lambda_t$ and the offline data. So the oracle for Lagrange multiplier in Definition 3 does not explicitly rely on the approximate value function.\n\n**Re: “alternative approach: offline RL with lambda as a hyper-parameter”**\n\nNote that in our framework the term \"online\" refers to the use of online/streaming optimization algorithms (e.g. bandits) for optimizing parameters, here Lagrange multipliers. This is different from how \"online\" is used in the context of online RL, where algorithms are allowed to run the policies online in the environment. Our work is within offline RL.\n\nWe conducted the experiment that the reviewer suggested, where we fix the value of $\\lambda$ throughout training and evaluate the resulting policy at the end. This setup violates the spirit of offline RL, as it assumes  we can execute policies in the environment to collect validation data for selection of $\\lambda$. We are aware that some works tune key hyperparameters this way, but it remains inconsistent with the offline RL setting. Moreover, we cannot exhaustively search the entire $\\Lambda$ space. \n\nIn our method (O3SRL), we define a discrete set of $\\lambda$ values: $\\lambda \\in lambdas=[0, 0.4061, 0.9081, 2.0306, 5.0]$. Each value corresponds to an \"arm\". During training, we sample a $\\lambda$ from a learned distribution over these arms and use it to shape the reward. This distribution is updated at each batch following EXP3 logic, allowing the policy to adaptively mix behaviors instead of committing to a single $\\lambda$.\n\nFor the experiment, we train five separate policies, each with a fixed $\\lambda=lambdas[i]$ (arms 0 through 4). No updating or mixture is used in these runs. The table below compares their final performance with O3SRL on the DroneCircle environment. Results are averaged over 3 seeds. We report normalized rewards and normalized costs (i.e., total cost divided by the budget) for three budget levels: 5, 20, and 40.\n### DroneCircle Results\n| cost limit          | Arm 0 | Arm 1 | Arm 2 | Arm 3 | Arm 4 | O3SRL |\n|---------------------|-------|-------|-------|-------|-------|--------|\n| **5 - (R)** | 0.90  | 0.86  | 0.55  | 0.47  | 0.39  | 0.49   |\n| **5 - (C)**   | 18.18 | 16.52 | 2.47  | 0.00  | 0.00  | 0.23   |\n| **20 - (R)**| 0.90  | 0.87  | 0.75  | 0.49  | 0.40  | 0.53   |\n| **20 - (C)**  | 4.54  | 4.33  | 2.80  | 0.04  | 0.00  | 0.36   |\n| **40 - (R)**| 0.90  | 0.87  | 0.82  | 0.52  | 0.42  | 0.65   |\n| **40 - (C)**  | 2.27  | 2.21  | 1.88  | 0.08  | 0.00  | 0.85   |\n\nTakeaways:\n- Arm 0 (λ=0) obviously ignores cost and violates all constraints.\n- Arms 1 and 2 fail to enforce the constraints reliably since the penalty is low.\n- Arms 3 and 4 stay within budget but sacrifice reward, with Arm 4 being most conservative.\n- O3SRL outperforms all fixed-$\\lambda$  policies by dynamically adjusting $\\lambda$, achieving better reward and cost trade-offs.\n\n**Re: “Is data distribution assumed to have good coverage? What assumptions will the offline RL oracle need to guarantee the requirements in Definition 2”**\n\nYes, we assume good data coverage. We already discussed this in Remark 1 (Lines 158-170). The assumptions needed by offline RL oracle to guarantee requirements in Def 2 include the ones we have used in theory of offline RL: good data coverage and good representation assumptions. The offline RL algorithms might use the pessimism principle to implement offline RL oracle with a good estimation error rate. All these are abstracted away and inherently embedded in the offline RL oracle. \n\n**Re: “Why is the value function need to be estimated? There is no mention of this in Definition 3. Also, it is stated in line 200 that \"approximation of the value function of the optimal policy\" but then Equation 6 shows it is estimation of the policy $\\pi$”**\n\nWe employ value-based methods and thus, the value function needs to be estimated. One can also opt for model-based methods to get rid of value function estimation, at the cost of using more complex models.\n\n**Re: \"What is a \"stochastic approximation\"?\"**\n\nIt was a typo. It should be “the stochastic approximation of the value function of the current policy (distribution)”. \n\n**Re: “Related work: Provable Safe Reinforcement Learning with Binary Feedback, Bennett et al. AISTATS 2023”**\n\nThis paper is considering a very different framework with very different assumption. First, the paper is in the online RL setting whereas we are in the offline setting. Second, this  paper assumes an offline oracle that provides binary feedback on the safety of state-action pairs. In our setting, such a safety signal is not available, instead there is a cost associated with each state-action pair. Third, the nature of our offline RL oracle is also fundamentally different from the oracle in this paper: their oracle provides safety signal, whereas our oracle abstracts away standard offline RL algorithms. \n\n**Minor**\n\nThank you for pointing this out. We will fix those in the final paper."}, {"review_id": "Zk0yaWWQyZ", "reviewer": "Reviewer_zh33", "summary": "This paper proposes to solve the problem of offline safe RL by combining offline RL and online optimization. The incorporation of the cost into the reward enables the use of offline RL, while discretizing the value of the Lagrange multiplier enables the use of multi-armed bandit algorithms, eliminating the need for estimating off-policy value functions. Theoretical results are provided to show the optimality of the proposed framework, and modifications are made upon the framework to make it practically usable. Empirical evaluation on a set of benchmark tasks showcases the effectiveness of the proposed method.", "questions": "1. Line 286 concludes that higher values of K leads to diminishing returns, but from Table 1 the trend is the opposite.\n2. Why does discretizing the Lagrange multiplier help stabilize learning?\n3. Could you provide a more intuitive explanation of how the EXP3 algorithm updates the distribution over arms? Although this is not the contribution of this work, I think an intuition of how it works to avoid OPE can be insightful to readers. Maybe putting it in the Appendix would be a good choice.\n4. Table 1: Why does increasing the granularity reduce the safety of the agent? How can interpret this empirical result? I couldn't see an intuitive causal relationship between this hyperparameter and the safety performance.\n5. Table 3: \n   - Does the results mean that the agent becomes closer to the safety budget when the budget is higher? Shouldn't the normalized cost stay relatively the same regardless of the budget? \n   - Will it finally reach or even violate the constraint when the budget is high enough? \n   - Line 331 says approaching the budget can \"make better use of the available budget\", does this mean that the results of O3SRL in Table 2 can still be improved since the normalized cost is mostly close to 0?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The flow of this paper is clear and easy to follow.\n- The incorporation of the cost into the reward enables the use of well-developed offline RL algorithms, which is novel in offline safe RL as far as I know.\n- The discretization of the Lagrange multiplier is novel and empirically effective. Combined with the EXP3 algorithm, this eliminates the need for estimating the off-policy values which could be erroneous.\n- The empirical results are pretty promising, especially in terms of the safety metric, which is desirable for offline safe RL algorithms.\n\n\n\n### Weaknesses\n\n- The number and variety of the tasks evaluated on can still be improved. A more comprehensive evaluation like the one in the FISOR paper \\[1\\] would make the comparative results much more convincing.\n- Missing related work (and probably a baseline): TREBI \\[2\\], which serves as a baseline in the FISOR paper.\n\n#### Minors\n\n- Line 86: missing in-text reference.\n\n\n**References**\n\n\n\\[1\\]: Zheng et al., 2024, \"Safe offline reinforcement learning with feasibility-guided diffusion model\".\n\n\\[2\\]: Lin et al., 2023, \"Safe offline reinforcement learning with real-time budget constraints\".", "was_revised": true, "final_justification": "I raised a few concerns about the evaluation and related work, and the authors addressed them sufficiently. So I decided to raise my confidence in giving this paper a high score.", "rebuttal": "Thanks for your constructive feedback and review comments. We address them below.\n\n**Re: “The number and variety of the tasks evaluated on can still be improved. Missing related work (and probably a baseline).”**\n\nWe have extended our evaluation by running O3SRL on additional environments from the Gymnasium Safety Suite, following the FISOR setup with a tight cost limit of 10. Results are summarized in the table below. (R) is the normalized reward and (C) is the normlaized cost (<1 means safe) (costs are not normalized in the last table).\n\nRegarding TREBI [2], we will add it to the related work section. We attempted to include it as a baseline, but faced persistent issues with installing its dependencies and setting up the environment correctly. Unlike more recent baselines such as CAPS and CCAC [3], TREBI predates the standardized OSRL baselines framework, which makes integration and reproducibility more difficult.\n\nAdditionally, both the FISOR and CCAC papers report that TREBI underperforms compared to their approaches. We opted to include CCAC as an additional baseline in our experiments. We also already compare against CAPS, a recent method with strong performance. Both CCAC and CAPS are cost-limit agnostic, similar to TREBI, and can adapt across different budget levels.\n\n[3] Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning. https://openreview.net/pdf?id=nrRkAAAufl\n\n| Environment           | CAPS         | FISOR         | CCAC          | O3SRL         |\n|-----------------------|--------------|---------------|---------------|---------------|\n| **PointCircle1 (R)**  | 0.31 ± 0.05  | 0.43 ± 0.05   | 0.57 ± 0.08   | 0.53 ± 0.02   |\n| **PointCircle1 (C)**  | 0.94 ± 1.53  | 14.93 ± 4.24  | 6.65 ± 3.51   | 1.31 ± 0.62   |\n| **PointCircle2 (R)**  | 0.44 ± 0.06  | 0.76 ± 0.05   | 0.03 ± 0.75   | 0.52 ± 0.04   |\n| **PointCircle2 (C)**  | 0.10 ± 0.09  | 18.02 ± 4.17  | 3.99 ± 4.30   | 0.55 ± 0.41   |\n| **AntVelocity (R)**   | 0.87 ± 0.04  | 0.89 ± 0.01   | -1.01 ± 0.00  | 0.45 ± 0.43   |\n| **AntVelocity (C)**   | 0.19 ± 0.06  | 0.00 ± 0.00   | 0.00 ± 0.00   | 0.05 ± 0.06   |\n| **HalfCheetah (R)**   | 0.86 ± 0.01  | 0.89 ± 0.01   | 0.91 ± 0.04   | 0.93 ± 0.02   |\n| **HalfCheetah (C)**   | 0.27 ± 0.07  | 0.00 ± 0.00   | 0.97 ± 0.12   | 0.00 ± 0.00   |\n| **HopperVelocity (R)**| 0.29 ± 0.14  | 0.12 ± 0.03   | -0.01 ± 0.02  | 0.51 ± 0.31   |\n| **HopperVelocity (C)**| 1.61 ± 1.53  | 0.90 ± 1.07   | 0.00 ± 0.00   | 0.20 ± 0.35   |\n| **SwimmerVelocity (R)**| 0.38 ± 0.10 | -0.02 ± 0.05  | 0.06 ± 0.25   | 0.05 ± 0.02   |\n| **SwimmerVelocity (C)**| 2.11 ± 3.01 | 0.23 ± 0.20   | 0.91 ± 1.57   | 0.00 ± 0.01   |\n| **Walker2dVelocity (R)**| 0.78 ± 0.01| 0.51 ± 0.03   | 0.40 ± 0.17   | 0.77 ± 0.02   |\n| **Walker2dVelocity (C)**| 0.08 ± 0.07| 1.92 ± 0.57   | 5.20 ± 1.16   | 0.02 ± 0.02   |\n\nO3SRL satisfies the cost constraint on 6 out of 7 additional tasks, demonstrating strong generalization across environments. The next best method, CAPS, is safe on 5 out of 7. In terms of performance, O3SRL achieves the highest reward on 3 tasks, the most among all methods. Additionally, it remains competitive with the best-performing method on SwimmerVelocity (0.05 vs. 0.06) and Walker2dVelocity (0.77 vs. 0.78), while maintaining safety.\n\nThe table below compares O3SRL and CCAC on the Bullet Gym environments, as reported in Table 2 of the main results.\n| Environment        | CCAC (Mean ± Std) | O3SRL (Mean ± Std) |\n|--------------------|-------------------|---------------------|\n| **BallRun (R)**     | 0.31 ± 0.01       | 0.25 ± 0.03         |\n| **BallRun (C)**     | 0.00 ± 0.00       | 0.00 ± 0.00         |\n| **CarRun (R)**      | 1.82 ± 0.84       | 0.96 ± 0.01         |\n| **CarRun (C)**      | 24.57 ± 20.94     | 0.02 ± 0.03         |\n| **DroneRun (R)**    | 0.50 ± 0.08       | 0.32 ± 0.05         |\n| **DroneRun (C)**    | 16.29 ± 9.35      | 0.68 ± 1.18         |\n| **AntRun (R)**      | 0.03 ± 0.10       | 0.33 ± 0.13         |\n| **AntRun (C)**      | 0.00 ± 0.00       | 0.14 ± 0.10         |\n| **BallCircle (R)**  | 0.47 ± 0.38       | 0.62 ± 0.01         |\n| **BallCircle (C)**  | 10.19 ± 17.65     | 0.06 ± 0.07         |\n| **CarCircle (R)**   | 0.70 ± 0.01       | 0.66 ± 0.03         |\n| **CarCircle (C)**   | 1.61 ± 0.51       | 0.11 ± 0.16         |\n| **DroneCircle (R)** | 0.43 ± 0.04       | 0.49 ± 0.07         |\n| **DroneCircle (C)** | 0.06 ± 0.07       | 0.23 ± 0.34         |\n| **AntCircle (R)**   | 0.49 ± 0.07       | 0.48 ± 0.06         |\n| **AntCircle (C)**   | 0.02 ± 0.03       | 0.00 ± 0.00         |\n\nO3SRL satisfies the cost constraint on 8 out of 8 tasks, while CCAC is safe on only 4. Additionally, O3SRL achieves the highest reward on 6 out of the 8 tasks.\n\n**Re: “Line 286 concludes that higher values of K leads to diminishing returns, but from Table 1 the trend is the opposite.”**\n\nThank you for pointing this out. The sentence was not phrased well. Our intended meaning was that increasing K beyond a certain point (e.g., from 5 to 10) provides relatively smaller gains compared to the improvement seen when increasing from K=2 to K=5.\n\n**Re: “Why does discretizing the Lagrange multiplier help stabilize learning?”**\n\nIf we optimize for continuous Lagrange multiplier, we generally need Off-Policy Evaluation (OPE) procedures on top of the offline RL oracle. OPE is known to be typically unstable especially when the target policy, found as the return of the offline RL oracle, keeps changing during the iterative process. By discretizing $\\lambda$, we completely avoid the need for OPE.\n\n**Re: “Could you provide a more intuitive explanation of how the EXP3 algorithm updates the distribution over arms?”**\n\nEXP3 maintains a distribution over arms, where the mass is biased toward the arms that have been observed to achieve higher rewards in the past rounds. As per your suggestion, we will provide more intuition in the main paper and detailed description in the Appendix.\n\n**Re: “Table 1: Why does increasing the granularity reduce the safety of the agent? How can interpret this empirical result?”**\n\nIn our setup, safety is defined as keeping the normalized cost below 1, i.e., staying within the given cost budget. From this perspective, all policies that respect the budget are considered safe.\nIncreasing granularity helps the agent discover better policies within the safe region. The goal is not to minimize cost to zero, but rather to maximize reward while staying under the cost limit. An ideal policy would use the available budget effectively to collect as much reward as possible.\nWhen the budget is tight (a limit of 5 means the agent is allowed only five mistakes over an entire trajectory), the agent may behave conservatively and avoid costs entirely, even if using some of the budget could improve reward.\n\n**Re: “Table 3: questions on results”**\n\nAll reported costs are normalized (i.e., divided by the cost limit). For example, a normalized cost of 0.5 under a cost limit of 20 corresponds to an actual accumulated cost of 10. The goal is to stay below 1.0, and the percentage of budget used is not inherently important as long as this condition is met.\n\nGiven more budget, the agent accumulates more cost in absolute terms, which might not be fully reflected in the normalized cost values. In the table below, we report the non normalized costs.\nThe agent typically leaves a safety margin and avoids pushing right up to the limit. This is intentional. Since offline RL uses fixed data and faces distribution shift, the agent must be pessimistic to avoid violating the constraint during evaluation. A slightly conservative policy is preferable to one that risks infeasibility.\n\nRegarding Table 2: yes, the normalized costs are mostly close to 0, but this is because those results use a tight budget of 5, and the agent learns to avoid costs as much as possible to remain safe across episodes (200–500 steps). In that regime, even a few binary cost events can be enough to exceed the limit.\nWe include additional experiments with a larger cost limit of 80 below, where the agent accumulates more costs while still remaining safe. This supports the observation that O3SRL adapts based on the constraint and trades off conservatism appropriately.\n\n### O3SRL Performance with Unnormalized Costs\n| Environment   | Cost Limit 5 (R) |  Cost Limit5 (C) |  Cost Limit 20 (R) |  Cost Limit 20 (C) |  Cost Limit 40 (R) |  Cost Limit 40 (C) |  Cost Limit 80 (R) |  Cost Limit 80 (C) |\n|---------------|------------|------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| DroneRun      | 0.32       | 3.40        | 0.38        | 8.20        | 0.38        | 17.60       | 0.42        | 30.08       |\n| BallCircle    | 0.62       | 0.30        | 0.69        | 11.60       | 0.76        | 25.60       | 0.81        | 38.32       |"}, {"review_id": "8fu2gqNG5M", "reviewer": "Reviewer_AJWj", "summary": "The authors propose O3SRL, a training framework for offline safe RL. O3SRL builds on strong duality of the Lagrangian to solve the constraint optimization problem, by iteratively solving a cost penalized MDP and then updating the Lagrange multiplier with a no-regret “online” learning algorithm. The authors highlight an inherent limitation of offline policy evaluation that is required in their theoretical framework (used to update the Lagrange multiplier by the no-regret algorithm). To address this limitation practically, they discretize the Lagrange multiplier and place a multi-armed bandit to pick it, without directly evaluating policies offline.\nFinally, the authors demonstrate the performance of O3SRL on several offline safe RL tasks, and show that their method outperforms existing baselines, especially when the budget for the constraint is tight.", "questions": "No questions.", "limitations": "I believe that the main limitation of the proposed work is the practicality of the proposed algorithm:\n* Using a bandit to pick the Lagrange multiplier will require practitioners to implement an additional (non-trivial) subroutine.\n* The algorithm returns a “mixing” policy that combines several policies together and draw them randomly, as common in previous works that propose primal-dual algorithms to solve CMDPs. This is usually a limitation in practice, because such mixing policies are not easy to implement, and might execute severely unsafe actions.\n\nWhile the use of a bandit limits the practicality of O3SRL, I believe that this is still a useful contribution.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* The empirical evaluation is comprehensive and adequate. The authors show that their method outperforms many existing baselines.\n* The use of EXP3 to circumvent offline policy evaluation is novel and interesting.\n\n## Weaknesses\n* The proposed framework and theory has been studied before in [1]. I couldn't find any reference for that.\n* While providing convergence guarantees (Theorem 1 & 2) adds value to the paper, I believe that showing the conditions for constraint satisfaction upon deployment, after training with offline data, would give a more interesting theoretical insight to readers, and would strengthen a lot the paper. This is more so given that the main theoretical argument that is used in the paper has already been shown by [1].\n* Some theoretical claims are not precise or not rigorous enough:\n  * L.128: the objective and constraints are generally linear only when there is no (non-linear) function approximation involved; this is usually not the case in many practical problems and even in your own experiments. The implication here is that, formally speaking, strong duality does not necessarily hold in many of these problems.\n  * L. 129: Slater's condition is typically assumed, i.e., Equation 1 not always satisfies it.\n  * Remark 1, L. 162-164, “notions of coverage depend very mildly on the reward functions, if at all”. I don't think that this statement is true. What about sparse reward tasks? Consider for example the sparse reward tasks in [2].\n* While the use of the bandit algorithm is novel, and seem to yield strong results, the practicality of it is somewhat limited.\n\n## Minor\n* Using only three seeds for evaluation should be improved.\n\n## Presentation\n* I believe that the empirical result is more significant in this paper and should receive more focus.\n* The significance of low budget constraints should be further motivated and highlighted.\n* Consider presenting your empirical results in a plot instead of a table---it is much easier the parse and could convey your empirical result better.\n\n[1] Le, Hoang, Cameron Voloshin, and Yisong Yue. \"Batch policy learning under constraints.\" In International Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.\n\n[2] Nair, A., Gupta, A., Dalal, M., and Levine, S. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv, June 2020.", "was_revised": true, "final_justification": "The method seems to be empirically strong. The use of MAB is novel and interesting.\n\nI still have some concerns regarding the rigorousness of the presentation, however, since the main contribution of this work is not theoretical, I think this is a minor limitation.", "rebuttal": "Thanks for your constructive feedback and review comments. We address them below.\n\n**Re: “The proposed framework and theory has been studied before in [1]”**\n\nThanks for pointing out this paper. We will discuss [1] and its connections in the final paper. \n\nLagrangian formulation for offline safe RL is quite standard that both our work and [1] share. [1] and our work also share the general idea that we can use online optimization algorithms to tune the Lagrangian multiplier. The critical difference is in the design of the algorithmic frameworks: our framework requires one call to an offline RL oracle per loop and does not rely on any OPE oracle. On the other hand, [1] requires two calls to an offline RL oracle and four calls to an OPE oracle per loop. Since OPE procedures are known to be typically unstable, we believe that our O3SRL framework has unique advantages.\n\n**Re: “convergence guarantees with conditions for constraint satisfaction after deployment…main theoretical argument that is used in the paper has already been shown by [1]”**\n\nThe solutions that our algorithms converge to (at the rate specified by Theorem 1 and 2) satisfy the cost constraint for deployment, by the very definition of the Langragian formulation in Eq. 3. The nature of our theoretical argument is completely different from those in [1]: our framework works for a general offline RL oracle, allowing us to integrate any state-of-the-art off the shelf offline RL algorithm and doesn’t need OPE procedures, whereas [1] only uses FQI and relies on OPE procedures, which are known to be unstable. \n\n**Re: “L.128: the objective and constraints are generally linear only when there is no (non-linear) function approximation involved”**\n\nThe Lagrangian formulation in Section 4.1 has nothing to do with function approximation. When the policy class $\\Pi$ is expressive enough, Slater’s condition easily holds. The linearity we exploit is the linearity of the objective with respect to λ.\n\n**Re: “L. 129: Slater's condition is typically assumed, i.e., Equation 1 not always satisfies it.”**\n\nWhen the policy class $\\Pi$ is expressive enough, Slater’s condition easily holds.\n\n**Re: “Remark 1, L. 162-164, “notions of coverage depend very mildly on the reward functions, if at all”. I don't think that this statement is true. What about sparse reward tasks?”**\n\nWe refer the reviewer to our explanation in Lines 164-167 where we say “most notions of coverage”, not “all notions of coverage”. Even in sparse reward tasks, we still believe the statement applies – though the extremity of sparse rewards would place much emphasis on the influence of a particular (sparse) reward function to the data coverage.\n\n**Re: “While the use of the bandit algorithm is novel, and seem to yield strong results, the practicality of it is somewhat limited.”**\n\nWe use the multi-arm bandit approach because of the practicality it offers by avoiding the reliance on off-policy evaluation procedures. We demonstrate its practicality through experiments on a standard suite of benchmarks.\n\nNote that the bandit algorithm is used only during offline training. The final output is a standard policy that maps states to actions and can be deployed directly. No bandits or additional machinery are involved at inference time. The bandit component is used solely during training to embed the desired behavior into the final policy.\n\n**Re: “Suggestions for presentation and experimental results”**\n\nWe will adjust the focus to better highlight the empirical results and the significance of low-budget constraints. Initially, space limitations influenced our choice of presentation. We also note that, per NeurIPS recent guidelines, we are not allowed to share external visualizations or link to supplemental PDFs during the review process. However, we agree that plots can improve clarity.\n\n**Re: “Using a bandit to pick the Lagrange multiplier will require practitioners to implement an additional (non-trivial) subroutine”**\n\nThe subroutine of multi-arm bandit (MAB) is standard. MAB methods are quite commonly used in practice for adaptive decision-making applications (e.g., A/B testing). The simplicity is apparent in the pseudo-code we provide, which shows this is a very small wrapper around an offline RL algorithm.\n\nNote that the bandit component is used only during training to guide the learning process. After training, O3SRL produces a single policy that maps states to actions and can be deployed directly, with no bandits or additional components involved at inference time.\n\n**Re: “The algorithm returns a “mixing” policy that combines several policies together and draw them randomly. This is usually a limitation in practice, because such mixing policies are not easy to implement, and might execute severely unsafe actions.”**\n\nThe mixture policy is only used for the theoretical analysis. In practice, we find that a single policy from the last training iteration is sufficient for strong results (see Lines 231–234 for the concrete approach used in evaluation). The final output is a standard policy that embeds the effect of the mixture during training. Although we use a mixture of shaped rewards based on sampled \nλ values during training, the resulting policy maps states to actions directly and requires no runtime mixing or sampling.\n\n**Re: “Using only three seeds for evaluation should be improved.”**\n\nWe agree that more seeds can help provide a more reliable estimate of performance. To address this, we include results from three additional seeds (40, 50, and 60) on top of the original 10, 20, and 30.\nColumns 1 and 2 report the rewards and costs (mean ± standard deviation) over the original 3 seeds. Columns 3 and 4 show the same metrics computed over all 6 seeds.\n\nWe note that using three seeds with 20 evaluation episodes per seed is the standard practice in OSRL, as followed in prior works. Nonetheless, we appreciate the suggestion and have included the extended results for completeness.\n\n### Comparison over 3 Seeds vs. 6 Seeds\n| Environment   | Reward (3 seeds) | Cost (3 seeds) | Reward (6 seeds) | Cost (6 seeds) |\n|---------------|------------------|----------------|-------------------|----------------|\n| BallRun       | 0.25 ± 0.03      | 0.00 ± 0.00    | 0.26 ± 0.02       | 0.00 ± 0.00    |\n| CarRun        | 0.96 ± 0.01      | 0.02 ± 0.03    | 0.96 ± 0.01       | 0.00 ± 0.00    |\n| DroneRun      | 0.32 ± 0.05      | 0.68 ± 1.18    | 0.37 ± 0.03       | 0.74 ± 0.67    |\n| AntRun        | 0.33 ± 0.13      | 0.14 ± 0.10    | 0.22 ± 0.10       | 0.17 ± 0.19    |\n| BallCircle    | 0.62 ± 0.01      | 0.06 ± 0.07    | 0.63 ± 0.04       | 0.26 ± 0.25    |\n| CarCircle     | 0.66 ± 0.03      | 0.11 ± 0.16    | 0.66 ± 0.03       | 0.04 ± 0.10    |\n| DroneCircle   | 0.49 ± 0.07      | 0.23 ± 0.34    | 0.47 ± 0.03       | 0.02 ± 0.04    |\n| AntCircle     | 0.48 ± 0.06      | 0.00 ± 0.00    | 0.44 ± 0.05       | 0.02 ± 0.03    |\n\nThe results across 6 seeds confirm the stability of O3SRL, with consistent rewards and safe cost levels across tasks. Variance remains low, and performance trends observed with 3 seeds largely hold, indicating robust and reliable behavior."}]

Author rebuttal:

Meta-review: The paper studies offline safe RL: each transition in the environment yields both a cost and a reward, and the goal is to have expected total discounted cost below a threshold while maximizing expected total discounted reward. We are only allowed to observe a dataset of trajectories collected ahead of time — i.e., we can’t try out our new policy to collect more data or make sure it is safe and rewarding.

The paper designs a wrapper that repeatedly calls an ordinary offline RL subroutine as a black box (i.e., an ORL subroutine with no consideration of cost/safety constraints). By editing the rewards in our training data to penalize unsafe behavior to varying degrees, the wrapper explores different policies; via no-regret learning, it guarantees convergence to a policy that is estimated to satisfy the safety constraint. (This kind of wrapper design is classical: the earliest wrapper modifying cost/reward that I know of is Dantzig-Wolfe-Benders decomposition from the 1960s, and the earliest I know of using no-regret is in RL and online convex programming research in the 1990s; boosting (1980s) is a no-regret wrapper design for a different problem.) The exact behavior of the combined method depends on the ORL subroutine: e.g., different ORL methods might be more or less conservative about returning a policy whose reward is imperfectly known due to coverage problems in the offline data, and different ORL methods might need different assumptions about data collection in order to work. An advantage of the wrapper construction is generality: we delegate such choices to the ORL subroutine.

The experiments verify that the wrapper (wrapped around an existing ORL method from the literature) is able to uncover safe policies in a variety of RL domains. The results show a better tradeoff than competing methods: higher reward while still maintaining safety. The need to call the ORL subroutine multiple times increases the computational cost, but in many domains, trajectories are much more expensive than computation, so this tradeoff can be worthwhile. (But a full exploration of scalability is left to future work, so the conclusion of a worthwhile tradeoff is provisional.)

The key contribution is the design of the wrapper to be both theoretically and practically sound. One of the key obstacles to be overcome was that there is necessarily high variance in value estimates based on offline data: ORL needs to worry about this for comparing policies, and safe RL needs to worry about this for satisfying the cost constraint. The paper mitigates this concern by switching to a multi-armed bandit formulation, which only needs to observe the value (under the safety-modified rewards) of the single policy selected by the ORL subroutine on each iteration. Unfortunately this aspect needs much better explanation in the final paper: several of us on the review team are RL experts, the authors were responsive during discussion, and we still had trouble following this argument. Key points to clarify:
* The paper repeatedly mentions that OPE is difficult and high variance (true). But the paper still relies on the ORL subroutine’s estimate of the value of its returned policy. Naively, estimating this value is the same problem as OPE, but the paper claims an advantage by using it instead of OPE. Worse, what we need to estimate in OPE is actually the exact same policy, just with a different set of rewards — an even more similar problem. Reading between the lines, the conservatism inherent in most ORL methods would tend to make the value estimate for the returned policy under the passed-in reward structure more reliable than estimates for other policies or reward structures — is this what is being taken advantage of?
* To get to MAB, the paper discretizes lambda. The paper states that continuous lambda would require OPE, but this isn’t obvious: yes, it would require OPE if we treat the learning problem as having infinitely many unrelated arms, one for each lambda value. But that ignores structure: why not treat it as an online convex programming problem, for which we just need a subgradient at the current value of lambda (which we can get from the output of the ORL subroutine, just as we do for the MAB).

Despite the above clarity problem, the paper’s contributions are strong: it’s an interesting problem, framed well, with a novel (though classically inspired) solution that seems to work well in experiments. We hope that the final version of the paper will take the above questions into account, as well as the other results of the reviewer-author discussion. (I (the area chair) have posted a few final suggestions in the review thread.) We expect that these updates will lead to an excellent final paper.
Decision: Accept (poster)