Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
Authors: Honghao Wei
Xiyue Peng
Arnob Ghosh
Xin Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: RL
safe-RL
Offline RL
Primary area: reinforcement_learning
TL;DR: We propose a novel algorithm for Safe Offline RL under functional approximation; our algorithm achieves robust safe policy improvement (first such result) using only single policy concentrability assumption.

Abstract: We propose WSAC (Weighted Safe Actor-Critic), a novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed as a two-player Stackelberg game to optimize a refined objective function. The actor optimizes the policy against two adversarially trained value critics with small importance-weighted Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, we demonstrate that when the actor employs a no-regret optimization oracle, WSAC achieves a number of guarantees: $(i)$ For the first time in the safe offline  RL setting, we establish that WSAC can produce a policy that outperforms {\bf any} reference policy while maintaining the same level of safety, which is critical to designing a safe algorithm for offline RL. $(ii)$ WSAC achieves the optimal statistical convergence rate of $1/\sqrt{N}$ to the reference policy, where $N$ is the size of the offline dataset. $(iii)$ We theoretically show that WSAC guarantees a safe policy improvement across a broad range of hyperparameters that control the degree of pessimism, indicating its practical robustness. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-art safe offline RL algorithms in several continuous control environments. WSAC outperforms all baselines across a range of tasks, supporting the theoretical results.

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
Honghao Wei
Xiyue Peng ShanghaiTech University pongxyy20244shaangatteee. tech edu. cn
Washington State University honghao wwi edu
Arnob Ghosh
Xin Liu
New Jersey Institute of Technology arnob. gwwshhnjit.eee.eoom edu
ShanghaiTech University Liuxin/@@hanghhattechh edu. cn
Abstract
We propose WSAC (Weighted Safe Actor-Critic), novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed refined objective function. The actor optimizes the policy against two adversarially trained value critics with small apportance-weightee Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, demonstrate that when the actor employs no-regret optimization oracle, WSAC achieves number of guarantees: (i) For the first time in the safe offline RL setting, we establish that WSAC produce policy that outperforms any reference policy while maintaining the same level of safety, which is critical designing safe algorithm for offline RL. (fi) WSAC achieves the optimal statistical convergence rate of 1/vN the reference policy, where N is the size of the offline dataset. (iii) We theoretically show that WSAC guarantees a safe policy improvement across a broad range of hyperparameters that control the degree pessimism, indicating its practical robustness. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-ar safe offline RL algorithms in several continuous control environments. WSAC outperforms all baselines across a range of tasks, supporting the theoretical results.
two-player Stackelberg game to optimize
Introduction
Online safe reinforcement learning (RL) has found successful applications in various domains, such as autonomous driving (Isele et al., 2018), recommender systems (Chow al., 2017), and robotics (Achiam et al., 2017). It enables the learning of safe policies effectively while satisfying certain safety constraints, including collision avoidance, budget adherence, and reliability. However, collecting diverse interaction data can be extremely costly and infeasible in many real-world applications, and this challenge becomes even more critical scenarios where risky behavior cannot be tolerated. Given the inherently risk-sensitive nature of these safety-related tasks, data collection becomes feasible only when employing behavior policies satisfies the safety requirements.
To overcome the limitations imposed by interactive data collection, offline RL algorithms are designed learn a policy from an available dataset collected from historical experiences by some behavior policy, which may differ from the policy we aim to learn. A desirable property of an effective offline algorithm the assurance of robust policy improvement (RPI), which guarantees that learned policy is always at least as good as the baseline behavior policies (Fujimoto et al., 2019; Laroche et al., 2019:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Kumar et al., 2019; Siegel et al., 2020; Chen et al., 2022a; Zhu et al., 2023; Bhardwaj et al., 2024). We extend the property of RPI to offline safe RL called safe robust policy improvement (SRPI), which indi- cates the improvement should be safe as well. This is particularly important in offline safe RI.. For ex- ample, in autonomous driving, an expert human driver operates the vehicle to collect diverse dataset under various road and weather conditions, serving as the behavior policy. This policy is considered both effective and safe, as it demonstrates proficient human driving behavior while adhering to all traf- fic laws and other safety constraints. Achieving a policy that upholds the SRPI characteristic with such a dataset can significantly mitigate the likelihood of potential collisions and other safety concerns.
In offline RL, we represent the state-action occupancy distribution of policy и over the dataset distribution using the ratio nd d /p.. A commonly required assumption is that the & concentrability CT is bounded, which is defined w" for all policies (Liu et al., 2019; Chen and Jiang, 2019; Wang et al., 2019; Liao et al., 2022; Zhang et al., 2020). A stronger assumption requires a uni- form lower bound on u(a(s) (Xie and Jiang, 2021). However, such all-policy concentrability assumptions are difficult satisfy in practice, particularly for of- fline safe RL, as they essentially require the offline dataset to have good coverage of all unsafe state- action pairs. To address the full coverage require- ment, other works (Rushidinejad
the infinite norm of
690
20% 100
PS00 parseroge
Figure Comparison between WSAC and the behavior policy in the tabular case. The behavior policy is mixture of the optimal policy and a random policy, with the mixture percentage representing the proportion of the optimal policy. The cost threshold is set to 0.1. We observe that WSAC consistently en- sures a safely improved policy across various scenarios, even when the behavior policy is not safe.
al., 2021; Zhan et al., 2022; Chen and Jiang, 2022; Xie et al., 2021; Uehara and Sun, 2021) adapt conservative algorithms by employing the principle of pessimism in the face of uncertainty, reducing the assumption to the best covered policy (or optimal policy) concerning too concentrability. Zhu et al. (2023) introduce & con- centrability to further relax the assumption, indicating that for concentrability is always upper bound of & concentrability (see Table for detailed comparisons with previous works). While provable guarantees are obtained using single policy concentrability for unconstrained MDP as Table 1 suggests for the safe RL setting, all the existing studies (Hong et al., 2024; Le et al., 2019) still require the coverage all the policies. Further, as Table 1 suggests, the above papers do not guarantee robust safe policy improvement. Our main contributions are summarized below:
1. We prove that our algorithm. which uses average Bellman error, enjoys an optimal statistical rate of 1/ 1/VN under partial data coverage assumption. This is the first work that achieves such a result using only fingle-polccy & concentrability.
We propose a novel offline safe RL algorithm, called Weighted Safe Actor-Critic (WSAC), which can robustly learn policies that improve upon any behavior policy with controlled relative pessimism. We prove that under standard function approximation assumptions, when the actor incorporates no-regret policy optimization oracle, WSAC outputs a policy that never degrades the performance of a reference policy (including the behavior policy) for a range of hyperparameters (defined later). This is the first work that provably demonstrates the property of SRPI in offline safe RL setting.
3. We point out that primal-dual-based approaches Hong et al. (2024) may require all-policy concen- trability assumption. Thus, unlike, the primal-dual-base appraoch, we propose novel rectified penalty-based approach to obtain results using single-policy concentrability. Thus, we need novel analysis techniques prove results.
Furthermore, we provide a practical implementation of WSAC following a two-timescale actor- critic framework using adversarial frameworks similar to Cheng et al. (2022); Zhu et al. (2023), and test it on several continuous control environments in the offline safe RL benchmark (Liu et al., 2023a). WSAC outperforms all other ofte-offtheeart baselines, validating the property of safe policy improvement.
2

---

Table 1: Comparison of algorithms for offline RL (safe RL) with function approximation. The parameters C2.y Ci C Bellmen refer to different types of concentrabilities, always hold C S C% and under certain condition Cj I C Bellman detailed definitions and more discussions can be found in Section 3.3.
Coverage assumption
Policy Improvement
Algorithm
Safe RL
Suboptimality
Xie and Jiang (2021)
No
all policy, Cj2
Yes
O(1/VN)
Xie et al. (2021)
No
single-policy, C
Yes
O(1/VN)
Cheng et al. (2022)
No
single-policy, Bellman
Yes & Robust
O(1/N1/3)
Ozdaglar et al. (2023)
No
single-policy, C
No
O(1/VN)
Zhu et al. (2023)
No
single-policy, C
Yes & Robust
O(I/VN)
Le al. (2019)
Yes
all policy, C
No
0(1/VN)
Hong et al. (2024)
Yes
all policy, C%2
No
O(1/VN)
Ours
Yes
single-policy, Cj2
Yes & Robust
O(1/VN)
2
Related Work
Offline safe RL: Deep offline safe RL algorithms (Lee et al., 2022; Liu et al., 2023b; Xu et al., 2022;
Chen et al., 2021; Zheng et al., 2024) have shown strong empirical performance but lack theoretical guarantees. To the best of our knowledge, the investigation of policy improvement properties in offline RL literature. Wu et al. (2021) focus
offline safe RL is relatively rare in the statc-of-the- on the offline constrained multi-objective Markov Decision Process (CMOMDP) and demonstrate that an optimal policy can be learned when there is sufficient data coverage. However, although they show that CMDP problems can be formulated as CMOMDP problems, they assume linear kernel CMOMDP in their paper, whereas our consideration extends to a more general function approximation setting. Le al. (2019) propose a model-based primal-dual-type algorithm with deviation control for offline safe RL in the tabular setting. With prior knowledge of the slackness in Slater's condition and a constant on the concentrability coefficient, an (e, 8)PAC error is achievable
when the number of data samples N is large enough (N = ((1/2)). These assumptions make the algorithm impractical, and their computational complexity is much higher than ours. Additionally, we consider a more practical, model-free funetion approximation setting. In another concurrent work (Hong et al., 2024), primal-dual critic algorithm is proposed for offline-constrained RL settings with general function approximation. However, their algorithm requires l concentrability for all policies, which is not practical as discussed. The reason is that the dual variable optimization in their primal-dual design requires an accurate estimation of all the policies used in each episode, which necessitates coverage over all policies. Morcover, they cannot guarantee the property of SRPI. Moreover, their algorithm requires an additional offline policy evaluation (OPE) oracle for policy evaluation, making the algorithm less efficient.
Preliminaries
3.1 Constrained Markoy Decision Process
We consider Constrained Markov Decision Process (CMDP) M, denoted by (S,A, P.R. C. p). S is the state space, .A is the action space, P S x A + A(S) is the transition kernel, where A(-) a probability simplex, R S A + [0, 1] is the reward function, C S A + [-1, 1] is the cost function, 7 E [0. 1) is the discount factor and p: S - [0, 1) is the initial state distribution. We assume A finite while allowing to be arbitrarily complex. We use π S - A(A) to denote
stationary policy, which specifies distribution over actions for each state. At each time, the agent
3

---

observes state s, S, takes an action de E S according to policy т, receives reward rt and cost cr. where r; R(s,, at), C. = C(t,, a,). Then the CMDP moves the next state 8t+1 based on the transition kernel P(-| az). Given policy we use V. (s) and V (s) to denote the expected discounted return and the expected cumulative discounted cost of π, starting from state s, respectively.
(s)
B
rralso 0, (T1ss))
1=0 V
(8) = B Σ
7 7/0050 - a, (-11+))]-
(2)
1=0
Accordingly, we also define the Q value function a policy π for the reward and cost as
(s, = E
7 rre((so an) (8, a x-1sst)
==
(s,a) =
/cc on) (s, ~ x-11s))),
(4)
t=
respectively. As rewards and costs are bounded, have that 0 I O 1-7. and
1-7
1
Qº
S
T We let Vmax - to simplify the notation. We further write
(T) = (1 (V, Je(T) = (1 1) VV.
to represent the normalized average reward/cost value of policy и. In addition, d= to
denote the normalized and discounted state-action occupancy measure of the policy x
([s,a):= 7)
1( = =s = =a]aa, T-|ss)))
t=
where (-) is the indicator function. We also d= d=((s) =
- Enc d= (s, a) to denote the discounted state occupancy and F shorthand of (s. [.] Eywa [-]to denote the expectation with respect to d= Thus The objective in safe RL. for an agent is to find a policy such that
π E arg max (x) s.t. Je(T) S
(5)
Remark 3.1. For ease of exposition, this paper exclusively focuses single constraint. However, it is readily extendable to accommodate multiple constraints.
3.2 Function Approximation
In complex environments, the state space S is usually very large even infinite. We assume access to a policy class II C (S + (AA)) consisting of all candidate policies from which we can search for good policies. We also assume access to value function class J C (S x A - [0, Vmax)) model the reward functinss and g S (S x A + [-Vmax, Vmma)] to model the cost Q functions of candidate policies. We further assume access to function class W (w S x A - [o, Bwd)) that represents marginalized importance weights with respect to the offline data distribution. Without loss of generality, we assume that the all-one function is contained in W.
For given policy x II, we denote f(s', .) := E.r (aa1('') aa) for any & E S. The Bellman operator T. SxA > SxA for the reward is defined as
(T f) a) = R(s,a) + TE [f(( .T],
The Bellman operator T RSA + 12S for the cost is
(T. (8, = =C()) + [S( ()),
Let || ||,,, V [. denote the Fuclidean norm weighted by distribution μ.. We make the following standard assumptions in offline RI setting (Xie et al., 2021; Cheng et al., 2022; Zhu et al., 2023) on the representation power of the function classes:

---

Assumption 3.2 (Approximate Realizabilityy. Assume there exists 61 2 O, s.t. for any given policy x E II, we have min maX Il T fll,,- I e,, and fer maXadnissble If T fl I 61, where v is the state-action distribution of any admissible policy such that v E {d" VT E II}.
Assumption 3.2 assumes that for any policy π E 11, Q and Q are approximately realizable in F and G. When é₁ is small for all admissible v, we have fr R Q7, and fe ~ QT. In particular, when 6 = O, we have O E F, F. E F for any policy æ E II. Note that we do not need Bellman completeness assumption Cheng al. (2022).
3.3 Offline RI.
In offline RI., assume that the available offline data D {(8i, 04, Vi, G, & consists of N samples. Samples are i.i.d. (which are common assumptions in unconstrained Cheng et al. (2022), as well as constrained setting Hong al. (2024)), and the distribution of each tuple (s, a, r, c, s') is specified by a distribution μ4 E ASS x A), which is also the discounted visitation probability of a behavior policy (also denoted by μ for simplicity). In particular, (s, a) u, R(s, a),c C(s,a) 5 ~ P(-|s, a). We use ~ p(-[s), to denote that the action is drawn using the behavior policy and (s, a, s') ~ je to denote that (s, a) ~ j., and s' ~ P(-ls,a).
For given policy r,, we define the marginalized importance weights w (s. := (s.a)
x(s.a) which is
the ratio between the discounted state-action occupancy of π and the data distribution μ. This ratio can be used to measure the concentrability of the data coverage (Xie and Jiang, 2020; Zhan et al., 2022; Rashidinejad et al., 2022; Ozdaglar et al., 2023; Lee et al., 2021).
In this paper study offline RL with access to dataset with limited coverage. The coverage of policy x is the dataset can be measured by the weighted &g single policy concentrability coefficient (Zhu et al., 2023; Yin and Wang, 2021; Uehara et al., 2024; Hong et al., 2024):
Definition 3.3 (fg Concentrability). Given a policy π, define C₁2 || I12, = /pll2,,
Remark 3.4. The definition here much weaker than the all policy concentrability used in offline RL (Chen and Jiang, 2019) and safe offline RL (Le et al., 2019; Hong et al., 2024), which requires the ratio d" (s.a) to be bounded for all E S and a E A and all policies π. In particular, the all-policy concentrability assumption essentially requires the dataset to have full coverage of all policies ((nearly all the state action pairs). This requirement is often violated in practical scenarios. This requirement even impossible to meet safe offline RI. because would require collecting data from every dangerous state and actions, which clearly is impractical.
p(s,a)
In the following lemma, we compare two variants of single-policy concentrability definition with the l defined in Definition 3.3.
Lemma 1 (Restate Proposition 2.1 in Zhu et al. (2023)). Define the lox single policy concentra- bility (Rashidinejad et al., 2021) as CT= = =|d= /////l and the Bellman-consistent single-policy
1f-T"f1ll,ar (T could be Tr or Te in our
concentrability (Xie et al., 2021) as C Bellman maxfeF [|5-T*
setting) Then, it always holds (CT,)² S CT., CT < C and there exist offline RL instances where (CE,) )2 S Bellman C% S Bellmann Remark 3.5. It is easy to observe that the E variant is bounded by lx and C Bellman under some cases. There is an example (Example 1) in Zhu et al. (2023) showing that C7, is bounded by a constant
v2 while CF could be arbitrarily large. For the case when the function class F is highly expressive, C Bellman could be close to C and thus possibly larger than Cτ,- Intuitively, C₁, implies that only Egg (w* a)] is bounded, rather, w (s, a) is bounded for all (s, a) in lo concentrability bound.
Given the definition of the concentrability, we make the following assumption the weight function class W and a single-policy realizability:
Assumption 3.6 (Boundedness of W in &g norm). For all w E W, assume that || S Cj Assumption 3.7 (Single-policy realizability of wx). For some policy x that we would like to compete with, assume that wº E W.
In this paper, we want to study the robust policy improvement on any reference policy, then we assume that we are provided a reference policy net. Note that in many applications (e.g., scheduling,
5

---

networking) we indeed have reference policy. We want that while applying sophisticated RL policy it should do better and be safe as well. This is one of the main motivations behind this assumption.
Assumption 3.8 (Reference Policy). We assume access reference policy Teet E II, which can be queried at any state.
In many applications such as networking, scheduling. and control problems, there are existing good enough reference policies. In these cases, robust and safe policy improvement over these reference policies has practical value. If Tet is not provided, we can simply run a behavior cloning on the offline data to extract the behavior policy Tel accurately, as long as the size of the offline data set is large enough. More discussion can be found in Section C in the Appendix.
4 Actor-Critic with Importance Weighted Bellman Error
Our algorithm design builds upon the constrained actor-critic method, in which we iteratively optimize a policy and improve the policy based on the evaluation of reward and cost. Consider the following actor-critic approach for solving the optimization problem (5):
Actor: #" E arg max fr (so: T) s.t. f (so, X) $ hell
Critic: J E Brg (OC(J-TT,,C, a)) f E arg feg feg Bul(U-Ta((sss ()))
where we assume that 8q is fixed initial state, and ()( EesA T(als) a). The policy is optimized by maximizing the reward function fr while ensuring that fc satisfies the constraint, and the two functions are trained by minimizing the Bellman error. However, this formulation has several disadvantages. 1) It cannot handle insufficient data coverage, which may fail to provide an accurate estimation of the policy for unseen states and actions. 2) cannot guarantee robust policy improvement. 3) The actor training step is computationally intractable especially when the policy space is extremely large.
(()s) a), fo(s,
To address the insufficient data coverage issue, as mentioned in Xie et al. (2021) the critic can include Bellman--onsistent pessimistic evaluation of ,, which selects the most pessimistic function that approximately satisfies the Bellman equation, which is called absolute pessimism. Then later as indicated by Cheng et al. (2022), instead of using an absolute pessimism, a relative pessimism approach by considering competing to the behavior policy can obtain a robust improvement over the behavior policy. However, this kind of approach can only achieve a suboptimal statistical rate of /1/3, and fails to achieve the optimal statistical rate of 1/ I/VN, then later a weighted average Bellman error (Uehara et al., 2020; Xie and Jiang, 2020; Zhu et al., 2023) could be treated as one possible solution for improving the order. We remark here that all the discussions here are for the traditional unconstrained offline RL. Regarding safety, no existing efficient algorithms in safe offline RL have theoretically demonstrated the property of robust policy improvement with optimal statistical rate.
Can Primal-dual based approaches achieve result using only single policy coverability?: The most commonly used approach for addressing safe RL problems is primal-dual optimization (Efroni et al., 2020; Altman, 1999). As shown in current offline safe RL literature (Hong et al., 2024; Le et al., 2019), the policy can be optimized by maximizing a new unconstrained "reward" Q- function f (Su, x) Xf. (su. x) where À is dual variable. Then, the dual-variable can be tuned by taking gradient descent step. As we discussed in the introduction, all these require all policy concentrability which is not practical especially for safe RL. Important question is whether all policy concentrability assumption can be relaxed. Note that primal-dual algorithm relies on solving the min-max problem miny max, f. (so. T7) - Xf[(so. T). Recent result (Cui and Du, 2022) shows that single policy concentrability assumption is not enough for offline min-max game. Hence, we conjecture that using the primal-dual method we can not relax the all policy concentrability assumption. Intuitively, the primal-dual based method (Hong et al., 2024) rely on bounding the regret in dual domain (Ax - X²) (fT* - 0). hence, all the policies { 1K encountered throughout the iteration must be supported by the dataset to evaluate the dual value X" X"(f=* - 0) where X is the optimal dual value.
objective function f= .) X [J((So. T]]+ to control aggressive policies, where (-)+ : max 0). The high-level intuition behind this aggression-limied objective function is that by appropriately selecting a (usually large enough), we penalize all the policies that are not safe. As result, the policy that maximizes the
Our novelty:
contrast, we propose an aggression-limines

---

objective function is the optimal safe policy. This formulation is fundamentally different from the traditional primal-duat approach as does require all policy concentrability. In particular, we only need to bound the primal domain regret which can be done as long as the reference policy is covered by the dataset similar the unconstrained setup.
tuning, and thus, does not require
Combining all the previous ideas together provides the design of our main algorithm named WSAC (Weighted Safe Actor-Critie). Section we will provide theoretical guarantees of WSAC and discuss its advantages over existing approaches in offline safe RL. WSAC aims to solve the following optimization problem:
# E arg max (x(m, (L TEll
(6)
s.t. f. E arg min CT (E, ff E arg min - (T, BE (T), f.e0
FREF
[(s,a)), where C,(T, == (,, [(s,a)), and {) = max (()
T, f)(ss ())], ( f) == maxwEw
af -T, This formulation can also be treated as Stackelberg game (Von Stackelberg, 2010) bilevel optimization problem. We penalize the objective function only when the approximate cost Q-function fe of the policy x is more perilous than the behavior policy (f x) f. (s, a)) forcing
policy be as safe the
behavior policy. Maximization over w in for training the two critics can ensure that the Bellman error is small when averaged over measure μ for any W, which turns out to be sufficient control the suboptimality of the learned policy.
In the following theorem, we show that the solution of the optimization problem (6) is not worse than the behavior policy μ in both performance and safety for any 33 2 0, than the policy μ under Assumption 3.2 with éj = O.
Theorem 4.1. Assume that Assumption 3.2 holds with = 0, and the behavior policy p E II, then for any 0, X we have J,(+*) 2 J,(,,, and (Je(a S (Jc(n))+
The result in Theorem 4.1 shows that by selecting À large enough, for any 8 2 0, the solution can achieve better performance than the behavior policy while maintaining safety that arbitrarily close to that of the behavior policy. The Theorem verifies the design of our framework which has the potential to have a robust safe improvement.
In the next section, we will introduce our main algorithm WSAC and provide its theoretical guarantees.
Theoretical Analysis of WSAC
5.1 Main Algorithm
In this section, we present the theoretical version of our new model-free offline safe RL algorithm WSAC. Since we only have access dataset D instead of the data distribution. WSAC sovles an empirical version of (6):
# E arg max (D(TT MCD TEll
(7)
s.h. ff arg min f.) BED(T, ft), f.EF
fe E Erg ACD( (TD(T, f.eo
where
(D(T, :=<D]( f(s
ED( = max Ep Ep(w(s,
- -
wEW
(8)
ED(T,)) max wEW
Ep as - - c- c- 7f m)]]
As shown in Algorithm 1, at each iteration, WSAC selects fM maximally pessimistic and fh maximally optimistic for the current policy т with a weighted regularization on the estimated Bellman error for reward and cost, respectively (Line 4 and 6) to address the worse cases within reasonable range. In order to achieve a safe robust policy improvement, the actor then applies a no-regret policy optimization oracle to update the policy πk+1 by optimizing the aggression- limited objective function compared with the reference policy (Line 7) (*. Xf (s,
7

---

Algorithm Weighted Safe Actor-Critic (WSAC)
1: Input: Batch data D, coefficient B, A. Value function classes F, importance weight function class W, Initialize policy T1 randomly. Any reference policy Tet No-regret policy optimization oracle PO (Definition 5.1).
2: for k ===,, =........ K do
3: Obtain the reward state-action value function estimation of Ta:
4: f <- arg e, (D(Tk J+) + 3ED(TT: fr)
5:
Obtain the cost state-action value function estimation of ITg:
6:
f <- arg min (T fe) (ED(T. fc)
7: Update policy: Tk+1 -- PO (s, (s) (s, m.t))+, LD, ED
are defined in (5.1)
8: end for
9: Output: क = Unif(m1,...... T
I/ Uniformly mix M1..... K
(s, m.))) Our algorithm very computationally efficient and tractable compared with existing approaches (Hong et al., 2024; Le et al., 2019), since we do not need another inner loop for optimizing the dual variable with an additional online algorithm or offline policy evaluation oracle. The policy improvement process relies on a no-regret policy optimization oracle, a technique commonly employed in offline RL literature (Zhu al., 2023; Cheng al., 2022; Hong et al., 2024; Zhu et al., 2023). Extensive literature exists on such methodologies. For instance, approaches like soft policy iteration (Pirotta et al., 2013) and algorithms based on natural policy gradients (Kakade, 2001; Agarwal et al., 1) can function as effective no-regret policy optimization oracles. We now formally define the oracle:
Definition 5.1 (No-regret policy optimization oracle), An algorithm PO is called no-regret policy optimization oracle if for any sequence of functions f ,..... fk with f* S x A + [0, Vmax., Vk [K]. The policies ......y TK produced by the oracle PO satisfy that for any policy π E II:
K E. (s, ft [x) o(1)
Cppt
K
(9)
There indeed exist many methods that can serve the no-regret oracle, for example, the mirror- descent approach (Geist et al., 2019) or the natural policy gradient approach (Kakade, 2001) of the form T (a|s) x (a]s) exp(nf* (s, a)) with 17 2v K (Even-Dar et al., 2009; Agarwal et al., 2021). In the following define cppt the error generated from the oracle PO by considering f (s. a) - Af a) - (s, as the sequence of functions in Definition 5.1, then we have the following guarantee.
log (AI
Lemma 2. Applying no-regrei oracle PO for K episodes with (J+ -MJ (f),,
for an arbitrary policy n, can guarantee
- Copt
(10)
K
E. (s, 1: (8, Copt + Vnax
(11)
K
Lemma 2 establishes that the policy outputted by PO with considering the aggression-limited "reward" can have a strong guarantee on the performance of both reward and cost when À is large enough., which is comparable with any competitor policy. This requirement is critical to achieving the performance guarantee of our algorithm and the safe and robust policy improvement. The detailed proof is deferred to Appendix B.2 due page limit.
5.2 Theoretical Guarantees
We are now ready to provide the theoretical guarantees of WSAC Algorithm The complete proof deferred to Appendix B.3.

---

Theorem 5.2 (Main Theorem). Under Assumptions 3.2 and 3.6, let the reference policy oee E II in Algorithm 1 be any policy satisfying Assumption 3.7. then with probability at least 1
(T (+) S 50( Estot + C& VEi
And Eopt
(12)
Vinxx
Jo(T)- Jo((Tee)) S 0 Estat + C2a E1
+
Tnd Copt
(13)
X
where Estat == Vimx C22 log ( Br log (11 by Algorithm/ I with 3 > 0 and Fef as input.
and ßt is the policy returned
Remark 5.3. When 61 = O, i.e., model misspecification, which states that the true value function belongs to the function class being used to approximate it (the function class is right enough), let The be the optimal policy, the results in Theorem 5.2 achieve optimal dependence statistical rate of VN (for large k), which matches the best existing results. Our algorithm is both statistically optimal and computationally efficient with only single-policy assumption rather relying much stronger assumptions of all policy concentrability Hong et al. (2024); Le et al. (2019). Hence, if the behavior policy or the reference policy is safe, our result indicates that the policy returned by our algorithm will also be safe (nearly). Such a guarantee was missing in the existing literature.
Remark 5.4. We also do not need a completeness assumption, assumpttion.hhich requires that for any ʃ E F or G and π E 11, it approximately holds that T,f Tef F as required in Xie et al. (2021); Chen et al. (2022b). They need this assumption to address over-estimation issues caused by the Eg square Bellman error, but our algorithm can get rid of the strong assumption by using a weighted Bellman error which is a simple and unbiased estimator.
Remark 5.5. Our algorithm can compete with any reference policy Tet E II as long as "m d= //u is contained in W. The importance ratio of the behavior policy is w" = dd /p = j/p μ = which always satisfies this condition, implying that our algorithm can have safe robust policy improvement (in Theorem 5.6 discussed below).
5.3 A Safe Robust Policy Improvement
A Robust policy improvement (RPI)(Cheng et al., 2022; Zhu et al., 2023; Bhardwaj et al., 2024) refers to the property of an offline RI. algorithm that the offline algorithm can learn to improve over the behavior policy, using a wide range of hyperparameters. In this paper, we introduce the property of Safe Robust policy improvement (SRPI) such that the offline algorithm can learn to improve over the behavior policy in both return and safety, using wide range of hyperparameters. In the following Theorem 5.6 we show that long the hyperparameter 3 ol o(vN), our algorithm can, with high probability, produce a policy with vanishing suboptimality compared to the behavior policy.
Theorem 5.6 (SRPI). Under Assumptions 3.2 and 3.6, then with probability at least
(n( - (,( 0
Cata + Ci.Va
capt
(14)
Jo(T) Jo(p)= so
Vmax X
Gtat + Cê.vei
+ cope +
(15)
V Vimax Ci2 log (1/8)
Vanas Be log [F (1108)))))) and i is the policy returned
where Cetat
by Algorithm I with 3 0 and a as input.
The detailed proofs are deferred to Appendix B.4.
6 Experiments
6.1
WSAC-Practical Implementation
We introduce deep RL implementation of WSAC in Algorithm 2 (in Appendix), following the key structure its theoretical version (Algorithm 1). The reward, cost Q-functions fr, and the policy network are all parameterized by neural networks. The critic losses (line 4) (rewora and

---

Table 2: The normalized reward and cost of WSAC and other baselines. The Average line shows the average situation various environments. The cost threshold is 1. Gray: Unsafe agent whose normalized cost is greater than 1. Blue: Safe agent with best performance
Enviconeest
BC
Sate-BC
BCQL
BEARL
CPQ
COgaDICE
WSAC
Reward
Cost
Roward
Cost J
Rewadd
Cost.
Reward f
Cost 2
Reward
Coot
Roward 1
Cost 1
Rewaed
Cost J
BallCircle
0,70
0.95
0.61
0.19
0.75
0.82
0.80
1.23
0.62
0.76
0.71
1,13
0.75
0.27
CarCirole
0,57
1.43
0.57
0.65
0.79
1.19
0.84
1.57
0.67
0.28
0.49
1.52
0.68
0.59
PointRation
0.26
1.75
0.12
0.69
0.36
1.76
0.32
1.71
0.43
3.10
0.15
1.02
0.13
0.57
PointPush
0.13
0.67
0.30
1.35
0.16
1.01
0.12
0.50
-0.01
2.39
0.07
1,18
0.07
0.52
Average
1.20
0.38
0.52
1,43
1.46
lesst (fe) are calculated based on the principles of Algorithm 1, on the minibatch dataset. Optimizing the actor aims to achieve a no-regret optimization oracle, we use a gradient based update on the actor loss (line 5) Lactor (T.) In the implementation we use adaptive gradient descent algorithm ADAM (Kingma and Ba, 2015) for updating two critic networks and the actor network. Algorithm follows standard two-timescale first-order algorithms (Fujimoto et al., 2018; Haarnoja et al., 2018 18) with fast learning rate astt on update critic networks and a slow learning rate nstow for updating the actor.
6.2 Simulations
We present scalable deep RL version of iS Algorithm 2, following the principles of Algorithm 1. We evaluate WSAC and consider Behavior Cloning (BC), safe Behavior Cloning (Safe-BC), Batch- Constrained deep Q-learning with Lagrangian PID (BCQL) Fujimoto et al. (2019); Stooke et al. (2020). bootstrapping error accumulation reduction with Lagrangian PID (BEARL) Kumar et al. (2019); Stooke et al. (2020), Constraints Penalized Q-learning (CPQ) Xu et al. (2022) and one of the state-of-ther thr algorithms, COptiDICE (Lee et al., 2022) as baselines.
We study several representative environments and focus on presenting "BallCircle". In BallCircle, it requires the ball on circle a clockwise direction without leaving the safety zone defined by the boundaries proposed by Achiam et al. (2017). The ball is spherical-shaped agent which can freely move on the xy-plane. The reward is dense and increases by the car's velocity and by the proximity towards the boundary of the circle. The cost is incurred if the agent leaves the safety zone defined by the boundaries.
We use the offline dataset from Liu et al. (2019), where the corresponding expert policy are used to interact with the environments and collect the data. To better illustrate the results, we normalize the reward and cost. Our simulation results are reported in Table 2, we observe that WSAC can guarantee that all the final agents are safe, which is most critical in safe RI. literature. Even in challenging environments such as PointButton, which most baselines fail to learn safe policies. WSAC has the best results in of the environments. Moreover, WSAC outperforms all the baselines in terms of the average performance, demonstrating its ability to learn a safe policy by leveraging offline dataset. The simulation results verify our theoretical findings. We also compared WSAC with all the baselines in the case where the cost limits are different, WSAC still outperforms all the other baselines and ensures a safe policy. We further include simulations to investigate the contribution of each component of our algorithm, including the weighted Bellman regularizer, the aggression- objective, and the no-regret policy optimization which together guarantee the theoretical results. More details and discussions are deferred to the Appendix D due to page limit.
7 Conclusion
In this paper, we explore the problem of offline Safe-RL with a single policy data coverage assumption. We propose a novel algorithm, WSAC, which, for the first time, is proven to guarantee the property of safe robust policy improvement. WSAC able to outperform any reference policy, including the behavior policy, while maintaining the same level of safety across a broad range of hyperparameters. Our simulation results demonstrate that WSAC outperforms existing oftt-oo-thh-ar offline safe-RL algorithms. Interesting future work includes combining WSAC with online exploration with safety guarantees and extending the approach to multi-agent settings to handle coupled constraints.
10

---

References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In Int. Conf. Machine Learning (ICML), volume 70, pages 22-31. JMLR.
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2021). On the theory policy gradient
methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76.
Altman, E. (1999). Constrained Markov decision processes, volume 7. CRC Press.
Bhardwaj, M., Xie, T., Boots, B., Jiang, N., and Cheng, C.-A. (2024). Adversarial model for offline reinforcement learning. Advances in Neural Information Processing Systems, 36.
Chen, F,, Zhang, J., and Wen, Z. (2022a). near-optimal primal-dual method for off off-policy learning in emdp. In Advances Neural Information Processing Systems (NeurIPS), volume 35, pages 10521-10532.
Chen, J. and Jiang, N. (2019). Information-theerrtiicc considerations in batch reinforcement learning. In Int. Conf. Machine Learming (ICML), pages 1042-1051. PMLR.
Chen, J. and Jiang, N. (2022). Offline reinforcement learning under value and density-ratie realizabil- ity: the power of gaps. In Uncertainty Artificial Intelligence, pages 378-388. PMLR.
Chen, I.., Jain, R., and Luo, H. (2022b). Learning infinite-horizon average-reward markov decision process with constraints. In Int. Conf. Machine Learning (ICML), pages 3246-3270. PMLR.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, (2021). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097.
Cheng, C.-A., Kolobov, A., and Agarwal, (2020). Policy improvement via imitation multiple oracles. In Advances Neural Information Processing Systems (NeurIPS), volume 33, pages 5587- 5598.
Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. (2022). Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR.
Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. (2017). Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1)607006120.
Cui, Q. and Du, S. S. (2022). When offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems, 35:25779-25791.
Efroni, Y., Mannor, S., and Pirotta, M. (2020). Exploration-explootatioo in constrained MDPs. arXiv preprint 1rXiv:2003.02/89.
Even-Dar, E., Kakade, S. M., and Mansour, Y. (2009). Online markov decision processes. Mathemat- ics of Operations Research, 3463)7726-736.
Fujimoto, S., Meger, D.,, and Precup, D. (2019). Off-policy deep reinforcement learning without
exploration. International conference on machine learning, pages 2052-2062. PMLR.
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In Int. Conf. Machine Learning (ICML), pages 1582-1591.
Geist, M., Scherrer, B., and Pietquin, O. (2019). A theory of regularized markov decision processes. In International Conference on Machine Learning, pages 2160-2169. PMLR.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Int. Conf. Machine Learning (ICML), pages 1861-1870.

---

Hong, K., Li, Y., and Tewari, A. (2024). A primal-dul--rie algorithm for offline constrained reinforcement learning. Int. Conf. Artificial Intelligence and Statisties (AISTATS), pages 280- 288. PMLR.
Isele, D., Nakhaei, A., and Fujimura, K. (2018). Safe reinforcement learning autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-6. IEEE.
Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. In Int. Conf. Machine Learning (ICML), pages 267-274.
Kakade, S. M. (2001). A natural policy gradient. In Advances Neural Information Processing Systems (NeurIPS).
Kingma, D. P. and Ba, J. (2015). Adam: method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, Inf. Conf. on Learning Representations (ICLR).
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. (2019). Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32.
Kumar, A., Hong, J., Singh, A., and Levine, S. (2022). Should i rum offline reinforcement learning behavioral cloning? In Int. Conf. on Learning Representations (ICLR).
Laroche, R., Trichelair, P., and Des Combes, R. T. (2019). Safe policy improvement with baseline bootstrapping. In International conference on machine learning, pages 3652-3661, PMLR.
Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy learning under constraints. In International Conference Machine Learning, pages 3703-3712. PMLR.
Lee, J., Jeon, W., Lee, B., Pineau, J., and Kim, K.-E. (2021). Optidice: Offlin policy optimization via stationary distribution correction estimation. In Meila, M. and Zhang, T., editors, Int. Conf. Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages -120- PMLR.
Lee, J., Paduraru, C., Mankowitz, D. J.. Heess, N., Precup. D., Kim, K.-E., and Guez, A. (2022). Cop- tidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.0895:
Liao, P.. Qi. Z., Wan, R., Klasnja. P., and Murphy, S. A. (2022). Batch policy learning in average reward markov decision processes. Annals of statistics, 50(6):3364.
Liu, B., Cai, Q., Yang, Z., and Wang, Z. (2019). Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems, 32.
Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., Hu, H., Yu, W., Zhang, T., Tan, J., et al. (2023a). Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09300
Liu, Z., Guo, Z., Yao, Y., Cen, Z., Yu, W., Zhang, T., and Zhao, D. (2023b). Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 21612111122211 1661--21630. PMLR.
Ozdaglar, E., Pattathil, S., Zhang, J., and Zhang, K. (2023). Revisiting the linear- linear- peeer-programming framework for offline rl with general function approximation. In International Conference on PMLR.
Machine Learning, pages
Pirotta, M., Restelli, M., Pecorino, A., and Calandriello, D. (2013). Safe policy iteration. In Int. Conf. Machine Learning (ICML), pages 307-315. PMLR.
Rajaraman, N., Yang, L., Jiao, J.. and Ramchandran, (2020). Toward the fundamental limits of
imitation learning. Advances Neural Information Processing Systems (NeurIPS), 33:2914-2924
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Advances Neural Information Processing Systems (NeurIPS), volume 34, pages 11702-11716.
12

---

Rashidinejad, P. Zhu, H., Yang, K., Russell, S., and Jiao, J. (2022). Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716.
Siegel, N. Y., Springenberg, T., Berkenkamp, F., Abdolmaleki, A., Neunert, M., Lampe, T., Hafner, R., Heess, N., and Riedmiller, M. (2020). Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint XXX2:2002.08836
Stooke, A., Achiam, J., and Abbeel, P. (2020). Responsive safety in reinforcement learning by pid lagrangian methods. In Int. Conf. Machine Learning (ICML), pages 9133 9143. PMLR.
Uchara, M., Huang. J., and Jiang. N. (2020). Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pages 9659-9668, PMLR.
Uchara, M., Kallus, N., Lee, J. D., and Sun, W. (2024). Offline minimax soft-q-learning under realizability and partial coverage. Advances in Neural Information Processing Systems, 36.
Uchara, M. and Sun, W. (2021), Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226.
Von Stackelberg, H. (2010). Market structure and equilibrium. Springer Science & Business Media.
Wang, I.., Cai, Q. Yang. 7., and Wang, 7. (2019). Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01 150.
Wu, R., Zhang, Y., Yang, Z. and Wang. Z. (2021). Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration. In Advances Neural Information Processing Systems (NeurIPS), volume 34, pages 25439-25451
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683- 6694.
Xie, T. and Jiang, N. (2020). Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550-559. PMLR.
Xie, T. and Jiang, N. (2021). Batch value-function approximation with only realizability. In Int. Conf. Machine Learning (ICML), pages 11404-114 11413. PMLR.
Xu, H., Zhan, X., and Zhu, X. (2022). Constraints penalized q-learning for safe offline reinforcement learning. AAAI Conf. Artificial Intelligence, volume 36, pages 8753-8760.
Yin. M. and Wang. Y.-X. (2021). Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065-4078.
Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. (2022). Offline reinforcement learning with realizability and single-policy concentrability. In Proc. Conf. Learning Theory (COLT), pages 2730-2775. PMLR.
Zhang, J., Koppel, A., Bedi, A. S., Szepesvari, C., and Wang, M. (2020). Variational policy gradient method for reinforeement learning with general utilities. Advances in Neural Information Processing Systems, 33:4572 4583.
Zheng, Y., Li, J., Yu. D., Yang. Y., Li, S. E., Zhan, X., and Liu, (2024). Safe offline reinforcement learning with feasibility-guider diffusion model. arXiv preprint 2411:22001.17000
Zhu, H., Rashidinejad, P., and Jiao, J. (2023). Importance weighted actor-critic for optimal conserva- tive offline reinforcement learning. arXiv preprint arXiv:2301.127714
13

---

Supplementary Material
A Auxiliary Lemmas
In the following, we first provide several lemmas which are useful for proving our main results.
Lemma 3. With probability at least for any fr E F. E G, E II and w E W, we have
ExI(fr - T, f)oll
Σ
w(s,a)(ff(ssa) - - -frls m>)|
N
(s,0.T,e")
<0 Vimx|
log(IF
IT [1
Vinx Bur log( [1/111/01
(16)
N
N
Bullfe
- -Teffuu
Σ
wls ((ss,)(fe(s,a) c 7fcls', ,
N
(s,a.n,")
<0(1 <0(Vins
log( (0(((1111))118)
Vmax Bwr log(19) (0mmiil1
Vimax|
(17)
N
N
The proofs can be found in Lemma in Zhu et al. (2023).
Lemma 4. With probability at least - 26, for any fr E F, E G and π E 11, we have
(T, (T, fe)
ED( Sc)] S Estat
(18)
[,, fo)
ED(T, fa)l S Estat
(19)
where Estat == Vmaa Ct,1
logt [og[1""99 19911111)
Vmnv B.. log(12) (1
Proof. Condition on the high probability event in Lemma 3,for any f. E F, e EI, define WH, arg max (T, fx) arg max
E [w][ a) T, (,),
wEW
and define
Ux.s- = arg max ED = arg max wEW N
Σ
- -
WEW
(s,0,r.')
Then we can have
(T ED(TT
1 (s, TT
Σ - .T)
N
(e,u.r,e))
-IE (8, aJ T,, TT,' f.) a)]l
Σ ()) - -
(s.a.r,"))
20 Estat = -Estati
where the inequality is true by using the definition of wh,, and Lemma 3. Thus
(n fr) ST)
[E (T, 1 Σ
wh (s - (''
(sia.na))
(s)0 - - T))
(s,a,n;'))
Wn. (s,a((s)))( - (s' x))
N
(K,0,1,"))
Sestat
The proof for the case (T, fc) ED(TT, Estat is similar.
14

---

Lemma's 5. (Empirical weighted average Bellman Error) With probability at least for any II, we have
{T) SCi Estat
(20)
Ed f fo) SC /a1 Catat,
(21)
where
f7 = arg min sup admissible y
Il - T, fr VT E
J.EF
115c - T Sellz Vr ET. J arg min sup
f.eg admivsible
Proof. Condition on the high probability event in Lemma 4, we have
(T, f7) = max [u a)((f - T, (7,,)) wEW
SE (T) max
1| T," TT" f) (s)s))))22. a))
<CryVei
where the first inequality is because of Cauchy-Schwarz inequality and the second inequality comes from the definition of f7 and Assumption 3.2, thus we can obtain
ED( I (x S.) + Estat SS Cū /c1 + Estat-
(22)
Following a similar proof we can have
Ep( S Ex( + Estat S Cig V + Cstat-
(23)
Lemma 6. (Performance difference decomposition. restate of Lemma 12 in Cheng et al. (2022)) For an arbitrary policy n, t E II, and I be an arbitrary function over S x A. Then we have,
(O( - (ft
[(s, c, - Lu((,, ²a), (To + E. (T (s) Bx
(24)
where o == or c.
Proof. We prove the when r, the other case is identical. Let R (s,a) = s(s ( [f #)] be a virtual reward function for given f and æt According to performance differ- ence lemma (Kakade and Langford, 2002), We first have that
(Jr(+) (x))) ( Q
=(( + (f,
(A(#) = Cn,t,: Q - Ct(
= =(A)) + [s, -((,
-({) + (1 - (J., J RI (n))
-(+)) + (1 7)Q (80. 4) - E,
A(()) (1 7) #) RR
(s, (
where the last equality is true because that
Phra(1,00 (Thia (),,a) RI + + (5(8 ((,
Thus we have
=(J((T)) Jr(1)) - (J,(+) - Jr(1))
=(J+(T) f(do, #)) + -J.(m))))))))))))))))))))))))- - A(+).
J.(1))
(25)
15

---

For the first term, we have
(J+(T) f(do: R)) (J) f(s. R))
(deterministic initial state)
=J, -E [R (s, a) E.. [Ra./ / (s. a)] f(s. #)
-Ed- [R(s,a) R (s. E. [f(s. - f(s, #)]
-E,- I(T," (s, Ea - f(s. R)],
(26)
where the second equality is true because E [R. (s, a)] f(0,, #)
-Ea- (J) 7B [f(s', [s]] f(80, +)
-Ea- [[(,,))] - Σ
7 Pr(s: = ~ do, [(, f(s0, R)
8
-Ea- (f([ Σ
7 s| ~ 2),
-Ea- [f(s, [)
y Pr((s; s. =. 0/so do, ~ ()
t=U [(, #)].
-Er[s((s x)
For the second term we have
Ep R (s, as) Jr(H)
= =Ep RR (s, R(s,a)]
EEp [f T, f)ss a)l.
(27)
Therefore plugging 26 and (27) into Eq. (25), we have
(T) (+)
=B,[(1 - (,) [1, (s,
R.
E,(((s, f(8 L,(t,f) - Q+)).
The proof is completed.
Lemma 7. With probability at least 26,for any Sr E F, 9, and ι E IT, we have:
(T, [r) - CD(TT, [r)] S Estat
(28)
1 fo) - LD( fo) I Cstat
(29)
where Estat
Vinax Ci
+ Viaa B. log( F (1mliiil15
(CD (T, f.)] = Lu (T, f) and fr(s, T) fr(s, a) S Vmax- For any fr E F,, policy II, applying Hoeffding's inequality and union bound obtain with probability 1 6,
Proof. Recall that E,
fx) - (D(T, f+l 1l s of ( (Vmax V
log( log([F] (1)8)) N
< Estat-
(30)
The inequality for proving the fes æ is the same.
B Missing Proofs
B.1 Proof of Theorem 4.1
Proof. According to the performance difference lemma (Kakade and Langford, 2002), we have (J,(T) (1)) - (TJ.(T)) Ja(1)}+
=,,( \T, Q7)]+
=C,(T, Q7) (T, Q7) - (T, Q2) + (ē, Q2)
2T, 17) + (E, (f) 38, (T,
(T, - (T,
(31)
16

---

where the second equality is true because (T, Q = (T. Q.) = 0 by Assumption 3.2, and the first inequality comes from the selection fr and Je in optimization (6).
Therefore, we can obtain
Jx(+*) (1) 2 (f 15 ff - Je(1+++
2 ( (2t - ME)(,, -
((f((t)JJ((l))) 20
(32)
and
(Jo(*))+ (Jc(1)++ $ (Jo(#) Jo(N)}+ S (J,( J,(N)) S
1
(33)
B.2 Proof of Lemma 2
Proof. Denote Tief as π. First according to the definition for the no-regret oracle 5.1, we have
K
> E (s, - (s,
(s, +),+)
(t),
(34)
Therefore,
K
Ex
-M
K
Ept
E. -
Eopt
(35)
K
k=1
and
K
Vmax
B (s, - (s,
Eppt
Ex (8, - (s, S Epp +
K
AK
X
km
k
(36)
We finish the proof.
B.3 Proof of Theorem 5.2
Theorem (Restate of Theorem 5.2). Under Assumptions 3.2 and 3.6, let the reference policy Tef E II be any policy satisfying Assumption 3.7, then with probability at least 1 8,
Jr(Tn)) J,(7) < 20 Estat + Cigy Vei
cxpt
(37)
V Vinax
Je(T)- Jo(TTre) < 0 Extat + Cū, Vei
Eopt
(38)
A
where Estat := Vina Cr log(LF (1/88)
Vinax Bu log W(/8)
and के is the policy re-
N
N
turned by Algorithm I with 3 > 0 and Tny input.
17

---

Proof.
Denote Tett as т. According to the definition of म,, and Lemma 6 we have
Jx(
Σ (z) (T+)))
K
By
pk
T
Ex
pk 1
K
()
E. (s, (s, Tk))
LTk 8:) - (Tk: 1Th)
(39)
(II)
(IV)
Condition on the high probability event in, we have
(1) (I) ( f*) (T f+) + 2Estat
(40)
According to similar argument as that in the Lemma 13 in Cheng al. (2022), have that
(TR, ²T (TR, 17
= =IE,IQT (s, T2) - Q7* a)] - ( s:m+)l
=I(J((Tk) (r)) - (
=I(F" (80, Tk) Jx)) + (Jr( - (80: T&.) (T ST
= -IE (8, (T, f.h (s,, Ex [TT, (s, a) fr (s, ( f.
(by the extension of performance difference lemma (Lemma Cheng et al. (2020)))
(Th f:²²) + (f - (T, (s, a)] + Ear (TT fs.)) f (s. C (TkK. f7+
$11f7"* (s,a) - (T, f:x (s, a) 122, + [1(T," (TT)0*((+))))) (s, a) - fo (s, a)l22,00
<V(VE)),
(41)
where 17 = arg min sup ||5. - T,- 12, / VT E II. By using Lemma 7, we have f.eF admissible
(T 1 LD(T,, [+) C f7a) (D( s7 S OEEsaat)
(42)
Therefore
(I) + (II) (IV) Ca(TT 11: 28 (Tk, f + 2e
1Tth) ((vi)
(43)
CD(TTK ( + 2E
Tb; f + (Esrra)) CD( 17 ((vE))
(44)
s CD( f + 2E fo + ((Estat) (T. ( f:²)) O(v
(45)
SO(saa Ci VG)
(46)
where the third inequality holds by the selection of ff, and the last inequality holds by Lemma 5. Therefore by using Lemma B.2 we obtain
(x( J. (x(x) S (Estaa Ci Vei) + Copt-
(47)
Following a similar argument, we have that
Jo( - Jo(TT) = Σ - - Jo(TT) S OEEstat + VE2)EL) + + + Viar
K
(48)
K
X
B.4 Proof of Theorem 5.6
Theorem (Restate of Theorem 5.6). Under Assumptions 3.2 and 3.6, let the reference policy oeef E 1I be any policy satisfying Assumption 3.7, then with probability at least 1 8,
Jr(1) - (+) < so Estat + Ci,
+ c
(49)
+ t + Vmxx
Ja(T) - Jp) < 0
Cgaa Cia V
(50)
A
18

---

where Cstat = Vinase Ct2 [/ Vinx Be (1 by Algorithm I with 3 0 and I input.
and itt is the policy returned
N
Proof. Following a similar proof in Theorem 5.2. But when the reference policy is the behavior policy, have (I) (II 0. Therefore have have
(IV) = (T f,) (T QT*)
(T f/) - (TE, ²**) + (ED(TA 1.
(T f²) (T( 2**²)²²+))0 + (T 1+)) [E ( SaJ) + 3C2 Vi + Bestat (Lemma 5)
(TD( f.)) ( f+) (T f,²)) - 3ED(1t fx.) (3 1)(cstat Ci (i)
<(3 ()((sttt Ct VEi). We finish the proof.
C Discussion on obtaining the behavior policy
To extract the behavior policy when it is not provided, we can simply run behavior cloning on the offline data. In particular, we can estimate the learned behavior policy î. as follows: Ys D, (a + a(s.a) and Vs & D, (al()))) + LAT where n(s, a) is the number of times (s, a) appears in the offline dataset 2) Essentially, the estimated BC policy matches the empirical behavior policy on states in the offline dataset and takes uniform random actions outside the support of the dataset. It is easy to show that the gap between the learned policy म and the behavior policy Th is upper bounded by ((min (1, (SI/N) (Kumar et al., 2022; Rajaraman et al., 2020). We can have a very accurate estimate long the size of the dataset is large enough.
1
*(*)
D Expermintal Supplement
D.1 Practical Algorithm
The practical version of our algorithm WSAC is shown in Algorithm 2.
Algorithm 2 WSAC Practical Version
1: Input: Batch data D. policy network म, network for the reward critic fr. network for the cost critic fc, B 0, À > O.
2: for k= 1, 2.., K do
3: Sample minibatch Dmini from the dataset D.
4: Update Critic Networks:
Geward (fr) = C (T, + BE fr);
Jr t- ADAM(J, Wast Verean (fr)),
lona (fc) = -AC (TT, Sc) + BED (T,, fc),
fo +- ADAM(fo Thatt Vooost (fo)).
5: Update Policy Network:
lactor (T) := -Cmini (T, fr) + M fc) }+
x ADAM(T Tsslow Vl (T)).
6: end for
7: Output:
D.2 Environments Description
Besides the "BallCirele" environment, we also study several representative environments follows.
All of them are shown in Figure 2 and their offline dataset is from Liu et al. (2023a).
19

---

Figure 2: BallCircle and CarCirele (left), PointButton (medium), PointPush(right) This environment requires the car move a circle in clockwise direction within the safety zone defined by the boundaries. The car a four-wheeled agent based on MIT's race car. The reward is dense and increases by the car's is velocity and by the proximity towards the boundary the circle and the cost is incurred if the agent leaves the safety zone defined by the yellow boundaries, which the same "CarCircle".
CarCircle:
This environment requires the point to navigate to the goal button location and touch the right goal button while avoiding more gremlins and hazards. The point has two actuators, one for rotation and the other for forward/backward movement. The reward consists of two parts, indicating the distance between the agent and the goal and if the agent reaches the goal button and touches it. The cost will be incurred if the agent enters the hazardous areas, contacts the gremlins, or presses the wrong button.
PointButton:
PointPush: This environment requires the point to push box to reach the goal while circumventing hazards and pillars. The objects are in 2D planes and the point is the same as "PointButton". It has small square in front of it, which makes it easier determine the orientation visually and also helps point push the box.
D.3 Implementation Details and Experimental settings
We run all the experiments with NVIDIA GeForce RTX 3080 Ti 8 S-Core Processor.
The normalized reward and cost summarized as follows:
R7 Tinén (M)
Rnormalized Tmaz (M) Tmmnn (M)
(51)
Cnormatized Cn +E
(52)
where r(M) is the empirical reward for task M, is the cost threshold, is small number to ensure numerical stability. Thus any normalized cost below considered as safe. We use ft and CN to dentoe the cumulative rewards and cost for the evaluated policy, respectively. The parameters of r minn (M), (MM) and K are environment-depeedeen constants and the specific values can be found in the Appendix D. We remark that the normalized reward and cost only used for demonstrating the performance purpose and are not used the training process. The detailed value of the reward and costs can be found in Table 3. To mitigate the risk of unsafe scenarios, introduce hyperparameter
Table 3: Hyperparameters of WSAC
Parameters
BallCirele
CarCirele
PointButton
PointPush
Be
30.0
38.0
30.0
30.0
3r
10.0
12.0
10.0
10.0
UBQ
30.0
28.0
32.0
30.0
X
[1.0, 20.0
Batch size
512
Actor learning rate
0.0001
Critic learning rate
0.0003
K
40
T Tozin (M) T Tman M
0.3831 881.4633
3,4844
0.0141
0.0012 14.6910
534.3061
42.8986
U. UBQq to the cost Q-function as an overestimation when calculating the actor loss. We use two separate 3,, 8e for reward and cost functions make the algorithm more flexible.
20

---

Normal iied Cost
1.5
0.8
Normalized
1.0
0.6
0.5
0.5
0
5k
10k
15k
20k
25k
5k
10k
15k
20k
25k
Training Steps
Iraining Steps
(a) BallCircle
0.8
Normalized Cost 2
0.7
0.6
0.4
0.3
5k
10k
15k
20k
25k
Training Steps
Training Steps
(b) CarCircle
0.4
Formalized nteward
Normalized Cost
0.2
0.0
5k
10k
15k
20k
25k
15k
20k
Training Steps
Training Steps
PointButton
0.2
Normalized Reward
Normalized Cost
5+
-0.2
5k
20k
25k
5k
15k Training Steps
10k
20k
25k
Training Steps
(d) PointPush
Figure 3: The moving average of evaluation results is recorded every 500 training steps, with each result representing the average over 20 evaluation episodes and three random seeds. cost threshold I is applied, with any normalized cost below considered safe.
21

---

We use different 3 for the reward and cost critic networks and different Uo for the actor-network to make the adversarial training more stable. We also let the key parameter within certain range balance reward and cost during the training process. Their values are shown in Table 3. In experiments, we take W = {0, } for computation effective. Then we can reduce ED(T, f) to (T, this case, Co can be considered as a part of the hyperparameter (Bc)
Cx [f((s, a) - 7 - "f''s, x))) and reduce Ep (TT, f) to Ex (f(,,)) a) -c c- 7f(s', x))² In
D.4 Experimental results details and supplements
The evaluation performances of the agents in each environment after 30000 update steps of training are shown in Table 2, and the performance of average rewards and costs are shown in Figure 3. From the results, we observe that WSAC achieves best reward performance with significantly lowest costs against all the baselines. It suggests WSAC can establish a safe and efficient policy and achieve a steady improvement by leveraging the offline dataset.
D.5 Simulations under different cost limits
To further evaluate the performance of our algorithm under varying situations. We further compare our algorithm with baselines under varying cost limits, report the average performance of our method and other baselines in Table 4. Specifically, cost limits of [10, 20, 40) are used for the BallCirele and CarCircle environments, and [20, 40, 80] for the PointButton and PointPush environments, following the standard setup outlined by Liu et al. (2023a). Our results demonstrate that WSAC maintains safety across all environments, and its performance is either comparable to or superior to the best baseline in each case. These suggest that WSAC is well-suited for adapting to tasks of varying difficulty.
Table 4: The normalized reward and cost of WSAC and other baselines for different cost limits. Each value is averaged over 3 distinet cost limits, 20 evaluation episodes, and 3 random seeds. The Average line shows the average situation in various environments. The cost threshold is 1. Gray: Unsafe agent whose normalized cost is greater than 1. Blue: Safe agent with best performance. The performance of all the baselines is reported according the results in Liu al. (2023a).
BC
Safo-BCC
BCQI
IPARI
CPQ
COHIDEE
WSSC
Beward 1
Coal $
Rewadd
Cot
Rewarl 1
Cow
Clst
Becand
Cosl
Rewadd
Cot
Bowanl
Cow.
Rewadd
Cost
BallCirske
0.54
0.6
0.69
3.00
0.64
CarCiide
0.58
3.74
0.50
0.84
0.75
0.95
0.63
1.89
0.74
2.18
0.71
0.33
0.49
3.14
0.65
0.55
PointBution
0.22
2.02
0.16
1.10
0.46
1.57
0.40
2.66
0.43
2.47
0.55
4.30
0.15
1.51
0.11
0.55
PointPuhh
0.18
0.91
0.80
021
0.23
0.99
0.59
0.03
0.02
1.18
061
2.16
0.51
2.11
D.6 Ablation studies
To investigate the contribution of each component of our algorithm, including the weighted Bellman regularizer, the aggression-limited objective, and the no-regret policy optimization (which together guarantee our theoretical results), we performed an ablation study in the tabular setting. The results, presented in Table 5, indicate that the weighted Bellman regularization ensures the safety of the algorithm, while the aggression-limine limited objective fine-tunes the algorithm to achieve higher rewards without compromising safety.
Table 5: Ablation study under tabular case (cost limit is 0.1) over 10 repeat experiments
Components
cost
reward
ALL
0.014 +0.006
0.788 = 0.004
W/O no- no-regret policy optimization
0.014 = 0.006
0.788 + 0.004
W/O legression-limited objective
0.014 + 0.006
0.788
0.005
W/O Weighted Bellman regularizer
0.323 = 0.061
0.684
= 0.017
D.7 Sensitivity Analysis of Hyper-Parameters
We provide the rewards and costs under different sets of 8, = 3, E (1,0.5, 0.05} and e
{00 1J, [0. 2], []] (since our À only increases, the closed interval here represents the initial value
22

---

1oward
63-
6:0
613
425 20/4 60%
2001
805
Figure 4: Sensitivity Analysis of Hyperparameters in the Tabular Case. The left figure illustrates
tests conducted with various 3 values (For the sake of discussion, we denote 8 = 3, = (2)) with
A = 2]. while the right figure presents tests across different ranges of À with B, 3. 2.0.
and the upper bound of demonstrate the robustness of our approach in the tabular setting in Figure 4. We can observe that the performance is almost the same under different sets of parameters and different qualities of behavior policies.
23

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in Abstract and Intriduction reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been properly discussed in Conclusion and in Section 4.

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the assumptions have been properly stated. The complete proofs are in Appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Our experimental results are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ SguideesCooeeSumiiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: pooodeeummissionl
/ //nips. //wips.cc/publll/// https//cpppbclcguueeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
26

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The Experimental setting and details are Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the experiment, we used multiple random seeds ensure the statistical significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specified the computational resources we used in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
27

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ttttps://neurppi cc/public/ cettii////ieeeeee cecuiieleelleeees
Answer: [Yes]
Justification: This paper is theoretical in nature, and it has been conducted with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper has tremendous positive societal impact as it develops RL algorithms with provable safety guarantee using offline data. Such guarantee is essential for practical implementation of RL algorithms.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
28

---

Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi whrodeewthhheee... com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
29

---

Answer:
[N]]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "fJ3CKhNdJc", "reviewer": "Reviewer_1bfE", "summary": "This paper introduces a principled approach for safe offline reinforcement learning (RL), aimed at robustly optimizing policies beyond a given reference policy, particularly when constrained by the limited data coverage of offline datasets.\nThe traditional constrained actor-critic methods face challenges including (1) coping with insufficient data coverage, (2) ensuring robust policy improvement, and (3) facilitating computationally efficient actor optimization.\nTo address these limitations, this study presents the Weighted Safe Actor-Critic (WSAC) framework. WSAC incorporates (1) a pessimistic bias through a weighted average Bellman error, (2) theoretical assurances for robust policy improvement, and (3) an efficiency advantage over traditional primal-dual optimization methods.\nWSAC employs an aggression-limited objective function, which discourages unsafe policies, relying on less stringent assumptions compared to prior methodologies. Furthermore, WSAC leverages the reference policy as a no-regret policy optimization oracle, allowing for safe policy training.\nThe efficacy of WSAC is demonstrated across four benchmark environments: BallCircle, CarCircle, PointButton, and PointPush. The results indicate that WSAC effectively optimizes policies to maximize cumulative rewards while maintaining cumulative costs below predefined thresholds.", "questions": "Q1.  Could you elaborate on the sensitivity of the hyperparameters such as $\\beta_c$, $\\beta_r$, $\\lambda$? Understanding their influence on the model's performance and robustness would be beneficial, especially in varying training conditions.\n\n(Minor Comments)\n\n1. In obj 2, the Bellman error coefficients $\\beta$ used in the reward and cost constraints appear to have different values, indicated by $\\beta_c, \\beta_r$. To avoid confusion, it would be prudent to denote these coefficients separately throughout the manuscript to reflect their distinct roles and values.\n\n2. The typo in line 231 : \"WSAC sovles\" →  \"WSAC solves.\"", "limitations": "The authors addressed their limitations in Section 4 and Conclusion and the broader societal impact in Checklist #10: Broader Impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is well-grounded in rigorous theoretical principles that effectively support the proposed WSAC method. \nThe method's foundation on pessimistic value estimation and robust policy improvement is both mathematically sound and appropriate for the challenges of safe offline RL.\nFurthermore, the integration of an adversarial training component within the actor-critic architecture introduces a novel strategy for mitigating common issues such as insufficient data coverage in offline RL. \nThis significantly bolsters the robustness of the resulting policies against shifts in data distribution, a crucial factor for applications in real-world scenarios.\nBy addressing the critical issue of safety in policy optimization with a computationally efficient approach, I believe that the paper makes a substantial contribution to moving the field towards practical, deployable reinforcement learning systems capable of addressing real-world challenges.", "weaknesses": "The proposed Weighted Safe Actor-Critic (WSAC) method in this submission is contingent upon the availability of an explicit reference policy, such as a behavior policy derived from the offline dataset. \nThis requirement could make training difficult in scenarios where extracting a reliable reference policy from the offline data is challenging, particularly for algorithms aiming to be behavior-agnostic in offline RL settings.\n\nAdditionally, the authors claim in line 239 that \"Our approach is very computationally efficient and tractable compared with existing approaches.\" However, the absence of empirical evidence, such as wall-clock time comparisons, to substantiate this claim weakens their argument. \nProviding such comparative data would significantly strengthen their case for computational efficiency.\n\nMoreover, the paper does not include ablation studies to elucidate the contributions of the three key components of WSAC: (1) weighted Bellman error, (2) aggression-limited objective, and (3) no-regret policy optimization using a single reference policy. Identifying which of these components is most critical to performance enhancement would provide clearer insights into the framework's effectiveness and areas for potential improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper! We appreciate your support and comments. Please find our point-by-point response to your questions below.\n\n-  **Response to the reference policy:** We would like to mention that extracting the behavior policy from an offline dataset is not difficult with behavior cloning (BC). In particular, we can estimate the learned behavior policy $\\hat{\\pi}\\_\\mu$ as follows: $\\forall s \\in D, \\hat{\\pi}\\_\\mu(a \\vert s) \\leftarrow \\frac{n(s,a)}{n(s)}$, and $ \\forall s \\notin D, \\hat{\\pi}\\_\\mu(a \\vert s) \\leftarrow \\frac{1}{\\vert A \\vert} $, where $n(s,a)$ is the number of times $(s,a)$ appears in the offline dataset $D$. Essentially, the estimated BC policy matches the empirical behavior policy on states in the offline dataset and takes uniform random actions outside the support of the dataset. It is easy to show that the gap between the learned policy $\\hat{\\pi}\\_\\mu$ and the behavior policy $\\pi_\\mu$ is upper bounded by $ \\min \\lbrace1, \\vert S \\vert / N \\rbrace$ ([R1, R2]). We can have a very accurate estimate as long as the size of the dataset is large enough. In addition, in many applications such as networking, scheduling, and control problems, there are existing good enough reference policies. In these cases, a safe robust policy improvement over these reference policies has practical value.\n\n\n[R1]: Kumar, Aviral, Joey Hong, Anikait Singh, and Sergey Levine. \"Should i run offline reinforcement learning or behavioral cloning?.\" In International Conference on Learning Representations. 2021. \n\n[R2]: Rajaraman, Nived, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. \"Toward the fundamental limits of imitation learning.\" Advances in Neural Information Processing Systems 33 (2020): 2914-2924.\n\n- **Response to computationally efficient:** We thank the reviewer for pointing out our statements on computational efficiency compared to existing approaches. We claim our approach is *efficient* and *tractable* compared to [19, 26] mainly because: [26] requires two FQI inner loops for policy improvement and three additional inner loops for policy evaluations, while [19] also requires an inner loop for offline policy evaluation. However, our algorithm does not have any inner loop for extra OPE. To demonstrate the efficiency, in the following Table (Table 1), we report the training times of our method and other baselines for the Car Goal environment using one NVIDIA GeForce RTX 3080 Ti. We observe that our practical version still has a faster training time, which is very time-efficient compared to others. We will make this clear in the revision.\n\n|         | BEARL | CPQ | CDT | COptiDICE | WSAC |\n|:---------|:-------:|:-----:|:-----:|:------------:|:------:|\n| Time    |  121  | 118 | 465 | 117        | 115  |\n\n**Table 1: Training Time (seconds) for 200 steps**\n\n- **Response to ablation studies:**  The weighted Bellman regularize, aggression-limited objective, and no-regret policy optimization together guarantee our theoretical results. We did an ablation study in the tabular setting and the results can be found in the following table (Table 2). The results of the ablation study indicate that the weighted Bellman regularization and no-regret policy optimization ensure the safety of the algorithm, while the aggression-limited objective ensures the algorithm to achieve higher rewards without compromising safety. \n| Components                                                   | Cost  | Reward |\n|--------------------------------------------------------------|-------|--------|\n| ALL                                                          | 0.016 | 0.766  |\n| W/O  no-regret policy optimization    | 0.016 | 0.766  |\n| W/O  Aggression-limited objective  | 0.016 | 0.765  |\n| W/O   Weighted Bellman regularize | 0.181 | 0.624  |\n\n**Table 2: Ablation study under tabular case (cost limit  = 0.1)**\n\n - **Response to hyperparameter sensitivity of $\\beta$ :**  Note that our main result remains the same for SRPI as long as $\\beta \\geq 0$, indicating that our approach is highly robust across a wide range of $\\beta$ values. We use different $\\beta_r$ and $\\beta_c$ in the practical version because it allows us to easily make a trade-off between rewards and costs, as different environments have varying sensitivities to rewards and costs. To address the reviewer's comments about hyperameter sensitivity, we provide the rewards and costs under different sets of $\\beta_r=\\beta_c\\in \\{1,0.5,0.05 \\}$ and $\\lambda\\in\\{[0,1],[0,2],[1,2]\\}$ (since our $\\lambda$ only increases, the closed interval here represents the initial value and the upper bound of $\\lambda$) to demonstrate the robustness of our approach in the tabular setting in Figure 1 (Please see the uploaded file). We can observe that the performance is almost the same under different sets of parameters and different qualities of behavior policies. We will add more details in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "530wBKcwf1", "reviewer": "Reviewer_U4TL", "summary": "This paper proposes weighted safe actor-critic, and provides corresponding theoretical analysis on its optimal statistical rate. Some interesting technical tools were introduced. The authors also implement a practical version of WSAC and evaluate it against SOTA offline safe RL baselines in continuous control tasks.", "questions": "(1) **Clarification of \"Adversarial\"**: I'd like to have a clarification of the term \"adversarial\" in this offline safe RL problem. Do the authors mean \"adversarial\" in that the cost critic always update cost critic via optimism and reward critic via pessimism? Since there are other formats of adversarial robustness in other components of safe RL [1], I may be helpful if the authors could clarify it in the early stage of this paper. \n\n(2) **Finite selection of W**: the current WSAC algorithm prototype only considers a discrete selection of $w$, can they be arbitrarily assigned for different offline datasets and environments? Intuitively if an arbitrary $w$ is close enough to its neighbor $w$ in set $\\mathcal{W}$, we can still provide analytical bound of performance under slightly different $w$. \n\n(3) **Lack of discussion of assumption gap in the pratical version**: the authors may describe how certain assumptions may not hold in practice, as they are some seemingly relatively strong assumptions in the theoretical analysis, like approximate realizability, though some of them have already got loosened. Also, some experiment details (e.g. the behavior policy or the oracle policy) can be discussed in the appendix to provide more contexts in how WSAC can help practically. \n\n(4) **Capability under sparse-cost setting**:  in many real-world applications, safety violations will occur only in long-tail events. Can the reweighting scheme also address such long-tail cases in the cost critic learning?  Can the current WSAC framework can be extended for this kind of analysis based on the weighting technique? \n\n(5) **Extension of the current framework to multi-constraint settings**: in real-world applications, there might be multiple objectives and constraints, can the WSAC framework adapt to similar settings? \n\n(6) **Selection of weight W in the experiments**: Compared to algorithm 1, the practical implementation of WSAC seems to miss $\\mathcal{W}$, how is this importance weight computed in practice? \n\n\n> [1] Liu, Zuxin, et al. \"On the robustness of safe reinforcement learning under observational perturbations.\" *arXiv preprint arXiv:2205.14691* (2022).", "limitations": "The authors have clearly defined the scope and discussed the limitations of this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper addresses offline safe RL with adversarial trained weighted AC framework, showing its optimal statistical convergence rate. \n\n(2) Under the perfect function approximation assumption, the authors show WSAC outperforms any reference policy while maintaining the same level of safety. Besides, the theoretical finding on safe robust policy improvements bring insight to the offline safe RL methods. \n\n(3) Empirically, the authors provide a comparison to a set of baselines in OSRL benchmarks.", "weaknesses": "See more discussion in the question parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper! We appreciate your support and comments. Please find our point-by-point response to your questions below.\n\n- **Response to clarification of \"Adversarial\":** The adversarial training in this paper is designed based on the concept of relative pessimism [11,4,53], which aims to optimize the worst-case relative performance over uncertainty. In particular, we adversarially train the critic to find weighted Bellman-consistent scenarios where the actor is inferior to the reference/behavior policy (Eq. (2) and Eq. (4)). Note that we consider an offline setup, hence, we cannot access any new data, rather whatever data is available, we have to find policy based on that. Without adversarial training, if some state-action pairs are not covered in the dataset, one can set (incorrectly) a very high $ Q$ value without affecting the training loss. The adversarially trained critic addresses this issue by setting the lowest possible $Q$-value avoiding setting a high value for our-of-distribution values. We will clarify and discuss the differences between our approach and the paper mentioned by the reviewer in the final version.\n\n- **Response to finite selection of $W$:** The selection of $W$ is not arbitrary: it needs to be chosen by maximizing the importance-weighted average Bellman error (Eq. (2) and Eq. (4)). This is crucial in the formulation because maximization over $w$ in the importance-weighted average Bellman regularizer ensures that the Bellman error is small when averaged over measure $\\mu \\cdot w$ for any $w \\in W$. This can control the suboptimality of the learned policy and guarantee the optimal statistical rate of $1/\\sqrt{N}.$\n\n- **Response to lack of discussion of assumption gap in the practical version:**  For the assumptions made in our paper, as far as we know, we require minimal assumptions in offline RL (as shown in Table (1)), especially in safe offline RL. The single-policy coverage required by our approach is far milder than the all-policy coverage assumption as it is impractical to assume that the dataset would contain state-action pairs from unsafe policies. Thus, we achieve our result for a more practical and realistic set of assumptions (and it matches the same set of assumptions for the unconstrained case). For the practical purpose, we think the most concerning part is that approximate realizability in Section 3.2 may not hold in practice. However, in our experience, as we usually use neural networks to approximate the function class $F$, it is likely that the approximate realizability error is small as long as we have a sufficiently rich neural network. The behavior policy is easy to achieve, even if it is not given to us, since extracting the behavior policy from an offline dataset is not difficult with behavior cloning (BC). In particular, we can estimate the learned behavior policy $\\hat{\\pi}\\_\\mu$ as follows: $\\forall s \\in D, \\hat{\\pi}\\_\\mu(a \\vert s) \\leftarrow \\frac{n(s,a)}{n(s)}$, and $\\forall s \\notin D, \\hat{\\pi}\\_\\mu(a \\vert s) \\leftarrow \\frac{1}{\\vert A \\vert}$, where $n(s,a)$ is the number of times $(s,a)$ appears in the offline dataset $D$. Essentially, the estimated BC policy matches the empirical behavior policy on states in the offline dataset and takes uniform random actions outside the support of the dataset. It is easy to show that the gap between the learned policy $\\hat{\\pi}\\_\\mu$ and the behavior policy $\\pi\\_\\mu$ is upper bounded by $\\min\\lbrace 1, \\vert S \\vert / N \\rbrace$ [R1, R2]. We can have a very accurate estimate as long as the size of the dataset is large enough. We will add more details in the revision.\n\n[R1]: Kumar, Aviral, Joey Hong, Anikait Singh, and Sergey Levine. \"Should i run offline reinforcement learning or behavioral cloning?.\" In International Conference on Learning Representations. 2021. \n\n[R2]: Rajaraman, Nived, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. \"Toward the fundamental limits of imitation learning.\" Advances in Neural Information Processing Systems 33 (2020): 2914-2924.\n\n\n- **Response to capability under sparse-cost setting:**  Maximizing the importance-weighted average Bellman regularizer aims to control the Bellman error. We believe this can also be applied to the long-tail constraint case. Our results can be generalized to a high probability bound (might be loose) using Markov's inequality, and it is possible to show some results under the CVaR objective. Of course, rigorous proofs and experiments are needed. We will explore these interesting directions in the future.\n\n\n- **Response to extension of the current framework to multi-constraint settings:**  Yes, the current approach is readily to generate to the case with multiple constraints by simply considering multiple constraints in the objective function: $f_r^k(s,a) - \\sum\\_{i=1}^I \\lambda\\_i \\lbrace f_{i,c}^k(s,a) - f_{i,c}^k(s.\\pi_{ref}) \\rbrace\\_+,$ where $f\\_{i,c}^k$ is the $i$th constraint. We only have to tune $\\lambda_i$, and $\\beta_{c,i}$ values corresponding to constraint $i$. We will add some discussions in the revision. \n\n\n- **Response to the selection of weight W in the experiments:** In experiments, we take $\\mathcal{W} = \\{0, C_\\infty\\}$ for computation effectiveness. Then we can reduce $\\epsilon \\_{D}(\\pi, f)$ (Eq. (4)) to $C\\_{\\infty}E\\_{D}[(f(s,a)-r-\\gamma f(s',\\pi))^2]$. In the environment, we simply set $C_\\infty$ to be 1 in the neural case and 5 in the tabular setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccEXUefZ8v", "reviewer": "Reviewer_Tiai", "summary": "For safe RL methods, a desired property is Safe Robust Policy Improvement(SRPI), which means the learned policy is always at least as good and safe as the baseline behavior policies. But it's not achieved yet.\n\nAlso, the traditional Actor-Critic framework may suffer from insufficient data coverage, which may fail to provide an accurate estimation of the policy for unseen states and actions. To address the issue, [45] and [11] use absolute pessimism or relative pessimism. However, his kind of approach fails to achieve the optimal statistical rate of $\\sqrt{N}$. For addressing efficient policy improvement, the most commonly used approach for addressing safe RL problems is primal-dual optimization, but this method requires all policy concentrability, that is, the dataset must cover all possible strategies, which is impractical for the safe-related dataset.\n\nIn contrast, the authors propose an aggression-limited objective function,  the high-level intuition behind it is that by appropriately selecting a 𝜆,  all unsafe policies are penalized.  As a result, the policy that maximizes the objective function is the optimal safe policy. This formulation is fundamentally different from the traditional primal-dual approach as it does not require dual-variable tuning, and thus, does not require all policy concentrability.\n\nBeyond that, the proposed method also proved to achieve SRPI.\n\n[45] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning.\n\n[11]Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning", "questions": "The paper looks good. And there is a suggestion for future directions.\n\nBridge the gap between theory and practice. Current RL methods have some bottlenecks like long-horizon tasks, safety, sample efficiency, etc.. Most of the methods to solve these bottlenecks come out of intuition instead of theory deduction. And some methods leverage external tools like foundation models(e.g. LLM), and control theories. It might be interesting to explain why these intuition or external tools work in view of theories.", "limitations": "The authors did not provide limitations.\n\nA possible limitation may come from the experiment on PointPush, in which the simple BC's reward outperformance proposed method without sacrificing too much safety.  \n\nAnd as stated in weakness, safety is expressed purely by the cost function, which sometimes is hard to get in the real world.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The writing logic is clear and reasonable. This paper is written with solid insights and emphasizes research gaps and innovations.\n\n2. The authors conducted wide-range experiments and comparisons with other methods. And in terms of safety, the proposed method achieved SOTA performance.\n\n3. There are no obvious red flags or drawbacks in this paper.", "weaknesses": "1. In this paper's setting, safety is measured purely by cost, which is not always practical. e.g. in the real world, the cost function could be implicit or impossible to get.  \n\n2. The authors could try to combine the proposed method with other RL methods for a certain application to further justify its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ positive evaluation of the novelty of this paper. The current formulation can be applied to the case with $0-1$ cost, indicating whether a constraint is violated or not at each step. Nevertheless, if we only get feedback over the entire trajectory whether it is safe or not rather per step feedback, it is an indeed interesting future research direction on how to address such a scenario. In fact, even in the online setting such a scenario is yet to be resolved. We add a discussion. \n    \n We strongly agree with the reviewer on the gap between theories and practical algorithms in RL and safe RL. Our approach is motivated to take a step towards bridging the gap as we consider offline setup and we only need single policy coverability for the theoretical results. Regarding the comments on external tools like LLMs, one possible direction could be to consider the cost preference (like RLHF) instead of cost function observations and determine whether it is possible to design an algorithm with provable guarantees of safety.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cRgwS0KM7o", "reviewer": "Reviewer_HDvF", "summary": "The Weighted Safe Actor-Critic (WSAC) is a new Safe Offline Reinforcement Learning algorithm designed to outperform any reference policy while ensuring safety with limited data. It uses a two-player Stackelberg game to achieve optimal convergence and safe policy improvements. In practical tests, WSAC surpasses baselines in continuous control environments.", "questions": "- What about the performance under different cost limits? The average cost in Table 2 does not adequately reveal the safety of the algorithm.", "limitations": "n/a", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The ability of WSAC to outperform the behavior policy over a wide range of hyperparameters is a crucial property for practical use.\n- The author provides theoretical proof.", "weaknesses": "- Given that I haven't examined the mathematical details, I find that many of the assumptions and proofs of key theorems in the paper resemble those in ATAC [1]. The primary differences are the authors' focus on the safe offline RL setting and the inclusion of a cost value in their theory. However, the use of a primal-dual approach in the algorithm's implementation may introduce training stability issues [2]. From both theoretical and practical implementation perspectives, it is difficult to identify novel insights in the paper.\n- The author needs to compare more state-of-the-art baselines, such as CDT [3] and FISOR [2].\n- Line 34 contains a duplicate citation.\n\n[1] Cheng, Ching-An, et al. \"Adversarially trained actor critic for offline reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2022.\n\n[2] Zheng, Yinan, et al. \"Safe offline reinforcement learning with feasibility-guided diffusion model\". *International Conference on Learning Representations* (2024).\n\n[3] Liu, Zuxin, et al. \"Constrained decision transformer for offline safe reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on our paper. Please find our point-by-point response to your questions below.\n\n- **Response to contributions:** We respectfully ask the reviewer to evaluate our theoretical contributions. We would like to mention that our approach has significant differences from ATAC. It is also unfair to say the \"primary differences are the authors' focus on the safe offline RL setting and the inclusion of a cost value in their theory,\" since all existing studies in safe RL consider the cost function in the formulation of RL, it is the default setting. This consideration makes the problem entirely different and more difficult than the unconstrained case; i.e., the optimal policy will no longer be a greedy policy, how to ensure safe learning and obtain a safe policy, and balancing the reward and cost is extremely important. We are the **first** to present a Safe Robust policy improvement over **any** reference policy with an optimal statistical rate. Following the approaches in ATAC can only guarantee a sub-optimal rate, and it is highly non-trivial to extend their results to the constrained setting. \n\n- **Response to compare with baselines:** We thank the reviewer for pointing out the papers CDT and FISOR. The reason we did not incorporate these two algorithms as additional baselines is that their setups do not align well with ours. CDT introduces additional information, target reward, and target cost, during evaluation. These pieces of information are not required in the evaluation process of WSAC and the other baselines we have chosen. Additionally, FISOR considers a different setting from ours as they consider a different type of constraint which is defined for any possible state. For the addition simulations with different cost limits, we report the average performance of our results and other baselines in the following Table (Table 1) under cost limits [10,20,40] for BallCircle and CarCircle and [20, 40, 80] for PointButton and PointPush following the standard setup ([R1]). We can observe that WSAC maintains safety across **all** environments, and WSAC's performance is comparable to or even better than the best baseline in each environment. We will add the details in the final revision.\n\n|   | BC Reward $\\uparrow$ | BC Cost $\\downarrow$ | Safe-BC Reward $\\uparrow$ | Safe-BC Cost $\\downarrow$ | CDT Reward $\\uparrow$ | CDT Cost $\\downarrow$ | BCQL Reward $\\uparrow$ | BCQL Cost $\\downarrow$ | BEARL Reward $\\uparrow$ | BEARL Cost $\\downarrow$ | CPQ Reward $\\uparrow$ | CPQ Cost $\\downarrow$ | COptiDICE Reward $\\uparrow$ | COptiDICE Cost $\\downarrow$ | WSAC Reward $\\uparrow$ | WSAC Cost $\\downarrow$ |\n|-------------|----------------------|----------------------|---------------------------|---------------------------|-----------------------|-----------------------|------------------------|------------------------|-------------------------|-------------------------|-----------------------|-----------------------|---------------------------|---------------------------|-----------------------|-----------------------|\n| BallCircle  | 0.74                 | 4.71                 | 0.52                      | 0.65                      | 0.77                  | 1.07                  | 0.69                   | 2.36                   | 0.86                    | 3.09                    | 0.64                  | 0.76                  | 0.70                      | 2.61                      | **0.74**                  | **0.51**                  |\n| CarCircle   | 0.58                 | 3.74                 | 0.50                      | 0.84                      | **0.75**                  | **0.95**                  | 0.63                   | 1.89                   | 0.74                    | 2.18                    | 0.71                  | 0.33                  | 0.49                      | 3.14                      | 0.65                  | 0.55                  |\n| PointButton | 0.27                 | 2.02                 | 0.16                      | 1.10                      | 0.46                  | 1.57                  | 0.40                   | 2.66                   | 0.43                    | 2.47                    | 0.58                  | 4.30                  | 0.15                      | 1.51                      | **0.11**                  | **0.55**                  |\n| PointPush   | 0.18                 | 0.91                 | 0.11                      | 0.80                      | 0.21                  | 0.65                  | **0.23**                   | **0.99**                   | 0.16                    | 0.89                    | 0.11                  | 1.04                  | 0.02                      | 1.18                      | 0.07                  | 0.61                  |\n\n\n**Table 1: The normalized reward and cost of WSAC and other baselines for different cost limits. Each value is averaged over 3 distinct cost limits, 20 evaluation episodes, and 3 random seeds.**\n\n[R1] uxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu,114\nTingnan Zhang, Jie Tan, et al. \"Datasets and benchmarks for offline safe reinforcement learning\". arXiv preprint115\narXiv:2306.09303, 2023.116", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their thoughtful evaluations. In this global rebuttal, we point out our main contributions and address the common concerns of the reviewers. We respond to each individual reviewer in each individual rebuttal separately as well.

**Our Contributions**:

* We consider an offline constrained MDP (CMDP) problem. Unlike the unconstrained MDP, here, one needs to learn a policy that maximizes reward while simultaneously satisfying constraint using only offline data. We prove that our algorithm, which uses weighted Bellman error, enjoys an optimal statistical rate of $1/\sqrt{N}$ under partial data coverage assumption.  Note that all the existing approaches for offline safe RL/CMDP (see Table 1) require **all**-policy concentrability which is not possible in practice. In particular, an offline database may not contain state-action pairs covered by every unsafe policy. We achieve our result using a more practical set of assumptions with only single policy concentrability. *This is the first work that achieves such a result using only single-policy $\ell_2$ concentrability.*

* We propose a novel offline safe RL algorithm, called Weighted Safe Actor-Critic (WSAC), which can robustly learn policies that improve upon any behavior policy with controlled relative pessimism. We prove that under standard function approximation assumptions, when the actor incorporates a no-regret policy optimization oracle, WSAC outputs a policy that never degrades the performance of a reference policy (including the behavior policy) for a range of hyperparameters. *This is the first work that provably demonstrates the property of SRPI in offline safe RL setting.*

 *  We point out that primal-dual-based approaches [19 ] must require all-policy concentrability assumption. Thus, unlike, the primal-dual-based approach, we propose a novel rectified penalty-based approach to obtain results using single-policy concentrability. *Thus, we need novel analysis techniques to prove results compared to existing approaches.*

* Furthermore, we provide a practical implementation of WSAC following a two-timescale actor-critic framework using adversarial frameworks similar to [11, 53 ], and test it on several continuous control environments in the offline safe RL benchmark [ 31 ]. WSAC outperforms all other state-of-the-art baselines, validating the property of a safe policy improvement. In particular, from Table 2 (in our paper), it is clear that across all the environments, WSAC is the **only** algorithm that achieves safety and yet has achieved a better or similar reward in most of the environments. Thus, our proposed approach has contributed significantly in both theoretical and practical fronts for safe offline RL. 

**New Results**:

* We have now achieved new empirical results. In particular, we evaluate the sensitivity of the hyper-parameters $\lambda, \beta_r, \beta_c$ and observe that our algorithm's performance is robust under different sets of parameters (see the attached Figure).

* We have now conducted an empirical evaluation of the computational efficiency of our proposed approach and observed that it takes a smaller time compared to the state-of-the-art approaches (see Table 1 in the response [here](https://openreview.net/forum?id=82Ndsr4OS6&noteId=T2H496Awkn))

* We have now conducted Ablation studies (see Table 2 in the response [here](https://openreview.net/forum?id=82Ndsr4OS6&noteId=T2H496Awkn))

* We have now compared our approach with another baseline mentioned by the reviewers. We have outperformed the approach. (Please see Table 1 in the response [here](https://openreview.net/forum?id=82Ndsr4OS6&noteId=fBIu70VHUw))

Meta-review: This paper introduces an approach for safe offline reinforcement learning (RL), by introducing Weighted Safe Actor-Critic (WSAC), which incorporates pessimistic bias through a weighted average Bellman error and theoretical guarantees. The efficacy of WSAC is demonstrated across four benchmark environments: BallCircle, CarCircle, PointButton, and PointPush; the results are promising. Although the work seems to be a little bit similar to ATAC, the reviewers overall all recommend acceptance as the work is both theoretically grounded and practically significant.
Decision: Accept (poster)