VOCE: Variational Optimization with Conservative Estimation for Offline Safe Reinforcement Learning
Authors: Jiayi Guan
Guang Chen
Jiaming Ji
Long Yang
Ao Zhou
Zhijun Li
changjun jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline safe reinforcement learning
Pessimistic conservative estimation
Variational optimization
Reinforcement Learning

Abstract: Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly in offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and robotics. However, the influence of safety constraints and out-of-distribution (OOD) actions have made it challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose a Variational Optimization with Conservative Eestimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions to make the optimization of policies more flexible. Subsequently, we utilize pessimistic estimation methods to estimate the Q-value of cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive experiments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety.

VOCE: Variational Optimization with Conservative Estimation for Offline Safe Reinforcement Learning
Jiayi Guan Tongji University guanjiayi @ www.gji..c..oo edu.cn
Guang Chen' Tongji University guangchen @ tongji.edu.cn
Jiaming Ji Peking University jiamg.ji@ gmail.com
Long Yang
Ao Zhou
Zhijun Li
Peking University
Tongji University 2211107 @ tongji.edu.cn
Tongji University zjli ieee.org
yanglong001 @ pku. pku.edu.cn edu.cn
Changjun Jiang Tongji University cjjiang @ tongji.edu.cn
Abstract
Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and roboties. However, the influence of safety constraints and out-of-distribution (OOD) actions have made challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose Variational Optimization with Conservative Estimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions make the optimization of policies more flexible. Subsequently, simistic estimation methods to estimate the Q-value cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive ex- periments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety. Our code available at nigithub.
utilize pes-
github.VOOCB VOCE
Introduction
Reinforcement learning (RL) algorithms have made remarkable achievements various fields such robot control [1+3] and strategy games [ However, limited by the online and trial-and-error nature, standard RL challenging to apply to dangerous and expensive scenarios 77 Offline safe RL is a promising and potential approach to tackle the above problem, which learns policy satisfying safety constraints from pre-collected offline datasets without interacting with the environment [10
Since offline safe RI. methods aim to learn policies that satisfy safety constraints from offline datasets, which requires the agent to not only comply with safety constraints but also consider the influence (OOD) actions [2(15). This makes it difficult for the algorithm to learn policies that achieve high reward returns while satisfying safety constraints [6]. There are two main
of
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

approaches to deal with the aforementioned challenges: one is based on linear programming [17 18).
and the other is based on exploration and evaluation 2T. Linear programming provides way to
research offline safe RI. problems, but it heavily depends on the F-divergence or KI. Kl.-divergence to
constrain the distance between the optimal distribution and the sample distribution [7]. Therefore,
is difficult for linear programming to eliminate the extrapolation errors caused by OOD actions [10]
I... The exploration evaluation approaches are to introduce conservative Q-value estimation under the actor-critic framework. Although exploration evaluation approaches avoid the overestimation issue through lower bound Q-values, they prematurely terminate trajectories that exceed the constraints during policy updates [16]. The exploration evaluation method with strict constraints on the sample space makes it challenging to search for reward maxxmizzng policies. In conclusion, existing methods face challenges in learning policies from offline datasets that maximize reward while satisfying safety constraints, particularly when considering the impact of extrapolation errors.
In this work, to solve the constrained policy optimization problem and eliminate the extrapolation errors caused by OOD actions, we propose variational optimization with conservative estimates for the offline safe RL algorithm. Concretely, probabilistic inference, enhancing the flexibility of policy optimization by replacing parameterized policies with variational distributions. Furthermore, to avoid extrapolation errors caused by OOD actions, we derive upper and lower bounds for Q-values and estimate the Q-values of costs and rewards based these bounds. Finally, adopt supervised learning approach to train parameterized policy network according to the variational distribution of policies. The main contributions of this work are listed follows:
reframe the objective of offline safe RL using
We utilize probabilistic inference to address the problem of offline safe RL, which intro- duces non-parametric variational distributions to replace parameterized policies, providing increased flexibility for optimizing safe policies in offline datasets.
We derived upper and lower bounds for Q-value estimation using the pessimistic estimation approach. Subsequently, we utilized these bounds to estimate the Q-value of costs and rewards, respectively, to avoid extrapolation errors caused by OOD actions.
We carry out extensive comparative experiments, and the results indicate that the VOCE algorithm outperforms state-of-the-ar algorithms, especially in terms of safety.
2 Preliminaries
Constrained Markov decision processes (CMDP) provide a theoretical framework to solve safe RL problems (22). where the agent is cost-bounded by safety constraints, A CMDP is defined as a tuple (S, A, C, P. r. Po: 7), where E R is the state space, A E TRm is the action space, P:SxA x S > [0, 1] is the transition kernel, which specifies the transition probability p( 1ss, 0z) from state 8 to state 81+1 under the action 0r, r S
-> IR represents the reward function, C is the set of costs {c: S x A + R,, i 1, ... m for violating
constraints, 7 (0. 1] is the discount factor, and po : + [0, 1] is the distribution of initial states. A policy क maps a probability distribution from state 8, to action dz, We utilize xa to denote the parameterization of the policy with parameter 0. In safe RL, the goal to maximize the cumulative reward while satisfying safe constraints:
N²' arg max B, most Σ (rr(ss.
s.t.
E
Σ rr(ess
Sc.
where the {80, ao } denotes the trajectory. ã is the i-th safe constraint limit.
In the offline RL setting, we learn policy according to the offline dataset D collected by one or more data-collection policies, and without online interaction with the environment (D = {(5e, Te, (t)) ( )). Although this way of not interacting with the environment brings lot of advantages, the offline dataset can not cover all action- state transitions, and the policy evaluation step actually utilizes the Bellman equation of single sample (23.. This makes the estimated Q-value vulnerable to OOD actions, which in turn severely affects the performance of the algorithm [24 25 As shown in Fig. | under the offline setting, off-policy safe RL methods neither learn high-reward policy nor guarantee safety. Additionally, although offline RL methods obtain high rewards under safe expert data, is difficult to guarantee safety because it directly ignores the cost. We can conclude that both offline RL and safe RL face challenges in learning policies that satisfy safcty constraints

---

from offline datasets. Therefore, it is important to design an algorithm that learns high-rewe reward and satisfies safety constraints, which is the focus of this paper.
Offline RL
Offline RL
Offipolicy Safe RL
Offi Offi-olicy Safe RL
Reswern oppandte
Reward coloods
CRA
T03-Up (070L))
2EMR
8 5
Cos
Codiet
(tep(1
Stepl(1ed)
Step1"led)
1e4)
Figure I: Reward and cost curves of safe RL and off-policy RL under high-reward and safe datasets in Point Button task [26]. The solid line is the mean of the performance, and the shaded line represents the variance of the performance. All performance curves employ more than 3 random seeds.
3 Method
In this section, we present the details of Variationally Optimized with Conservatively Estimated for offline-safe RL algorithm (VOCE), which is the main contribution of this paper. We first reformulate the objective offline-safe RL with probabilistic inference. Based on this, we derived upper and lower bounds for Q-value estimation, and adopted pessimistic and conservative estimation approach to estimate the Q-value of costs and rewards respectively to eliminate the extrapolation error of OOD actions. Finally, employ supervised learning transform the variational distribution into parameterized policy.
3.1 Offline-safe RL as Inference
From the probabilistic inference perspective, the offline safe RI can be viewed as the problem of observing safe actions in states with high rewards. As the probabilistic inference model and notation in [26] 27]. we introduce an optimal variable O represent the event of maximizing rewards. As- suming the likelihood of given trajectory being optimal is proportional to the cumulative discounted reward, the infinite discounted reward formula is expressed as P(O = 1/7) x E, 7'r((,,, at). Since P(O = 1|r) 2 O, we further confirm that the likelihood of given trajectory being optimal is propor- tional to the exponential of the accumulative discounted reward, then we rewrite the infinite discounted temperature parameter, and is short for r(s,, ). Let Px(r) be the probability of the trajectory under the policy π, then the log likelihood of optimality under policy can be written log P.(O- 1) log Σ P(O 1+ (T). Fur- thermore, according to importance sampling and Jensen's 's inequality, we obtain a lower bound of the log-likelihood of optimality under policy π.
reward formula as POO 1|r) x exp (rraar))) The
P(O (T) @(T)
P(O = (+))))))++7))
log (O = 1) = log Brae
2 -Brnglog
@(T)
α E
Σ
v'r (Dx(((( 1-0
where the Q(T) is the auxiliary trajectory distribution. Since offline safe not not only needs to consider maximizing cumulative rewards but also guaranteeing safety, we restrict Q(r) to the constrained feasible distribution space. According to the constraint threshold e; we define the feasible distribution space as:
T A (a(ar)s:): E.
Y' Ca Sa..
1=0
where each q(ae(s)) E D indicates that the action distribution of the sate-condition satisfies the safety constraint. By factorizing the trajectory distribution [27], we express the trajectory distribution
@(T) = =((s)) Il p(s 1 (s, ar)g(a st). ((ar(s)) 2
(4)
120
Pro (7) p(5))
I
p( a (arls))
(5)
220
3

---

where p(so) is the distribution of the initial state so. Substituting Eq. and ( into Eq. to eliminate the transitions p(5 |s,, az), and combining the feasible distribution space of Eq. (3 we obtain the object of offline safe RI. problem as shown in Proposition 3.1
Proposition 3.1. The objective of the offline safe RL problem be defined through probabilistic inference as follows:
20
C(g. T.) = 1B.Ting
((ails)) e e
(6)
t=0
Proposition provides new objective for the problem of offline safe RL. The aforementioned
probabilistic inference method has many optimizations over the previous dual methods in solving the problem of offline safe RL. This approach provides flexibility for optimizing the optimal policy by introducing variational distribution g(aa|s:) to connect the strong correlation between dual variable optimization and policy improvement. Furthermore, the approach decouples the optimal policy optimization problem into independent processes of optimal variational distribution optimization and parameterized policy update. We will introduce these two update processes in detail later.
3.2 Variational Optimization with Conservative Estimates for Offline-safe RL
The previous standard RL and safe RL also employed the above-mentioned idea of probabilistic inference to solve the policy optimization problem [26 28 However, due to the influence of OOD actions in the offline setting, previous RL algorithms based probabilistic inference difficult to address the tasks of offline-safe RL. Therefore, we adopt a pessimistic conservative estimation method improve the offline-safe RL algorithm based on probabilistic inference and obtain a Variational Optimization with Conservative Estimation algorithm (VOCE). We divide the VOCE into two steps: conservatively estimated variational optimization and parameterized policy update.
3.2.1
Conservatively Estimated Variational Optimization
The goal of the conservatively estimated variational optimization is to optimize the optimal variational distribution q(o, (st) with high reward while satisfying the safety constraints. In this step, we perform the maximization of the variational distribution q(ar)))) by fixing the policy ra(al)a (ac]se). According to
proposition 3.1 we write the objective of solving the optimal variational distribution as:
((g) p(oo) (e Q" (st, a.) DD ( (-)0))))))))))
s.t.TE [e(asls)) Q (#s, at) Se. (7)
The optimization objective of the above variational distribution can be regarded KL-regularized constrained optimization problem. However, since the expected reward term (sta Q" (st, az) may be different scales depending the environment, it is difficult set an appropriate penalty coefficient a for different environments. Therefore, we introduce hard constraints to replace the regularization term of the KL-divergence and rewrite the optimization objective of Eq. (7)
Σ ((arls)) (st, (st. s.t. Ep(ev) [ q(aelss) (se, (se, S Ci.
max Ex(oo)
(8)
Bp) ( st)) Se, Ed(a)))) = 1, DD
where e is the KL-divergence threshold between the variational distribution g(ar|s)) and the param- eterized policy we (0r (s)). To solve the above-constrained problem (8). we replace the parametric ((a.]s.) with the non-parametric to avoid performance degradation caused by approximation er- rors [29]. Furthermore, we apply the Lagrange multiplier method transform the above-constrained problem into an unconstrained problem. In the non-parametric form q(a, (s)), since the objective function linear and the constraints are convex, the constrained optimization problem shown in Eq. ( is convex optimization problem. Furthermore, through mild assumptions, we can obtain the strong dual form of Eq. (
Assumption 3.2. (Slater's condition). There exists a variational distribution that satisfies the safety constraints q(a/11) (8)) E 2 within the current policy trust region D (xL(g(aa|s)))]o//

---

Lemma 3.3, Under Assumption 3.2 there exists strong duality between the primal problem in and the unconstrained problem in Eq.
Eq.
plloo) Σ ((aels))Q' (st, x² X20.920.5 920
min max Baxxpplay
Σ ((xa)ss)
at
8 Σ 9(ar|s)) Disslyaaassl)llloaaaa -
where X, 77 are the Lagrange multipliers of the above multiple constraints. Based the strong duality of the unconstrained problem in Eq. 9.. we further derive a closed-form solution q (ag (x) for its internal maximization, shown in Proposition 3.4 Proofs and discussions are in Appendix B.
Proposition 3.4. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (9) is given as:
QT
exp (-
77
6
(ac(s:) ne(ar(s)) exp
(10)
where Qre(st, ax) - ("( d+) (st,
By substituting the closed-form solution (T) into Eq. ( and eliminating similar terms, we obtain
the convex minimization problem shown in Proposition 3.5 Subsequently, solve for the dual variables and X using Eq. (1I). Proofs and discussions are in Appendix B.2
Proposition 3.5. The X, 7 are dual variables and are solved via the convex optimization problem of the following formula.
( at) x² + ne
L(). min Boiso) log Σ To(as)s;) exp 220.120
(11)
The Proposition 3.4 provides a proposal to solve non-parametric variational distribution g(arls:)
for the given the Q-value Q" (st, ae) and Q (s) In addition, we provide the optimality and uniqueness of the above closed-form q" (ag st) solution under the premise of strong convexity in
the Appendix B.2 Note that it can be seen from Eq. (10) that providing accurate Q" 0z) and are the premise and guarantee for accurately computing the non-parametric variational distribution ((ar(1t), We can find similar conclusions in the online safe RI algorithm CVPO [26]. as described in Proposition 3.4 and 3.5] In the online setting, the empirical Bellman equation is directly applied to iteratively update the Q-value. However, in the offline setting, this approach would lead to clear extrapolation errors Q2 ae) and Q" (st, 0z) due the OOD actions. Furthermore, is difficult to accurately compute the variational distribution g(a: [(a,(s:) according to Eq. (IO).
To eliminate the extrapolation we utilize the pessimistic conservative estimation approach to estimate Q" and Q respectively. Specifically, eliminate the impact of extrapolation error on Q", we need to avoid overestimation of Q" [24]. Therefore, similar to CQL 30 to learn a conservative lower bound of the Q-function choose
caused by the OOD actions during the evaluation of the Q-value,
by additionally minimizing the Q-values alongside standard Bellman error objective, the penalty that minimizes the expected Q-value under special action-state transitions for unknown actions produced by (a:ls). Then, we define the Q-value of the iterative reward function as:
1 Qi-1 + arg min 2 Da [[[Q" (st, ax)) B- Qu(ss, (((ss,,)))* [+) +
[Q" ac)] (12)
where the Fe (x): r(s,, + 72"² (S8+1, dt+1). In Proposition 3.6
we show that Q" (st, a:) converges to lower bound on (st, However, we can tighten this bound if we are interested in V"(s)). We improve the bounds by introducing additional maximization term under the sample of the reward Q-value can be rewritten as:
distribution न... Then the iterative update Eq. (12)
1 Qoti f- arg min
2 [
EaDD ((Q((st.u)) B ( a.)
(13)
K
Q" (t) E anD Q (st,
where x is tradeoff factor. Note that the Eq. 12 and (3 utilize the empirical Bellman operator 8 instead of the actual Bellman Operator y Following the related work [23 31), we employ the
5

---

concentration properties to compute the enror. For any Vst, Or E D, with the probability 2 6, the sampling error can be written as:
|8" ((s). BT (st.. ai)l (1 (1-7)
(14)
where Cr.p.5 is constant depending on the concentration properties of r(s,, at), p(s++ 1 oz) and the 8, 8 E (0, 1). The VID(+,,a=) represents vector of size |S|||A] containing the square root inverse
count of each state action transition. Considering the sampling error as shown in Eq. (14), we can derive the condition for Q" converging the lower-bound Q" all (&t; (e) through Eq. 12). The
Proposition 3.6 provides the condition for Q" to converge the lower-bound of the Q", Proofs and discussions are in Appendix B.3
Proposition 3.6. For any á (ar()))) with supp (aee()))) C ng. V8 Og E D, the Q-value function
Q" via iterating Eq. (22 satisfies:
Q" (s), S SQ" ("r (s), s [u- (1-)P")
TM ap
(st, ai) + (I - (P--pp--1 (1-r) (sc: a.), (15)
Cr.p.RRmas
Thus, i
Imfoel Fp(a)(s))
the iterative update Eq. (12) guarantees Q S SQ¹
(I(r)a))())
(a(a))))) we obtain not lower-bound for the Q-values estimates point- wise. based on Eq. 13 We abuse the notation
Next, when T.M (as St
to represent a vector of the inverse square root
[
of state counts. Proots and discussions are in Appendix B.4
Proposition 3.7. When T.M (arls) Te (arlse). according to Eq. (13, we obtain a lower bound for the true value of V (aa)))o(aa/).)1 1Q" 0e)] that satisfies the following inequality:
(*())) S V (5c) - [ (T- 7p-)-pt--1 "P
Ba
me T
(8t) + (T -P-)p--- Crg.s
(1 (1-1)/IP (8+), (16)
Thus, if 2 Cr.p. Rinsx
E Easnao
xo(ailas] [oc sx)
1
I1 the Eq. (13) can guarantees V S
(I(x)))((x))
On the other hand, taking into account the need to satisfy the safety constraint S c. the estimation of the cost Q-value must fulfill Q= 2 Q
Ep(so) Σ. ((aalss) Based on the above analysis we choose the penalty that maximizes the expected Q-value under special action-state transitions for unknown actions produced by Ty (a)s). Therefore, we define the Q-value of the iterative reward function as:
2% arg min
1 " Le 2
[[co (st, (s), BB Q (st, a:)1 -XB anD (st, (17) a~FR(110))
where is the trade-off factor. The TR denotes the marginal distribution corresponding to the unknown action. The Proposition provides an upper bound on the convergence of the fixed point of Qo: and clarifies the conditions for Qo: to converge to the upper bound of Qo: Proofs and discussions are in Appendix B.5
Proposition 3.8. For any TR. (apls))) with supp T (ax) (t) C 7t.. VSt, Og E D, the Q-value function
Q via iterating Eq. 17 safisfies:
Q (st. 2 Q* (s), [1 (T-7P---
RR tB
(st, ac)
-11 (I-7PT--
Crp.sCmua (s,, (18) (->)V/D)
Thus, if
Crepi Cmax (1-2) (1(ssaaa))
(s)))) [x]
the iterative update Eq. (17) can guarantee Q > Q
3.2.2 Parametered Policy Update
After solving the optimal variational distribution of each state via Eq. (0), we need to obtain the policy parameters 6. According to the solution target of Eq. (6). the optimization target can be obtained by eliminating the quantities irrelevant to 6.
((0) max Brig jaDDxi((( (g()))) m=)(s=))))
alBp (alo))))) [log (a:l(s)) log ((aass))). (19)

---

where 2 0 is the temperature parameters. In addition, the q(an|sr) is independent of To (t). Therefore, the optimization objective of the above Eq. (19p can be rewriten as:
[()) max Ep(eo) E le)) log (s)a)
(20)
4 Experimental Evaluation
In this section, we compare VOCE to previous offline safe RL methods in range of domains and dataset compositions, including different action spaces and observation dimensions. To the best of my knowledge, there is currently no standardized dataset available in the field of offline safe RL. To facilitate further research and reproducibility of this work, we have collected a series of datasets using a trained online policy. The parameter settings of the dataset are in Appendix C.I
4.1 Task and Baseline
Task. We choose Point-Gos Goal, Car-Goal, Point-Button and Car-Button four tasks widely adopted in the field of safe RL [26 29 32 as the experimental tasks for this work. A detailed description of each task is in Appendix
Point- Goal
Car-Goal
Point-Button
Car-Button
Reward of episode
Reward of episode
Reward of episade
Roward of opisode
Cost of episode
to
episade
episade
Cost episode
Figure 2: Distribution of rewards and costs for samples collected by different behavioral policies.
Baselines. BCQ-Lag is an offline safe RL algorithm that combines the Lagrange multiplier method with the BCQ (24] algorithm and employs adaptive penalty coefficients to implement offline con- straints task. The C-CRR is an offline safe RL algorithm improved by the CRR 35 algorithm. It implements cost constraints by introducing cost evaluation Q-value function and Lagrange multipliers. Coptidice [10] is a policy optimization method based on the optimal stationary distribution space. The aforementioned three approaches are currently state-of-the-an algorithms offline safe RL.
4.2 Performance Comparison Experiment
To evaluate the performance of VOCE on diverse task and behavior samples, we collected sample data from three distinct behaviors within four different tasks. We introduce parameter to represent the proportion of trajectories in the sample data that satisfy the constraints. Then employ y to characterize samples to different behaviors. Fig. 2 Iillustrates the marginal distribution of rewards and costs for sample trajectories in the dataset at different values of sp The results from Fig. 2 reveal that as the value oo increases, the mean cost decreases, and the mean reward increases. Fig. displays the reward and cost curves of VOCE and the ffute-of-th-aae art offline safe RL methods under different yp values for four tasks. The Fig. Fig.Bresults demonstrate that the VOCE achieves higher rewards across all tasks compared to the other three methods, while also satisfying or approaching the safety constraints threshold. Especially in the Goal task, VOCE consistently meets the safety constraints across different o values, while achieving higher reward returns. In the Butfon task, when the parameter yo is small, VOCE struggles to ensure safety; however, the cost curve of VOCE remains lower than the other three methods. Based on the aforementioned results analysis, can be concluded that the VOCE exhibits competitive performance across various combinations of samples from multiple tasks, particularly excelling in terms of safety compared to the state-of-the-art algorithms currently available.
4.3 Ablation Experiment
The parameter yo of the dataset. Fig. 4 displays the boxplots of rewards and costs for the VOCE under different parameter values of yo in the Point-Goal and Point- Button tasks. The results from
7

---

voer
apicode Rowaadd upiccnd
Poant-Goal (9=0.8)
Car Goal (g=0.8)
Point Button (s=0.8)
Car Button (p=0.8)
21
12
32
24
24
1/
10
14
Go
Button(g-D.6
(-
24
28
Cost aptcods Romandd
12
26
02 64
66
62
34
of
16 11
60
05
18
15
26
25
39
96
35
16
15
20
25
Cost optsod Boward apisdds
Ooint-Coallg-0.4)
Point
Car Bultonl pm0.)
62
CA
38
28
12
24
22 M
14
12
1.
62
18
15
24
33
6.
65
10
15
24
30
Step!(1o4)
Figure 3: The reward and cost curves of VOCE and baseline algorithms with different sample data under tasks. The curve is averaged over 3 random seeds, where the solid line is the mean and the shaded area is the standard deviation.
Fig. reveals intriguing phenomenon: VOCE does not achieve the highest reward and lowest cost at 1s=1.0 but instead attains the highest reward within the range of 0.7 to 0.9 (The range will vary depending
changes in the sampling method policy.). This indicates that appropriately increasing the number of constraint-satisfynng trajectories in the dataset benefits VOCE in improving its rewards and reducing costs. However, excessive augmentation constraint-satisfying trajectories in the dataset may lead to a decrease in rewards and even an increase in costs.
Point-Goal
Point-Goal
Point-Button
Point-Button
Reward of episnde
Cost of agicade
Cost epicode
2:
8
64
87
3.5
3.0 25
0.5
8.7
88
0.
3.3
65
24
08
0.
3.8
Figure 4: Ablation experiments with the same sampling policies and different coefficients.
Conservative estimation of rewards and costs. To assess the impact of conservative estimation on VOCE, we conducted ablation experiments on the conservative estimation of both reward and cost -avuuus. Fig. |lillustrssss the rewards and costs of VOCE, VOCE-Qr, and VOCE-QO Qc across four tasks. VOCE-O Qr stands for the VOCE method with the conservative estimation of Q-values removed for rewards. VOCE-Qe represents the VOCE method with the conservative estimation of Q-values removed for costs. The results presented Fig. demonstrate that the rewards VOCE-Qr are notably lower than those of VOCE. This indicates that employing a lower-bound conservative estimation of reward Q-values helps eliminate extrapolation errors caused by OOD actions, thereby significantly improving the reward of VOCE. Furthermore, the results from Fig. S reveal that the

---

rewards of VOCE-Qc are comparable to even surpass those of VOCE. However, in some tasks, the costs of VOCE-Qc exceed the cost constraints. This suggests that utilizing an upper-bound conservative estimation of cost Q-values helps reduce the costs of VOCE, thereby enhancing the safety of VOCE.
VOCE
VOCE Qr
vOGE_0c
Cost kmit
Point Goal
Point-Goal
Car- Goal
Point: Buint-Butoon
Car-Button
Cost of apisad Reward of episode
24
20
22
2.00
0.25
253
6.75
1:00
1.55
150 0.00
3.5
6.50
3.75
140
122
1.50
0.0
35
3.0
15
20
25
3.0
0.3
0.5
10
15
20
25
30
42
M/M
10
3.00 6.25
0.5 0.75
1.00
1.25
1.50
0.c0
925
6.55
5.85
1,00 1.25
3.50
0.5
3,0
1.5
2.0
255
0.0
0.5
1.0 1.5
2.5
3.0
Stop("1o4)
Step(*104)
Stop("1o4)
Stopp
Figure 5: Ablation study for conservative Q-value estimation of reward and cost, and constrained regularize of the policy update. The curve is averaged over random seeds, where the solid line is the mcan and the shaded area is the standard deviation.
Related Work
In this section, we elaborate on the work pertaining to offline safe RL and scrutinize three aspects of safe RL, offline RL, and offline safe RL.
Safe RL. Currently, Safe RL typically addresses optimization objectives with constraints using the primal-dual framework. The Lagrange version of PPO and TRPO 36 37
widely used as the underlying baselines combines the constraint information with the original objective function via Lagrange multipliers. CPO [38 the first general- purppse policy search algorithm for safe RL that guarantees that the near constraints are satisfied at each iteration. However, the sccond-ordes method [38] 39] requires second-order information and thus brings a higher computational burden. To address the approximation errors associated with Taylor approximations and inverting high- dimensional Fisher information matrix, first-order methods such CUP 40 and APPO [4] achieve better performance and implementation.
Offline RL. Offline RL, also known as batch RL, considers the problem of updating policy from an offline dataset without interacting with the environment. There are mainly two approaches of policy constraints and Q-value regularization solve the problem of OOD actions in offline RL 442 44).
The earliest proposed constrained offline RL algorithm, which employs CVAE to learn behavioral policy generation models, and finds the optimal policy by maximizing the Q-value. On this basis, a series of policy-constre
Boachh--osttrinedd deep Q-learning (BCQ) algorithm (24] is typical policy-
algorithms [45 48 are proposed, which mainly constrain the actions explored through the behavioral policy model. In addition, due to the obvious stability of the Q-value regularization approach, it has been extensively studied. conservative Q-learning algorithm [30] proposed by Kumar et al., learns conservative Q-value by enhancing the standard Bellman error objective with simple Q-value regularizer. Subsequently, a series of Q-value regularization methods [49 6 50) were proposed which generally learn a conservative Q- value through regularization items or clipping Q-learning.
Offline safe RL. Offfine safe RL to solve the optimization problem of safety policy under the offline dataset. An offline safe RL algorithm with constraint penalty is proposed [16], which improves the objective function of the cost Q-value via using the constraint penalty item, and ensures the safety of the policy via terminating unsafe trajectory in advance. On the other hand, constrained offline optimization algorithm (COPO) [T] defines the RL problem based on linear programming and constrains the distance between the final policy and offline sample behavior through regulation terms to solve the problem of offline safe RL algorithms. Additionally, this algorithm sets the discount factor 7 1, and changes the maximizing discount reward objective to the maximizing mean reward objective. Similar to the COPO algorithm, an offline policy optimization via stationary distribution correction estimation (CoptiDICE) (18) also utilizes the linear programming method of RL to solve

---

the optimal stationary distribution instead of the gradient strategy, which expands the discount factor to y (0, 1].
Conclusion
In this work, we propose a variational optimization with conservative estimation for offline safe RL algorithm address the problem of optimizing safe policies using offline datasets. We introduce probabilisticc inference to reframe the problem of offline safe RL, and we mitigate estimation errors caused by parameter approximation through employing nonparametric variational distributions instead of parameterized policies. Furthermore, we utilize the upper and lower bounds of Q-values to estimate the Q-values cost and reward, thereby mitigating extrapolation errors caused by OOD actions. Finally, extensive comparisons and ablation experiments demonstrate that the VOCE algorithm outperforms
algorithms in terms of safety.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (No. 62372329), in part by the National Key Research and Development Program of China (No. 2021YFB2501 104), in part by Shanghai Rising Star Program NN.211QC1400900 Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program. We thank Li Shen and Yiqin Yang for the insightful discussion.
in part by Tongji-Qomolo Autonomous
References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, pages 104-114, PMLR, 2020.
[2] Ruiqi Zhang, Jing Hou, Guang Chen, Zhijun Li, Jianxiao Chen, and Alois Knoll. Residual policy learning facilitates efficient model-free autonomous racing. IEEE Roboties and Automation Letters, 7(4): 1625-12211111 1632, 2022. doi: 10. 20910
[3] Jing Hou, Guang Chen, Zhijun Li, Wei He, Shangding Gu, Alois Knoll, and Changjun Jiang. Hybrid residual multiexpert reinforcement learning for spatial scheduling of high-density parking lots. IEEE Transactions on Cybernetics, pages 1-13, 2023. doi: 10. 109/TTB/ 2023. 3312647.
[4] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7844):580-588) 2021.
[5] Ying Jin, Zhuoran Yang. and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[6] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. dincertainty-basee offline reinforcement learning with diversified q-ensebble. Advances in Neural Information Processing Systems, 34, 2021.
[7] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
[8] Yunxiao Shan, Boli Zheng, Longsheng Chen, Long Chen, and De Chen. A reinforcement leasning-based adaptive path tracking approach for autonomous driving. IEEE Transactions on Vehicular Technology, 69(10) 1058::00511105555 2020.
[9] Xiaobai Ma, Jiachen Li, Mykel Kochenderfer, David Isele, and Kikuo Fujimura. Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6064-6071. IEEE, 2021.
10

---

[10] Jongmin Lee. Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup. Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022. URL httppss:/opperreiiieeeee http:///www.....eeoo/eodeaaaaaa.
[11] Nicholas Polosky, Bruno Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning, pages 17801-17810. PMLR, 2022.
[12] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[13] Katie Kang, Paula Gradu, Jason J Choi, Michael Janner, Claire Tomlin, and Sergey Levine. Lya- punov density models: Constraining distribution shift in learning-base control. International Conference on Machine Learning, pages 10708-10733. PMLR, 2022.
[14] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference Machine Learning, pages 3703-3712 PMLR, 2019.
[15] Michael Janner, Qiyang L.,, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34:1273- 1 1286, 2021.
[16] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. Proceedings of the AAAI Conference on Artificial Infelligence, 36(8): 8753-8760, 2022.
[17] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33:9398-9411 2020.
[18] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130. PMLR, 2021.
[19] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference machine learning, pages 1928-1937. PMLR. 2016.
[20] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[22] Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[23] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep explo- ration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL https: /ttpps://proceeddnngg Paper .dd
[24] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062.
PMLR, 2019.
[25] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances Neural Information Processing Systems, 14:0029911031. 2021.

---

[26] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
[27] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint arXiv: 1805000990 2018
[28] Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Nickum, and Ufuk Topeu, Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[29] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv: 111::190000/708, 7(1):2, 2019.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191 2020.
[31] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for re- inforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, ed- itors, Advances in Neural Information Processing Systems, volume 21. Curran As- sociates, Inc., 2008. URL https: http::///poeeeen....eeee wwt.eeeeee...pppeeeeeeeeeeeeeeee hccppappe//000/ff11/ 64622220bb34475550900000888888
Paper. paperrr...pppp.............
[32] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303. 2023.
[33] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint a:X15:2302.07:51, 2023.
[34] Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09300 2023.
[35] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[36] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint 191000 7:1, 2019.
[37] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Juntao Dai, and Yaodong Yang. Safety- :ymett-yymmsiimm A unified safe reinforcement learning benchmark, 2023.
[38] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[39] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- constrained policy optimization. arXiv preprint arXiv:2010.03152 2020.
[40] Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang. and Gang Pan. Constrained update projection approach to safe policy optimization. arXiv preprint arXiv:2209.07088) 2022.
[41] Juntao Dai, Jiaming Ji. Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy optimization for safe reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 5766):288-72993 Jun. 2023. doi: 10. 00.16609aaai..377..558888 URL https
7/ojs. . org/ httpp//pp////aaat//eeeettt.a
[42] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforce- ment learning for urban driving using implicit affordances. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 7153-7162, 2020.
12

---

[43] Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Representations. 2021.
[44] Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor- critic. Advances in Neural Information Processing Systems, 32, 2019.
[45] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference Robot Learning, 2020.
[46] Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling, In International Joint Conference Artificial Intelligence, 2021.
[47] Jiayi Guan, Shangding Gu, Zhijun Li, Jing Hou, Yiqin Yang, Guang Chen, and Changjun Jiang. Uac: Offline reinforcement learning with uncertain action constraint, IEEE Transactions on Cognitive and Developmental Systems, 2023.
[48] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference Learning Representations, 2020. URL https://///ooerreeeeee http://ppeneeiie...netttrrrrt Ad=rke7geittaa
[49] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-20145, 2021.
[50] Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xi- angyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, 2022.
13

---

A Implementation Details
A.1 Algorithm
The VOCE algorithm is shown in Algorithm 1
Algorithm 1: VOCE Algorithm
Dataset = ([st, de, Pr, Sc41) Output: Policy network Te: Q-value network of reward Q", Q Q-value network of cost . Q
Input:
for each batch
Sample batch of transitions (st, oe, rt, 8(41) from the buffer D;
Compute the Q-value ("1,,, and Q (st, via Bellman backup: 1 (st, ar) r(81, ax) + 2 (8141,
B"QQ"(* a.) ci(s. a.) 7 (4+1,0++1):
Update the Q" via the Eq.
Update the Q via the Eq. Update the optimal dual varaaless 1) Compute the optimal variational distribution q" (ai) sx) by the Eq. (10) via Q", Q and Lagrangıan multipliers 77 and A Update the Policy Parameter via the Eq. 20
solving the convex optimization issue the Eq. 11
10
Qi =Q" + (1 7)Qi: Q + TQ -1-1177 7)
12
B Proofs and Discussions
B.1 Proof and Discussion the Proposition 3.4
Proposition B.1. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (27) is given as:
q (ocls)) (o(a)ls)) exp
(Q' as)
exp
11
(21)
7
where (rt(s) a;) ("( a:) X (st,
Proof. The unconstrained problem derived through the Lagrange multiplier method is expressed as follows:
C(g. X, 5) = max min Bp(ea) 020 X20.220.6
Σ (x x2 Σ (sr
(22)
DTK Dx (s)ar)ss)]] ss))
- E 11]
To simplify Eq. (2 we introduce composite Q-value and define it as Q Q" ot) XT XTQ (St, at). Then we rewrite the Eq. 22 as:
Σ ((aalssr)(" ar) - - Σ g(cels)) log g(ar(sr)
C(g, X, n.
max min A20.720,6
E E((co)
ac 3o (a (s)
Σ a(ar(se) x + me +
(23)
For a fixed and n, we write the dual problem [7 18 of the above optimization problem of maximization max.20 L(g, X, n):
min Lig, X,
Σ p(si, ())())))))
(24)
200 a20
#2.00
14

---

Since the strong duality holds, satisfying the KKT condition is a necessary and sufficient condition for the solution of the primal and dual problems.
Condition I (Primal (Primal.easibblliy feasibility). (at 2 0, Vst, at
Condition 2 (Dual feasibility). ("t, ax) 2 0, Vst,
Condition (Stationarity). Q'(sr, (c) X" QF (5t, ae) - (agg 107 1, Te(ae()))
1r' (5t, a.) = 0, 0s 4z
Condition 4 (Complementary), q" (oelss)) (st, ai) 0, Vsi
Form Stationarity we have:
(r1(s), "
5 u (st. a()
("((( To(a, exp
exp
(25)
Since the (a(a,) (s) > O, thus the (ar((s)) > 0. Then the Primal feasibility is always satisfied. Furthermore, we only need consider the cases where q" (ac(s:) >
(Case 1 q (aols)) > 0) ) In this case, (*t, at) = O where Dual feasibility and Complementary slackness satisfied. In addition, Stationarity also applies to
(rr(ss, ax)
exp + 6
9" (a)ls:) To exp
(26)
Therefore, KKT conditions (conditions 1-4) are satisfied.
B.2 Proof and Discussion the Proposition 3.5
Proposition B.2. The X, 1I are dual variables and are solved via the convex optimization problem of the following formula.
((A. m) min Bp(so) 1 log Σ =s(assss) exp A20.220
XTa + t
(27)
"
Proof. By rearranging terms and taking the expectation over ae, Eq. can be rewritten as:
1 + S 17
Σ (arls:) Σ (e(aels)) exp
(s,
exp
(28)
"
where the Ea, 8º (aclse) 1, thus the Eq. (28) can be rewritten as:
1) 6 log Σ no (x) exp Q"T(sse ac)
(29)
?
By substituting Eq.
29 and 26
into Eq. 22 to eliminate g(arl11) (s)) we can rewrite Eq. 22 as:
Σ (aols: ) (s), - Σ (acls)) log (als) To(als)
L(), 7) = min 220
E B
Σ (arls)) + x m
Qr Σ q (atlor (" (s), - Σ (aels))
min
220.220.6
71
(30)
Σ (ar(s)) A + me +
min
Epcr)
Σ (aslst) 17 0)
Σ q' (aols)) + xTa + + me
X20.n20,6
[Qr ( at) + +xa mp
min x20.120.6
Eptco)
1 log
Σ (arls))) exp
Since Eq. 30 eliminates 6, the dual optimization problem of À and 17 can be written as:
(rt + T m76
(().n)
min
E Episa)
17 log
Σ no (a (s:) exp
(31)
X20.720
15

---

B.3 Proof and Discussion the Proposition 3.6
Proposition B.3. For any (ails))))) with supp (m((a() C ng. Vsa, ar E D, the Q-value function
Q" via iterating Eq. 42) satisfies:
t (st, + (I (I- Cr.psSRax Q"(st, S (s(,, x [(1 ---
TM (1-7p-)-1
1--) (st, (32)
Cr.pe Thus, if (1-7) (P)s,aa)
Emaaaele (s.) Tp(ac
[
the iterative update Eq. 12 can guarantees Q S S
Proof. The iterative equation for updating the reward Q-value, which involves an additional mini- mization of the Q-value and the standard Bellman error objective, is expressed as follows:
1h+1 f- arg min R aaDD [[(Q' (st, at)) B² Q a:)]
(33)
kR DRe [Q" (8t, at)],
where the Bdd (st, = r(8,, a+) + 2Q" (8++1, With importance sampling, the Eq. be written as:
can
2et arg arg 1 2 Em [[Q" (st, 8 B
(34)
[m((al))) Q" Q" (s) an)
Darttaa *a(ails))
Setting the derivative of Eq. 34 O we obtain the following expression:
(a,ls))))
( B Qu(s),a)) (T((a)(s)
(35)
Note that the Eq. 34b utilizes the empirical Bellman operator TTT instead of the actual Bellman Operator B" Following the related work (23 31. we employ the concentration properties to determine the error. For any Vst, E D, with the probability < 11
-1 5, the error can be written as:
1 ((s, a.) - B (()) Cr.p. Rmar ac]
(36)
where Cr.p.8 is constant depending the concentration properties of r(8t; p(s++ 19t, az) and the 8,8 E 1). The (D(s,, a;))= represents a vector of size |S||Ai] containing the square root inverse count of each state action transition. Rmaz maxx,,a.=- (r()),))) represents the
maximum reward obtained in a single step. Based on the inequality of Eq. (36) and Eq. (35) we obtain:
Tm(ar(se)
( = Ba (T a.)
(p(()()e)
(37)
<B" (T ar) + Cr.p. Rmax
Tm(aa)s:) (p(aa(s))
7) [D(s,, ar)
The fixed point in the updating process of Eq. 37 can be expressed as:
&omaz Pimaz
(a
Q" a.) s Bd (zt.
(1 (1-7) (D(ss, a:)l
(
Cr.p. Reman
K am(())))))
< krt <r+>p= ( + (1 (1-7)VI((s...
Crap Rimax
(arls)))
$ (ptpi")) + p= as)
(1- (D(),,
(a(ar(5))
(38)
Orp.BBmax
(a
(I Q--P""0'(ss, (8t, (z) S (1->7==)0'( + (1 (1- 7) (s,,,
(39)
Az(a (1)
16

---

Cr.p
Tm
1 ("( 0r) = y S {s) -([I P
(1 7) (DD(ss #g(ar(st)
(40)
+ (I --P")-1 (1- /D((,, 0z)
T (ar st))
To guarantee the lower bound of Q", we eliminate the potential overestimation caused by Cr.p. Rman via choosing an appropriate k. Since (7 --->> matrix of non-negative
(2-7) / [[(s).))] entries, the x that guarantees the lower bound of Q" needs to satisfy:
&r.p.
Tm((y a (8))
K z (D
(41)
B.4 Proof and Discussion the Proposition 3.7
Proposition B.4. When Tm (s.) To (ag st), according to Eq. 13 we obtain lower bound for the true value of V (a, 1Q" (st. az)l that satisfies the following inequality:
(+) S (xt) - K [(1 -T- pP*) Eainto
(arls))
$1] (8+) + (I --- Crp.s
(8)).
(a 88))
(1 (1-7)v(D)
(42)
Thus, if
Cr.p.Rmrr ((D(e))
Eaiwty
ma(ar se) (a #t)
'I the Eq. (3) can guarantee v SV
Proof. The following Eq. (43) represents the iterative formula for the reward Q-value with an additional maximization term introduced under the sample distribution āH..
Qb+t arg min
1 E EeLLaID 2
[(C ax)) - (T(())))))
(43)
anD ("(s),oe)= E anD aanmea(--les)
Q ('(ss,
as-itg(-lee)
where K is tradeoff factor. With importance sampling, the Eq.
(43) can be rewritten as:
Qut1 arg min E, BnDDD [[co a.) (s., 8 (i(s).
(44)
Tm(ais)) o ( (
Q" (8: (z) -
a anD ('(s)
Setting the derivative of Eq. 44 O, we obtain the following expression:
(a) t st)
(s a.) B Qu:
(45)
By taking the expectation of both side of Eq. (45) with respect of the action, we obtain:
Tap(ay)s)) -
Vr (s:) = B (sr) - k Σ no (ar(s))
(ap(())))
(46)
= =
( (st) - (ap(a)))) + क (arls))
-
When Tm((el50) Te(ar((t), we obtain from Eq. (46):
(Te(aa)s)) - + (ar(0)))
Vi (s:) = B Vi(e)) K Σ
Σ Te((c(s)) Σ #
(47)
17

---

Since E.. (a(a)(s) = En, ) = 1, we can conclude that the value estimate (st) is underestimated as Vr (8t) S BTt (s+)) based on Eq. 447.. Next, considering the sampling error,
obtain from Eq. 455) and Eq. 36
TTM (st)
(11) B QU (st, at)
R (ar(st)
(48)
S SB (u(L,
Ta (st)
+
Cr.p,5 Rmas
Atp(ac (st)
(1 (1- (I-s)//D(s,aa))
When (M(())))) (s) the fixed point of the update process in Eq. 48 can be expressed as: (o(())))))
Cr.p,ö Buuce
(x,,) S BT 1= (8(
(ac(8)))
(1 - 7) /D(() azll
(apals)))
Cr.p. Rmaz
< Sr+ +=
-
Ap(aa(s))
(1-7) /D((s,)))))))
To(ae
Cr.p.omaaz
- I -P + 7P (s),a)))))) x (e)
+
DD 0c)
(49)
To(ae(8))
-1 -1]+
Cr.p,RRmas
(I 1-=PP"Q"(ss,aaa S (1--)p" (s), (s((a(s()
(50)
(1 (1-7) (1
(a (ss) 1 + - Cripis Banua
(s,,,a))) 5 (st, x(I -pP-)-
(1 ()
(51)
Buswxy To( (82) {
Boc (sc, Barwno Q" (8 at) -KI--1(P) "-p")-
(52)
Cr.p.ōRmar
+ + 7PT) EaINTy (1- (1=))/P((s,, ar)
T. (aa(aal8)) (8x) (s(aa(ss) -1 + (T--
Pooos
(V - x([ - -x( ypT -t FFa, Frcemmo
(53)
(1 (1- (ID()/D(s)))
To guarantee the lower bound of Vr, we eliminate the potential overestimation caused by & via choosing appropriate x. Since (T- -P is matrix of non-negative (Pss)
entries, the that guarantees the lower bound of V needs satisfy: Braz
TT(aa (st) (t)(a)))))
1 II
K 2 (1 (D(s))(D(s.)
Eneano
(54)
B.5 Proof and Discussion the Proposition 3.8
Proposition B.5. For any TR (a((()) with supp गर (ao|st) tt. Vse, az D, the Q-value function
Q via iterating Eq. 177 satisfies:
(st, 2 Q* (sL, ar) + (1 - 7p-)
RR Rt
(si, - (I (1= (1-1)
Cr.p.
(s1, as).
(55)
Thus, if (1->) (()
Crind
[a. falaa
the iterative update Eq. (17) can guarantee Q > Q9.
Proof. Using the importance sampling, Eq. 17) can be written as:
1 at) c- arg min Q%
E En [[Q2 (st, at)) 13 Q (st, [z)]
(56)
(R(as)) Q (8e, ar) (g((((88))
X (ea
18

---

Setting the derivative of Eq. 566 to O, we obtain the following expression:
π (8)
a,) = B Q% +
(57)
Note that the Eq. 57p utilizes the empirical Bellman operator 2V instead of the actual Bellman Operator 8" Following the related work [23] 31, we employ the concentration properties to determine the error. For any Vis, E D, with the probability 1- the error can be written as:
Cr.p,sCmaa
18 (ss,, a+) - B ((s,, ail
(58)
(1 (D-r) 0t]l
where Cr.p.5 constant depending on the concentration properties of c(st, p(s ae)
and the 6,5 5 E (0, 1). The (D(s;, a;)) represents a vector of size |S|||A| containing the square
root inverse count of each state action transition. Cmaz max D (sr a,)) represents the
maximum cost obtained in a single step. Based on the inequality of Eq. (58) and Eq. (57 we can obtain:
TR (a)ls))))
(htn ar) = B Q% (se, ar)
x [
(59)
Cr.p.6mman
(aclst))
2 Tdth Q (6:,,r)
((s(s,
The fixed point in the updating process of Eq. (59 can be expressed
Cr.poTmaa
X कR (1)(8))
Q 0x) 2 Bd Q (1)
(1 (1- (D-s)v
A S2)
Or.p.C
X T TR(a118))
2c 7 Qu (1 1)- (((t.
(
Or.poomaz
R sx)
(8, - PP + + (A))(())))))))))
(1 (1-7) (1-7)/D((ss.)) ar)
k (a st) (60)
Cr.p.
n (R(at(s)) Aa)aa(())
(1 (=))"))("(,, 2 (I -1=PP")'' (s), ( (1-7)VD($se.)
(61)
Cr.piSman
(a(()(s))
(16, a() - (I->P=)-
X [p((a)())
(1 (DD()) at)l
(62)
2 (s,, x(I - ) (0e(8t)
Cr.po
- -P=)-1 (1-7)
We guarantee the upper bound of Q*: via choosing an appropriate value of x based on the Eq. 62
Beside the (I -P*)- is matrix of non-negative entries.
Cr.p.8Cmaz
TR (at
X (1 (1-7)pp ((s at)
(63)
(T((0((8))
C Experimental Details
C.1 Dataset Details
As shown in Fig. 6. we utilize the online policy CVPO to collect sample data with different values during three distinct stages the training process. The variable yp represents the proportion of safe trajectories in the collected samples compared to the total number of sample trajectories. Behavior I corresponds to o 0.4, behavior 2 corresponds to y = 0.6, and so on. ensure that the collected sample data satisfies the designated yo value, we incorporate an additional trajectory filtering policy into the data collection procedure. Furthermore, the total number of steps for each category of samples collected is 1e7.
19

---

Roward of apicades
M Th My a Manl Moote Mwhnhhhhii
Reward of esioode
Cost of episodes
Epach
2000
2000
2123
A 3903
Cost episodes
Figure 6: Collect datasets with different values şp in the Point-Button task.
C.2 Experimental Task Details
The Point-Goal and Car-Goal task. The objective of the agent is to reach goal while avoiding surrounding obstacles. Once the agent reaches the correct goal, the environment randomly generates a new goal. The agent receives reward when it approaches reaches the goal. There are hazards and vases as obstacles in the environment, and the agent incurs a cost when encountering them. Hazards immovable, while vases can be moved. The observation state is the relative pose between the agent and obstacles and goals, represented by a spint-based pseudo-LIDAR, We select two tasks, namely Point-Goal and Car-Goal tasks [26 29]. These tasks performed by the Point and Car agents, respectively. The Point agent is a robot constrained to a 2D plane, capable of rotation as well as forward and backward movement. The Car agent is slightly more complex robot with two independent parallel wheels and a freely rolling rear wheel.
Table 1: The Hyper-parametent of the VOCE algorithm model. The variables s and a represent the dimensions of the state and action, respectively.
Categories
Value
Sizes
256 256 x
Activation function
ReLu
Policy network
Number of networks
Learning rate
2.0c-3 Adam
Optimizer
Sizes Activation function
(s + a) x 256 x 256 x I
ReLu
Q-value network of reward
Number of networks
Learning rate Optimizer
1.0e-3 Adam
Sizes
(8 + a) x 256 256 T
Activation function
ReLU
Q-value network of cost
Number of networks
Learning rate
1.0e-3
Optimizer
Adam
Batch size
600
Discount factor y
0.99
KL threshold
0.1
Others
The clip of the dual variable 77
[1.0c-6. 1.0e5]
The clip of the dual variable À
[1.0e-6. 1.0e5]
The clip of the trade-off' factor
[0. 1.0c6]
The clip of the trade-off factor X
[0. 1.0e6]
The Point-Button and Car-Button task. The objective of the agent is to navigate around stationary and moving obstacles in the environment and press one of the multiple target buttons. The task is similar to the goal task, where the agent receives a reward for either approaching or pressing the target button. The Button task involves the presence dynamic obstacles that move rapidly along predefined trajectories. As the agent incurs costs upon colliding with these dynamic obstacles, the Button task is more challenging compared to the Goal task. In the Button task, we select two tasks: the Point- Button task and the Car-Button task [26] 29
20

---

C.3 Hyper-parameters
As shown in T we share the hyperparameters of the CEVO algorithm to facilitate reproducibility. Furthermore, we will share the code GitHub later time.
C.4 Datasets for the Preliminaries Section
In the preliminaries section, we employed dataset with yp 1.0 to evaluate offline RL methods. Additionally, we utilized a dataset with yp 0.8 evaluate off aff-policy safe RL methods. As shown in Fig. 7 1.0 indicates that the dataset includes samples with not only high rewards but also costs within cost limit. In addition, = 0.8 indicates that dataset includes samples with high rewards, while also containing samples that exceed the cost limit.
C.5 Datasets for the Ablation Experiment
Fig 7 presents the datasets collected under the same policy but with different ip values. To ensure the acquisition of datasets that satisfy the yp values, incorporate trajectory filtering strategy into the data collection procedure and introduce random actions. Concretely, when the random value is less than the predefined threshold e, we employ random actions instead of the actions provided by the policy.
Reward of episdes 8
episorbes
2009
2000
Epoch
Cost of episade
Figure 7: Collect datasets with the same policy and different values y in the Point-Button task.
C.6 The Ablation Experiment of the Parameters
We intuitively believe that the value of the parameter e in the o affeett the performance of the VOCE algorithm. Therefore, conduct ablation experiments this parameter. The results shown in Fig [ indicaae that setting the parameter e too small reduces the convergence speed of the policy and even decreases the algorithm's 's performance. Conversely, when the parameter is to larger, it may lead to instability in the policy training process. Furthermore, from the illustrated results, we can conclude that within this range of parameter e [0, 1]. ensures the policy's 's good convergence speed and stability. Note that the parameter E [0, 1] represents a feasible range rather than an optimal interval.
Point-Goal
Point-Goal
Roward of episode
Cost limit
Cost of episode
30
0.05
0.1
-20
10
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.5
0.8
1.0
1.2
1.4
(xep(*le4)
Step(*1e4)
Figure Reward and cost curves of VOCE the Point-Goal task for different values the parameter The parameter is the KL-divergence threshold between the variational distribution g(ar(sr) and the parameterized policy n (a))s)).
21

---

C.7 The trade-off factors
Regarding the trade-off factors a. and X, we adopt dual-tuning approach with gradient- baseed adaptation instead of a manual setting. To facilitate the observation of changes in the trade-off factors and X, we record the variation curves of these parameters during the training process. The results of indicaate that as the model gradually converges, the trade-off factors and X gradually decrease,
Fig. 9 eventually approaching zero.
1.00
1.00
0.75
0.75
0.50
X 0.50
0.25
0.25
0.00
0.00
Reward of episode
Cost episoda
100
75
-00
25
Step(*le4)
Step(*1e4)
Figure 9: The trade-off factors &. and X. are considered along with the curves of rewards and costs during the training process in task Point-button. Both the trade-off factors x and x are adaptively adjusted using the dual-tuning approach through gradient-based
C.8 The time costs of training and testing
We recorded the time costs of various modules of the VOCE and the baseline algorithm during both training and testing. The results presented in Table[2indices that during training, the VOCE algorithm incurs additional time costs per step compared to the baseline algorithm due to the execution of additional networks and optimization parameters. However, during the testing phase, as only the policy network executed, the time cost of VOCE is similar to the baseline algorithm. Furthermore, the execution time per step during testing is significantly lower than Ims, meeting the real-time requirements for robot and autonomous driving control. Additionally, the results from Table reveal that the evaluation time cost of the Q-values is relatively high due to the computation of multiple target Q- valuee and gradients.
Table 2:
The time costs per single step of the algorithm during training and testing processes. The units for the numerical values all in seconds.
Algorithms VOCE 3.1498
Training(s)
Testing(s)
3,0665 x lc-4
C-CRR
0.0891
1.5140 x lc-4
COptiDICE
0.0358
1.4316 x lc-4
BCQ-Lag
0.1104
2.7976 x 1e-4
Table 3: The time costs during the training processes of various modules in VOCE. The units for the numerical values are all in seconds.
Critic
Actor
Q Q'(s,a,)) Q' ac) 1.2668
(A, 17) 0.0026
((a,(s)) 0.5533
Tp(ay 0.0543
1.2728
D Limitations and Future Work
The setup of this work involves learning safety- canety-conttraine policies from offline data without in- teracting with the environment. Therefore, both the size and quality of the dataset directly impact
22

---

the algorithm's performance. Moreover, in the offline setting, the available sample data is limited, making challenging to adequately represent the state transition matrix P. in non-stationary environment. Consequently, the VOCE algorithm struggles to learn high-reward policies that satisfy safety constraints in non-stationary environments.
23


Reviews:
[{"review_id": "LQG9NdWA7V", "reviewer": "Reviewer_cmps", "summary": "The paper presents a variational approach to offline safe reinforcement learning (RL), where the class of variational distributions is the set of policies which satisfy the cost constraints. It is shown how to obtain the closed-form solution for the optimal variational distribution, and how to extract a parametric policy given the variational distribution. To guarantee that the reward Q function is not overestimated and the cost Q function is not underestimated, the tools of Conservative Q Learning (CQL) are incorporated.", "questions": "* How are the trade-off factors ($\\kappa$, $\\chi$) determined? Are you using a dual tuning method akin to the $\\alpha$ in CQL?\n* In Eqn. (20), should be $a_t$ instead of $\\cdot$?", "limitations": "No, limitations are not addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The variational formulation is novel to my knowledge and may inspire additional work in this area.\n* VOCE performance is good compared to existing algorithms, both in terms of high reward and satisfying the cost constraint.\n* The paper is generally clear and not hard to read.", "weaknesses": "* There is no discussion of the runtime of the algorithm, which I imagine is comparatively high, given that you have to solve an optimization problem (Eqn. 11) at every step.\n* While the final hyperparameters are provided, we are not told how they were tuned, nor do we have a sense of how sensitive VOCE’s performance is to their values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear review cmps:  \nThank you for your all valuable suggestions and meticulous comments. We will incorporate your suggestions in the revision. Below we respond to your key concerns point by point. Please let me know if there are any further questions.\n\n+ __Q1: \"There is no discussion of the runtime of the algorithm, which I imagine is comparatively high, given that you have to solve an optimization problem (Eqn. 11) at every step.\"__  \n__A1:__ We greatly appreciate your raising this question. We have documented the single-step time consumption of the VOCE and existing methods during both training and testing processes. Additionally, we have recorded the time consumption of each module of the VOCE algorithm during the training process. The recorded results are presented in Tables 1 and 2, respectively. The results shown in Table 1 indicate that during the training process, the inclusion of additional networks and optimization parameters in VOCE indeed leads to a noticeable increase in the time consumption per step compared to other methods. However, during the testing process, since only the policy network needs to be executed, the per-step time consumption of the VOCE algorithm is only slightly different from other algorithms. Furthermore, the execution time per step during testing is significantly below 1 ms, which satisfies the real-time requirements of practical applications such as robot control and autonomous driving. Furthermore, as evident from the results in Table 2, higher time consumption is observed during Q-value evaluation due to the computation of multiple target Q-values and gradients. On the other hand, the time consumption is relatively lower for the Lagrange multipliers $\\lambda, \\eta$ as they are updated only once per step utilizing the gradients.  \n__Table 1 The computational time per single step of the algorithm during training and testing processes.__\n|Algorithms|Training(s)|Testing(s) |\n| ---------| --------- | --------- |\n|VOCE      |3.1498     |3.0665×1e-4|\n|C-CRR     |0.0891     |1.5140×1e-4|\n|COptiDICE |0.0358     |1.4316×1e-4|\n|BCQ-Lag   |0.1104     |2.7976×1e-4|    \n\n__Table 2 Time consumption during the training processes of various modules in VOCE.__  \n| $Q^{r}(s_t,a_t)$ | $Q^{c}(s_t,a_t)$  |  $(\\lambda, \\eta)$  | $q(a_t\\|s_t)$ | $\\pi_{\\theta}(a_t\\|s_t)$      |\n|:---------:|:---------:|:---------:|:---------:|:---------:|\n| 1.2728 | 1.2668 | 0.0026 | 0.5533 | 0.0543 |  \n+ __Q2: \"While the final hyperparameters are provided, we are not told how they were tuned, nor do we have a sense of how sensitive VOCE’s performance is to their values. How are the trade-off factors ($\\kappa$, $\\chi$) determined? Are you using a dual tuning method akin to the $\\alpha$ in CQL? \"__   \n__A2:__ Interesting and insightful question. Yes, we have employed a dual-tuning approach similar to CQL. We take into consideration that setting the trade-off factor $\\kappa$ too large might result in a significant underestimation of the Q-values $Q^{r}$, while putting it too small could struggle to suppress the overestimation of Q-values $Q^{r}$ caused by OOD actions. Manually adjusting these parameters is time-consuming; hence, we follow the dual tuning methodology of CQL, utilizing gradient descent for adaptive parameter $\\kappa$ adjustment. The tradeoff factor $\\chi$ is also set using a similar method.  To facilitate the observation of the variations in trade-off factors $\\kappa$ and $\\chi$, we have documented the evolution curves of these factors during the training process. The variation curve of the trade-off factor $\\kappa$ and $\\chi$ during the model training process is depicted in Fig.2 . The results shown in the figure indicate that as the model gradually converges, the trade-off factor $\\kappa$ and $\\chi$ diminishes progressively and eventually stabilizes at a value greater than zero.  \n\n+ __Q3:\"In Eqn. (20), should be $a_t$ instead of $\\cdot$ ?\"__  \n__A3:__ Thank you for your suggestion. We have revised Eq. (20) to the following expression:  \n   $\\mathcal{L}(\\theta)=\\max \\mathbb{E}\\_{\\rho(s_0)}\\mathbb{E}\\_{q(a_t | s_t)}  [\\log {\\pi_{\\theta}(a_t |s_t)}]$        \nAdditionally, we have double-checked all the equations in the manuscript, and we have revised Eq.(19) to:  \n  $\\mathcal{L}(\\theta)=\\max \\mathbb{E}\\_{\\tau\\sim q} \\left[-\\alpha D_{KL}(q(\\cdot|s_t)||\\pi_{\\theta}(\\cdot|s_t))\\right]= \\max \\alpha \\mathbb{E}\\_{\\rho(s_0)} \\mathbb{E}\\_{q(a_t | s_t)} [\\log {\\pi_{\\theta}(a_t|s_t)}-\\log q(a_t|s_t)]$   \n  \n\n\n\n\n\n\n"}, {"review_id": "m5W5HiTYD6", "reviewer": "Reviewer_GTEu", "summary": "This paper applies CQL to offline safe RL tasks. It introduces a non-parametric policy to search for the actions that satisfy the constraints. Lagrangian multipliers are introduced for constraining the KL divergence w.r.t. the current policy and the additional constraints. CQL-style additional terms are introduced to avoid extrapolation issues during the offline training. The method is evaluated at four safe RL environments and demonstrates that the proposed approach can achieve better rewards while better approaching the cost limits.", "questions": "I would like to know the reason that the authors chose to follow CQL instead of other offline RL algorithms. Since there is already a KL term in the optimization objective, I think Behavior cloning or advantage-weighted approach, which directly constrains the KL divergence between the learned policy and the behavior policy, is more straightforward here and may achieve higher performance in practice. Is there any reason to consider the conservative Q estimation rather than other choices?", "limitations": "not discussed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper proposed a solid approach that consists necessary ingredients for offline-safe RL. The conservative estimation follows CQL and should be able to mitigate the issues in offline RL.\n- An optimal variational distribution is introduced to improve the policy in a non-parameterized way. This step is novel in offline safe RL.\n- The paper proposed an offline safe RL benchmark and made a careful study.", "weaknesses": "- The novelty of the method could be more extensive. The method mostly combines existing methods. The variational method, non-parameterized policy estimation, constraint formulation, and conservative Q estimation are all standard methods.\n- Some mathematical notations are not clear. For example, Eq (19) and Eq (20) are quite misleading. I think it should be log \\pi_\\theta(a_t|s_t) in Eq(2).\n- If my understanding is correct, the proposed method is not able to reach the cost limitation in several point-button and car-button environments, which suggests the inadequate of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear review GTEu:   \nThank you for your suggestions and constructive comments. We will incorporate your suggestions in the revision. Below we respond to your key concerns point by point. Please let me know if there are any further questions.   \n+ __Q1:\" The novelty of the method could be more extensive. The method mostly combines existing methods. The variational method, non-parameterized policy estimation, constraint formulation, and conservative Q estimation are all standard methods.\"__   \n__A1:__ We agree that our VOCE method has been inspired by these advanced RL techniques. Although our method is developed from the existing techniques,  we extend it to the setting of offline safe reinforcement learning, thereby significantly broadening their applicability. Furthermore, the proposed method achieves superior performance in offline safe reinforcement learning tasks compared to the current state-of-the-art approaches. Now, we proceed to delineate the primary contributions of our VOCE method and highlight its key distinctions from existing approaches in the following aspects.  \n(1) We propose a novel variational optimization with conservative estimation for offline safe reinforcement learning based on variational inference and pessimistic estimation methods to solve the problem of optimizing safety policies in the offline dataset. To the best of my knowledge, this is the first time that variational inference has been introduced to solve the task of offline safe reinforcement learning.  \n(2) Subsequently, we extend the pessimistic estimation to the offline safe reinforcement learning setting, deriving for the first time an upper bound for Q-value. We employ this upper bound to estimate the cost Q-value $Q^{c_i}$, thereby avoiding underestimation of the cost Q-value $Q^{c_i}$.   \n(3) In the field of offline safe reinforcement learning, we introduce for the first time the validation of algorithm performance using sample data under various safety ratios $\\varphi$. Furthermore, extensive experiments demonstrate that our algorithm significantly offers competitive performance, particularly in terms of safety, as evidenced by the results.    \n+ __Q2:\" Some mathematical notations are not clear.  For example, Eq (19) and Eq (20) are quite misleading.  I think it should be $log{\\pi}_{\\theta}(a_t|s_t)$  in Eq(20).\"__   \n__A2:__ We are sorry for the unclear mathematical representation, and we greatly appreciate your suggestions.  We have revised Eq.(19) and (20) according to your suggestions, as shown below:  \n$\\mathcal{L}(\\theta)=\\max \\mathbb{E}\\_{\\tau\\sim q} \\left[-\\alpha D_{KL}(q(\\cdot|s_t)||\\pi_{\\theta}(\\cdot|s_t))\\right]= \\max \\alpha \\mathbb{E}\\_{\\rho(s_0)} \\mathbb{E}\\_{q(a_t | s_t)} [\\log {\\pi_{\\theta}(a_t|s_t)}-\\log q(a_t|s_t)]$     \n$\\mathcal{L}(\\theta)=\\max \\mathbb{E}\\_{\\rho(s_0)}\\mathbb{E}\\_{q(a_t | s_t)}  [\\log {\\pi_{\\theta}(a_t |s_t)}]$   \nAdditionally, we have double-checked all the equations in the manuscript and will update any inaccurate representations in the final version.\n+ __Q3:\"If my understanding is correct, the proposed method is not able to reach the cost limitation in several point-button and car-button environments, which suggests the inadequate of the proposed method.\"__   \n__A3:__ Yes, your understanding is correct.  In the *Point-button* and *Car-button* scenarios, there are continuously moving obstacles, which significantly increase the dimensionality of the dynamic transition matrix $P$ compared to the *Point-goal* and *Car-goal* scenarios. However, the size of our sample dataset in all scenarios is 1e7 action-state pairs, which leads to a significant increase in OOD actions in the *Point-button* and *Car-button scenarios*, thereby affecting the algorithm's performance. Furthermore, even though the cumulative costs in these two scenarios do not meet the safety constraints, the cumulative costs achieved by the VOCE algorithm are lower than those of the previous state-of-the-art algorithms. This suggests that the performance of the VOCE algorithm surpasses that of the previous state-of-the-art algorithms. To ensure the policy's adherence to safety constraints in both the *Point-button* and *Car-button* scenarios, we can explore the following approaches:(1) Increasing the amount of sample data in these scenarios to mitigate the impact of OOD actions during policy training. (2) Increasing the minimum pruning threshold for hyperparameter $\\chi$, however, may lead to a significant reduction in reward values.\n+ __Q4: \"I would like to know the reason that the authors chose to follow CQL instead of other offline RL algorithms. Since there is already a KL term in the optimization objective, I think Behavior cloning or advantage-weighted approach, which directly constrains the KL divergence between the learned policy and the behavior policy, is more straightforward here and may achieve higher performance in practice. Is there any reason to consider the conservative Q estimation rather than other choices? \"__   \n__A4:__   Insightful question. You have presented an excellent approach by constraining the target policy to the behavior policy using the KL divergence. Under the safety of expert sample data, this method enables the target policy to achieve high rewards while adhering to safety constraints. However, it is challenging to guarantee that the policy satisfies safety constraints, especially with non-expert and unsafe datasets such as $\\varphi =\\{0.4,0.6 \\} $.  We estimate $Q^{r}$ and $Q^{c_i}$ respectively through the upper and lower bounds of Q-value estimation, which can theoretically ensure that the strategy meets the safe constraints, and also avoid the need to add additional constraints to increase the instability of model training when solving the optimal variational distribution.\n\n\n\n\n\n"}, {"review_id": "bRDYx4jhJH", "reviewer": "Reviewer_kEdv", "summary": "This paper introduces an interesting algorithm for offline safe RL, called Variational Optimization with Conservative Estimation (VOCE). The primary challenge tackled by the authors is the influence of safety constraints and out-of-distribution (OOD) actions, which often hamper the optimization of high-reward policies while maintaining safety. Traditional methods, such as linear programming and exploration-evaluation methods, have shown their limitations in coping with these challenges. In response, the authors propose a probabilistic inference-based framework that leverages Expectation and Maximization (EM) to provide more flexible policy optimization. It employs pessimistic estimation methods to calculate the Q-value of cost and reward, thus mitigating extrapolation errors due to OOD actions.", "questions": "1. I think one major challenge for these constrained optimization-based approaches for offline safe learning is to accurately estimate the Q-value functions, since a subtle inaccuracy will be amplified by comparing Qc estimates and the target threshold after more and more optimization iterations. So I am curious how the authors think and tackle this problem, apart from using pessimism. In fact, pessimism alone can not solve the Q value mismatch between the current optimized policy and the behavior policy in the datasets.\n2. Following up on the above question, I wonder how VOCE would perform in more diverse datasets with a wider span of trajectories over the cost-reward-return spaces. For example, collecting datasets with varying thresholds, hyper-parameters, and algorithms, as shown in [this](https://arxiv.org/abs/2306.09303) work.", "limitations": "I did not find related discussions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This works is an interesting extension of off-policy safe RL (particularly CVPO) to the offline setting. The application of probabilistic inference to the problem of offline safe RL introduces non-parametric variational distributions to replace parameterized policies, improving the flexibility for optimizing safe policies.\n2. The paper's methodology is clear and well-presented. The derivation of upper and lower bounds for Q-value estimation using a pessimistic estimation approach, which is then utilized to estimate Q-values of costs and rewards to mitigate extrapolation errors from OOD actions.\n3. Experiments demonstrate the superiority of the VOCE algorithm over baseline methods in terms of safety.", "weaknesses": "1. It seems that many claims and arguments are directly derived from CVPO, such as Propositions 3.4 and 3.5. The authors should properly cite the references accordingly. In addition, some other recent offline safe RL [methods](https://arxiv.org/abs/2302.07351) and [benchmarks](https://arxiv.org/abs/2306.09303) could be discussed in the paper.\n2. I am not sure whether the EM optimization objectives in CVPO could be directly applied to the offline setting, since the introduction of the KL constraints in the E-step and M-step aims to regularize the policy update such that the behavior policy will not deviate too much and cause catastrophic failure in the online setting; however, in the offline setting, since the policy will not interact with the environment, this constraint seems not very necessary? Would regularizing the KL constraints between the variational distribution and the behavior policy in the datasets (with some density estimations for the behavior policies) be a better choice? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear review kEdv:  \nThank you for your all valuable suggestions and meticulous comments. We will incorporate your suggestions in the revision. Below we respond to your key concerns point by point. Please let me know if there are any further questions.   \n+ __Q1: \"It seems that many claims and arguments are directly derived from CVPO, such as Propositions 3.4 and 3.5. The authors should properly cite the references accordingly.\"__  \n__A1:__ Thank you for your suggestions. This work of CVPO[1] has provided profound inspiration for our work. We have cited and discussed it in Section 3.2 of our manuscript. Furthermore, we will supplement the discussion in the final version regarding the proposition that \"Similar theoretical foundations can be found in the online safe RL of CVPO[1], as outlined in Propositions 3.4 and 3.5.\"  \n+ __Q2: \"In addition, some other recent offline safe RL methods and benchmarks could be discussed in the paper.\"__  \n__A2:__ Following your suggestion, we will incorporate a discussion on the CDT[2] algorithms in the related work section of the final version. Additionally, we will provide necessary comparative experiments based on the DSRL[3] dataset in the Appendix. Due to time constraints, we have done experiments of the VOCE algorithm in three scenarios of the DSRL dataset, and the experimental results are presented in Table 1 (The table is available in the supplementary PDF）. Furthermore, we consider CDT to be a highly meaningful work that leverages the return-conditioned sequence modeling framework to facilitate zero-shot adaptation to various constraint thresholds during deployment while ensuring both safety and high rewards.  \n+ __Q3: \"I am not sure whether $\\cdots$ this constraint seems not very necessary? \" from the second weakness.__  \n__A3:__ Insightful question. Introducing the KL divergence constraint during the optimization of the optimal variational distribution $q(a_t|s_t)$ can mitigate policy oscillations caused by the iteration of Lagrange multipliers in the optimization process. Furthermore, as we have already introduced the KL divergence constraint between the optimal variational distribution $q(a_t|s_t)$ and the previous iteration's policy $\\pi_{\\theta}(a_t|s_t)$ during the process of solving for the optimal variational distribution $q(a_t|s_t)$, imposing an additional constraint on the distance between the latest policy and the previous policy could lead to slow policy convergence and even performance degradation. Hence, we have not incorporated the KL divergence constraint in the process of updating the parameterized policy.   \n+ __Q4：\"Would regularizing the KL constraints between the variational distribution and the behavior policy in the datasets (with some density estimations for the behavior policies) be a better choice?\"__  \n__A4:__ Interesting and insightful question. You have provided a meaningful approach that utilizes KL divergence to constrain the distance between the variational distribution and the behavioral policy. This method aims to reduce the discrepancy between the learned policy and the behavioral policy based on sample data, potentially improving the algorithm's performance on expert and safe datasets. However, for non-expert and non-safe datasets, this approach might hinder the variational distribution from deviating away from unsafe behaviors, making it challenging for the final policy to satisfy safety constraints unless we specifically train our policy using expert safety data.  \n+ __Q5: \"I think one major challenge for $\\cdots$ optimized policy and the behavior policy in the datasets.\"  from the first question.__  \n__A5:__  I fully agree with your perspective. One of the main challenges in offline safe reinforcement learning based on off-policy methods is ensuring the accuracy of Q-value estimation. Indeed, relying on strict upper bound estimation for $Q^{c_i}$ can lead to error accumulation. To mitigate the issue of Q-value mismatch between the current policy and the behavior policy after multiple iterations, we place full trust in the action-state pairs present in the sample data. Therefore, during the Q-value of cost $Q^{c_i}$ update, we maximize the $Q^{c_i}$ of action-state pairs present in the sample data and incorporate this trick into the VOCE code.\n+ __Q6: \"Following up on the above question, I wonder how VOCE would perform in more diverse datasets with a wider span of trajectories over the cost-reward-return spaces. For example, collecting datasets with varying thresholds, hyper-parameters, and algorithms, as shown in this work.\"__  \n__A6:__ Thank you for your suggestion. You mentioned that the benchmark provides an excellent testing methodology, which evaluates the algorithm's performance by testing it under various cost thresholds. This method effectively captures the algorithm's performance across different cost thresholds and hyperparameters. Therefore, we are considering including our algorithm's performance on this benchmark in the final version. Currently, we have included the results from several tasks we have conducted, as shown in Table 1 (The table is available in the supplementary PDF）.  Additionally, another recent work of ours also considers evaluating our algorithm's performance based on this benchmark.  \n\n\n[1] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644–13668. PMLR, 2022.   \n[2] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351, 2023a.   \n[3] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao  Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning.  arXiv preprint arXiv:2306.09303, 2023b.\n"}, {"review_id": "73SzxbWxgn", "reviewer": "Reviewer_wQ6W", "summary": "This paper utilizes probabilistic inference to address the problem of offline safe RL by introducing non-parametric variational distributions. \n Pessimistic estimation of Q-values are used to avoid extrapolation errors caused by OOD actions. Extensive comparative numerical experiments are carried with respect to both reward and cost curve. ", "questions": "\n1. The algorithm involves lots of parameters to tune, like \\epsilon in equation 11, \\chi in equation 17. I am a little bit suspicious of how robust those parameters affect the numerical results. Maybe some ablation study is needed to further clarify. \n\n2. Why pessimistic estimation of Q-values can help avoid extrapolation errors caused by OOD actions. I don't get the intuition here. More theoretical analysis / explanation or numerical experiments are needed. ", "limitations": "No negative societal impact of their work is seen. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper has a solid chain of theoretical derivation.\n2. The numerical results of VOCE seems promising.\n", "weaknesses": "1. The paper is not well-organized written. For example, several notations are not defined before used or at least not easy to find. For example, Q^{r}(s,a) used in equation 7, \\pi_{M}(a_t|s_t) in equation 13. And it's more clear to have a complete summary of the main algorithm  VOCE, rather than deriving literally step by step. \n\n2. Some claims in the paper need further theoretical explanation or numerical experiments to clarify.\n\n3. It's not well studied how key tuning parameters affect the numerical conclusion. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear review wQ6W:  \nThank you for the insightful comments which help us improve the paper. We'll answer your questions one by one below. We are also very honored to share our understanding with you.   \n+ __Q1: \"The paper is not well-organized written. $\\cdots$ rather than deriving literally step by step.\" from the first weakness.__    \n__A1:__ Sorry for our unclear presentation. The $Q^{r}(s_t,a_t)$ represents the reward Q-value of taking action $a_t$ in state $s_t$. The $\\pi_{\\mathcal{M}}(a_t|s_t)$ refers to the policy corresponding to unseen action-state pairs. We have double-checked all variables in the manuscript and provided definitions for variables such as $\\rho(s_0)$, $\\epsilon$. The variable $\\rho(s_0)$ denotes the distribution of the initial state $s_0$. The $\\epsilon$ represents the KL divergence threshold between the variational distribution $q(a_t|s_t)$ and the policy $\\pi(a_t,s_t)$.  Then we will provide definitions for any undefined variables in the final version. We have presented the pseudo-code for the VOCE algorithm in Appendix A.1 of the manuscript. To enhance the clarity of the paper's structure, we will provide a summary and implementation details of the algorithm around its pseudo-code in the final version's appendix.    \n+ __Q2:\"Some claims in the paper need further theoretical explanation or numerical experiments to clarify.\"__   \n__A2:__ Thank you for your suggestion. If you have any further questions or concerns about our work, please do not hesitate to let us know; we would be more than happy to provide clarifications. Additionally, we have done ablation experiments concerning the manually tuned hyperparameter $\\epsilon$, and the results are depicted in Fig.1. The results displayed in the figure indicate that setting the parameter $\\epsilon$ too small would reduce the convergence speed of the policy and may even diminish the algorithm's overall performance. Conversely, when $\\epsilon$ is set too large, policy instability can arise. The experimental outcomes depicted in the graph demonstrate that within this range of $\\epsilon \\in [0.1,1]$, it ensures a favorable balance between convergence and stability for the policy. The parameter $\\epsilon \\in [0.1,1]$ represents a feasible range rather than an optimal interval. \n+ __Q3:  \"It's not well studied how key tuning parameters affect the numerical conclusion.\"__  \n__A3:__ We thank you for your pointing out this issue. Regarding the manually tuned parameter $\\epsilon$, we have done ablation experiments and the results are presented in Fig.1. Additionally, Regarding the hyperparameters $\\kappa$ and $\\chi$ mentioned in the manuscript, setting them too small makes it difficult to guarantee the algorithm's performance theoretically, while setting them too large leads to conservative behavior and a decrease in algorithm performance. Therefore, we adopt a dual-tuning approach with gradient-based adaptation instead of a manual setting.  To facilitate the observation of the variations in trade-off factors $\\kappa$ and $\\chi$, we have documented the evolution curves of these factors during the training process. Furthermore, the variation curve of the trade-off factor $\\kappa$ and $\\chi$ during the model training process is depicted in Fig.2 . The results shown in the figure indicate that as the model gradually converges, the trade-off factor $\\kappa$ and $\\chi$ diminishes progressively and eventually stabilizes at a value greater than zero.   \n+ __Q4:\"The algorithm involves lots of parameters to tune, $\\cdots$ Maybe some ablation study is needed to further clarify.\" from the first question.__  \n__A4:__ We thank you for your pointing out this issue. Regarding the manually set hyperparameter $\\epsilon$, we have done ablation experiments as shown in Fig.1. The rules governing its configuration have been analyzed in Q2. Additionally, $\\chi$ is adjusted using a dual-tuning approach with the gradient descent strategy, not manually set.  To facilitate the observation of the variations in trade-off factors $\\kappa$ and $\\chi$, we have documented the evolution curves of these factors during the training process. Furthermore, we have done a detailed analysis of its variation pattern in Q3.\n+ __Q5: \"Why pessimistic estimation of Q-values can help avoid extrapolation errors caused by OOD actions. I don't get the intuition here. More theoretical analysis/ explanation or numerical experiments are needed. \"__   \n__A5:__  Interesting and insightful question. The pessimistic conservative estimation of Q-values is rooted in a pessimistic perspective that purely assumes that OOD actions not present in the sample data possess lower rewards and higher costs. Building upon this intuitive notion, it involves minimizing or maximizing the reward Q-values and cost Q-values associated with OOD actions during the Q-value assessment process. This approach aims to induce lower estimated values for reward Q-values concerning unseen actions and higher estimated values for cost Q-values pertaining to unseen actions. Ultimately, this results in the policy reducing the probability of producing out-of-distribution (OOD) actions, thereby mitigating the extrapolation error induced by OOD actions. Furthermore, this viewpoint is substantiated by existing relevant theoretical underpinnings [1]. On the other hand, our manuscript has done ablation experiments concerning conservative estimates for $Q^{r}$ and $Q^{c_i}$. The experimental findings also indicate a notable enhancement in algorithmic reward returns due to the conservative estimate for $Q^r$, and a certain reduction in the probability of policy constraint violation attributable to the conservative estimate for $Q^{c_i}$.   \n\n\n[1] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.\n\n"}]

Author rebuttal:
Dear reviewers:   
We thank all reviewers for your time and suggestions, and we expect to have a further discussion. We have responded to your questions in detail accordingly. If you have further questions or concerns, we still reply before the end of the author-reviewer discussion. Thank you very much for your review time and efforts.   

+ __How should the hyperparameters $\kappa$, $\chi$, and $\epsilon$ be set?__  
__A:__  **(1)** Regarding the hyperparameters $\kappa$ and $\chi$ mentioned in the manuscript, setting them too small makes it difficult to guarantee the algorithm's performance theoretically, while setting them too large leads to conservative behavior and a decrease in algorithm performance. Therefore, we adopt a dual-tuning approach with gradient-based adaptation instead of a manual setting. To facilitate the observation of the variations in trade-off factors $\kappa$ and $\chi$, we have documented the evolution curves of these factors during the training process. The variation curve of the trade-off factor $\kappa$ and $\chi$ during the model training process is depicted in Fig.2 . The results shown in the figure indicate that as the model gradually converges, the trade-off factor $\kappa$ and $\chi$ diminishes progressively and eventually stabilizes at a value greater than zero.  
**(2)** Regarding the manually set hyperparameter $\epsilon$, we have conducted ablation experiments as shown in Fig.1 . The results displayed in the figure indicate that setting the parameter $\epsilon$ too small would reduce the convergence speed of the policy and may even diminish the algorithm's overall performance. Conversely, when $\epsilon$ is set too large, policy instability can arise. The experimental outcomes depicted in the graph demonstrate that within this range of $\epsilon \in [0.1,1]$, it ensures a favorable balance between convergence and stability for the policy. The parameter $\epsilon \in [0.1,1]$ represents a feasible range rather than an optimal interval.

+ __Limitations and Future Work__  
__A:__ The setup of this study involves learning policies that satisfy safety constraints from offline data without interacting with the environment. Consequently, the size and quality of the dataset have a direct impact on the algorithm's performance. Furthermore, within the context of the offline setting, the limited sample data available makes it challenging to adequately represent the state transition matrix p in non-stationary environments. As a result, the VOCE algorithm faces difficulties in learning high-reward policies that satisfy safety constraints in non-stationary environments. We will supplement the discussion of the limitations of this work in the final version.

Meta-review: The paper tackles Safe offline RL using Conservative Q Learning in a variational optimization framework, over set of policies that satisfy the cost constraints. The reviewers are mostly positive and the biggest criticisms are around hyperparameter choice, runtime and clarity on limitations. My own assessment on the paper is that the method is a combination of several known ML ideas albeit the application to Safe offline setting is new.

There was rich discussion between the authors and the reviewers, where a lot of technical aspects of the work were clarified. My recommendation is accept based on the strength of the reviews as well as the insightful discussion.
Decision: Accept (poster)