Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Authors: Yinan Zheng
Jianxiong Li
Dongjie Yu
Yujie Yang
Shengbo Eben Li
Xianyuan Zhan
Jingjing Liu
Venue: ICLR 2024 poster (2024)
Keywords: Safe offline reinforcement learning
Hamilton-Jacobi reachability
diffusion model
Primary area: reinforcement learning
TL;DR: We propose a safe offline RL method that considers hard safety constraint with superior safety performance and stability.

Abstract: Safe offline reinforcement learning is a promising way to bypass risky online interactions towards safe policy learning. Most existing methods only enforce soft constraints, i.e., constraining safety violations in expectation below thresholds predetermined. This can lead to potentially unsafe outcomes, thus unacceptable in safety-critical scenarios. An alternative is to enforce the hard constraint of zero violation. However, this can be challenging in offline setting, as it needs to strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regularization imposed by offline datasets. Interestingly, we discover that via reachability analysis of safe-control theory, the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset. This seamlessly converts the original trilogy problem to a feasibility-dependent objective, i.e., maximizing reward value within the feasible region while minimizing safety risks in the infeasible region. Inspired by these, we propose FISOR (FeasIbility-guided Safe Offline RL), which allows safety constraint adherence, reward maximization, and offline policy learning to be realized via three decoupled processes, while offering strong safety performance and stability. In FISOR, the optimal policy for the translated optimization problem can be derived in a special form of weighted behavior cloning, which can be effectively extracted with a guided diffusion model thanks to its expressiveness.  We compare FISOR against baselines on DSRL benchmark for safe offline RL. Evaluation results show that FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks. Code: https://github.com/ZhengYinan-AIR/FISOR.

Published as a conference paper at ICLR 2024
SAFE OFFLINE REINFORCEMENT LEARNING WITH FEASIBILITY-GIDDER DIFFUSION MODEL
Yinan Zheng Jianxiong Li!:2e, Dongjie Yu², Yujie Yang", Shengbo Eben Li²,
Xianyuan Zhan Jingjing Liu
1 Institute for AI Industry Research (AIR), Tsinghua University
2 School of Vehicle and Mobility, Tsinghua University
3 Department of Computer Science, The University of Hong Kong
Shanghai Artificial Intelligence Laboratory
(z hzhnnyn22,, 1 - jx21 @ma i 1s.. s nghua. edu. . en 2 chanxianyuane r. ts inghua edu.cn cn
ABSTRACT
Safe offline reinforcement learning is a promising way to bypass risky online in- teractions towards safe policy learning. Most existing methods only enforce soft constraints, i.e., constraining safety violations in expectation below thresholds predetermined. This can lead to potentially unsafe outcomes, thus unacceptable safety-critical scenarios. An alternative is to enforce the hard constraint of zero vi- olation. However, this can be challenging in offline setting, as needs to strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regularization imposed by offline datasets. Interestingly, we discover that via reachability analysis of safe-control theory, the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset. This seamlessly converts the orig- inal trilogy problem to a value within the feasible region while minimizing safety risks in the infeasible re- gion. Inspired by these, we propose FISOR Freasbbiltt-guidee Safe Offliee RL), which allows safety constraint adherence, reward maximization, and offline policy learning to be realized via three decoupled processes, while offering strong safety performance and stability. In FISOR, the optimal policy for the translated opti- mization problem can be derived in special form of weighted behavior cloning, which can be effectively extracted with guided diffusion model thanks its expressiveness. Moreover, we propose novel energy- -uidddd sampling method that does not require training a complicated time-dependent classifier to simplify the training. We compare FISOR against baselines DSRL benchmark for safe offline RL. Evaluation results show that FISOR is the only method that can guar- antee safety satisfaction in all tasks, while achieving top returns in most tasks. Project website: httpps: /tt:://heenee /zengyy nan-air. g thub. i 10/FISOR/
objective, i.e., maximizing reward
INTRODUCTION
Autonomous decision making imposes paramount safety concerns in safety-critical tasks (Li, 2023), such as industrial control systems (Zhan et al., 2022) and autonomous driving (Kiran et al., 2021),
where any unsafe outcome can lead to severe consequences. In these tasks, one top priority is to en- sure persistent safety constraint satisfaction (Zhao et al., 2023). Safe reinforcement learning (RL) holds the promise of safety guarantee by formulating and solving this problem as a Constrained Markov Decision Process (CMDP) (Gu et al., 2022). However, most previous studies focus on online RL setting, which suffers from serious safety issues in both training and deployment phases, especially for scenarios that lack high-fidelity simulators and require real system interaction for pol- icy learning (Liu et al., 2023a). Safe offline RI., on the other hand, incorporates safety constraints into offline RL (Levine et al., 2020) and provides new alternative for learning safe policies in fully offline manner (Xu et al., 2022b). In this setting, only the learned final policy needs safety guar-
"qqual contribution
Correspondence Xianyuan Zhan and Jingjing Liu

---

Published as a conference paper at ICLR 2024
antees. bypassing the inherent safety challenges during online training, thus providing an effective and practical solution to safety-critical real-world applications (Zhan et al., 2022).
Though promising, current safe offfine RL methods exhibit several limitations. Firstly, to the best of our knowledge, all previous studies adopt seft constrains and only require constraint violations in expectation to remain under given threshold throughout the entire trajectory (Xu al., 2022b; Lee et al., 2022), thus allowing potential unsafe outcome. Such soft constraint-basee widely employed in the research community but unacceptable to safety-critical industrial applica- tion (Yang et al., 2023a). Instead, hard construint is more preferable for these tasks, where state-wise zero constraint violation is strictly enforced. To achieve this, a far more rigorous safety constraint is in demand, which however, inevitably imposes severe policy conservatism and suboptimality. Sec- ondly, the difficulty is further exacerbated by the offline setting, as additional consideration of be- havior regularization imposed by offiie dataset is required to combat distributional shift (Fujimoto et al., 2019). Thus, striking the right balance among constraint satisfaction, reward maximization, and offline policy learning can be exceptionally challenging. Jointly optimizing these intricate and closely-correlated aspects, as in existing safe offline RL methods, can lead to very unstable training processes and unsatisfactory safety performance (Lee et al., 2022; Saxena & Cao, 2021).
formulation is
To tackle these challenges, we introduce a novel safe offline RL approach, FISOR (Feaslbility- guided Safe Offline RL), which provides stringent safety assurance while simultancously optimiz- ing for high rewards. FISOR revisits safety constraint satisfaction in view of optimization under different feasibility conditions and comprises three simple decoupled learning processes, offering strong safety performance and learning stability. Specifically, we first revise Hamilton-Jacobi (HJ) Reachability (Bansal et al., 2017) in safe-control theory directly identify the largest feasible re- gion through the offline dataset using a reversed version of expectile regression, which ensures zero constraint violation while maximally expanding the set of feasible solutions. With the knowledge of feasibility in hand, we further develop a fcasibility- dependent objective that maximizes reward values within the feasible regions while minimizing safety risks in the infeasible regions. This al- lows constraint satisfaction and reward maximization to be executed in decoupled offline learning processes. Furthermore, we show that the optimal policy for these reformulated problems has a special weighted behavior cloning form with distinct weighting schemes in feasible and infeasible regions. By noting the inherent connection between weighted regression and exact energy- guided sampling for diffusion models (Lu et al., 2023), we extract the policy using a novel time- indeenndeee classifier classifier-guided guided method, enjoying both superior expressivity and efficient training.
Extensive evaluations the standard safe offline RL benchmark DSRL (Liu al., 2023a) show that FISOR is the only method that guarantees satisfactory safety performance in all evaluated tasks, while achieving the highest returns in most tasks. Moreover, we demonstrate the versatility of FISOR in the context of safe offline imitation learning (IL), still outperforming competing baselines.
2 PRELIMINARY
Safe RI. is typically formulated as a Constrained Markov Decision Process (Altman, 2021), which is specified by tuple M := (S,A, T, r. h, 7). S and A represent the state and action space; T S x A + ASS) is transition dynamics; r S x A + IR is the reward function; : + IR is the constraint violation function;
S + [0, Cmax] is cost function; and y (0, 1) is the discount factor. Typically, c(s) = max (ss),, 0). which means that it takes on the value of h(s) when the state constraint is violated (h(s) > 0). and zero otherwise (h(s) < 0).
Previous studies typically aim to find a policy π. : + ((A) to maximize the expected cumulative rewards while satisfying the soft constraint that restricts the expected cumulative costs below pre- defined cost limit 2 0, Le., max, E " at)), s.t. E,t, >'c(s)) < I, where r is the trajectory induced by policy æ. Existing safe offline RL methods typically solve this problem in the following form (Xu et al., 2022b; Lee et al., 2022):
max E, (s)]
s.t. E. IV (+)) <4 D( S
where V.TT is state-value function, V is cost state-value function, Tg is the underlying behavioral policy of the offline dataset D == (s, a, s', r, c) with both safe and unsafe trajectories. (() is a divergence term (e.g., KL divergence DxL. ( (T)) used to prevent distributional shift from T3 in offline setting. To handle safety constraints, previous methods often consider solving the Lagrangian
2

---

Published as a conference paper at ICLR 2024
Reach-avoid Control
Offline learned feasible region under different velocity and yaw angle
Optimal Feasible value funntion
Agent
Hazard
(eamed)
0.8
0.0
-166
-2.4
(a) Task description
V
Optinal cool valiee linetion
(kamed)
6.0
4,5
3.0
1.5
0.0
High
O Region with unsafe states C Infeasible region (ground truth)
Infeasible region (learee))
(b) Data distribution
(c) 0.5,0=
(d) 1,0 /
(e) 1.5, 7= Ø /
Figure 1: (a) Reach-avoid control task: the agent (red) aim to reach the goal (green) while avoiding hazards
(bluc). (b) Offline data distribution. (c)-(c) Comparisons with the feasible region learned by feasible value
{s|V (s) s 0} and cost value {s|Va " (s) I -3 See Appendix for more details.
dual form of Eq. (1) with Lagrangian multiplier À (Chow et al., 2017; Tessler et al., 2018):
min max E, [V. X (V." (s) - [)] s.t. D( c. X20 x
Limitations of Existing Methods. The safety constraint Eq. (1) is a soft constraint enforced on the expectation of all possible states, meaning that there exists certain chance of violating the constraint with a positive 4, which poses serious safety risks in real-world scenarios (Ma et al., 2022). Additionally, choosing suitable cost limit that yields the best safety performance requires engineering insights, whereas the optimal limit may vary across tasks (Yu et al., 2022a). Besides, Eq. (1) shows that safe offline RL necessitates the simultaneous optimization of three potentially conflicting objectives: maximizing rewards, ensuring safety, and adhering to the policy constraint w.r.t the behavior policy. Finding the right balance across all three very difficult. Moreover, the optimization objective in Eq. (2) introduces an additional layer of optimization for the Lagrange multiplier, and the learning of V,", V and π are mutually coupled, where minor approximation emror can bootstrap across them and lead to severe instability (Kumar et al., 2019).
METHODS
Feasibility-Depeene Optimization. To address these challenges, we replace soft constraint with hard constraints that demand state-wise zero violations (h(ss) S 0, a VI N. Appendix A.2). straightforward way to remove the expectation and set the cost limit to zero, i.e., replacing the safety constraint in Eq. (1) with V < O:
max E, (s)]
s.t. (8) < O; D(TT|ITT) S.
(3)
However, this leads to highly restricted policy set (s) (z)) 0). since V is non- negative (c 2 0) and the presence approximation errors makes difficult to take the exact value of 0 (Figure 1). To address this issue. We replace (s) with new safe policy set (s) suitable for practical solving. Furthermore, not all states allow the existence of policy that satisfies the hard constraint, rendering the problem unsolvable at these states. We refer to these infeasible states and the others feasible states. To solve this, we consider infeasible states separately and modify the problem follows (Appendix B.I shows theoretical relationships between Eq. (3) and Eq. (4)):
Feasible: max E. [V," (s) [ees,]
Infeasible: max E, [V-: (s) Iops,
s.t. ग (s), Vs Sy (TT Se
s.t. (TT((T Se
(4)

---

Published as a conference paper at ICLR 2024
Eq. (4) presents a feasibility-dependee objective. Here, II is the indicator function, and Sy is the feasible region including all feasible states, for which at least one policy exists that satisfies the hard constraint. This objective allows to concentrate safety risks individually to obtain the best possible performance. In the case of infeasible states, maximizing rewards becomes meaningless since safety violations will always occur. Therefore, the focus is shifted to minimizing future constraint violation as much as possible (Yu et al., 2022a). For feasible states, adherence to the hard constraint ensured by policies from the safe policy set y, thus allowing for the focus on reward maximization. Next, we introduce using the reachability analysis from safe-control theory to determine Sj and yl
maximizing rewards and minimizing
3.1 OFFLINE IDENTIHCATION OF FEASIBILITY
Accurately determining Sj and y plays a crucial role in the success of the feasibility objective. This reminds of the effectiveness of Hamilton-Jacob (HJ) reachability (Bansal et al., 2017) for enforcing hard constraint in safe-control theory, which has recently been adopted in safe online RL studies (Fisac et al., 2019; Yu et al., 2022a). We first provide brief overview of the basics in HJ reachability (Definition 1) and then delve into the instantiation of the feasible region S and policy set I1, in the HJ reachability framework (Definition 2-3).
Definition 1 (Optimal feasible value function). The optimal feasible state-value function Vi and the optimal feasible action-value function Qi are defined as (Bansal el al., 2017):
(6) =- min V22 (s) == min max h(()), 80 8, a, T(- |s),,
(5)
EEN
(s) min 07 (s, = min max (se)), 80 8, 00 a, a++1 T(. S1+1),
(6)
EEN
where VW2 represents the maximum constraint violations in the trajectory induced by policy x start- ing from a state s. The (optimal) feasible value function possesses the following properties:
(s) < 0 Vs,, h(s)) S 0, indicating can satisfy the hard constraint starting from Moreover, (s) S 0 + ming VI S + 3τ, VT S O, meaning that there exists a policy that satisfies the hard constraint.
(s) > 0 = As,, h(8)) > 0, suggesting π cannot satisfy the hard constraint; and (s) > 0 ming V
Vn, V > O, meaning that no policy can satisfy the hard constraint.
We can see from these properties that the feasible value function indicates whether policies can sat- isfy hard constraints (i.e., feasibility of policies); while the optimal feasible value function represents whether there exists a policy that achieves hard constraints (i.e., feasibility of state), from which we define the feasible region (Definition 2) (Yu et al., 2022a) and feasible policy set (Definition 3): Definition (Feasible region). The feasible region of म and the largest feasible region are:
S = {s} Vs (s) S 50} Definition 3 (Feasible policy set). For state s, the feasible policy set is: I (s] S 0}.
ST [s|V (s) 0}
S7 is induced by policy л and includes all the states in which π can satisfy the hard constraint. S7 is the largest feasible region, which is the union of all feasible regions (Choi et al., 2021; Li, 2023). Intuitively, Sj includes all the states where there exists at least one policy that satisfies hard constraint, and this property formally satisfies the requirement of feasible region in Eq. (4). Besides, (s)(s) includes all policies that satisfy hard constraint starting from state 8. This set can be employed as constraint in Eq. (4). Note that the range of V4, is the entire real number space, making V,, more suitable as a hard constraint compared to Ve, as shown in Figure 1.
Learning Feasible Value Function from Offline Data. Although HJ reachability holds the promise for enforcing hard constraint, calculating the optimal feasible value function based on Definition requires Monte-Carke estimation via interacting with the environment (Bajesy et al., 2019), which is inaccessible in offline setting. Fortunately, similar to approximate dynamic programming, we can obtain the approximated optimal feasible value by repeatedly applying a feasible bellman operator with a discounted factor y + 1 (Fisac et al., 2019) (Definition 4).
Definition 4 (Feasible Bellman operator). The feasible Bellman operator defined below is a con- traction mapping, with its fixed point Vi,77 satisfying lim Di, + Q7, (Fisac et al., 2019).
P' p*Qn(s,a) = (1 -1-27 ())))((s) + mmmx(((s)), ((s) Vi((') min Qn ,o)
4

---

Published as a conference paper at ICLR 2024
Given the offline dataset D, approximate Q% by minimizing Ep PP Qn)".. Note that the min," operation in Eq. (7) could query the out-of-distribution (OOD) actions, potentially leading to severe underestimation issues in offline setting (Fujimoto et al., 2019). To mitigate this, consider a data support constrained operation ming' m that aims to estimate the minimum value over actions that in the support of data distribution, but this requires an estimation of the behavior policy тg. Inspired by Kostrikov et al. (2022) that uses expectile regression to approximate the maximum value function without explicit behavioral modeling, adopt a reversed version of this for learning the optimal (minimum) feasible value function via minimizing Eq. (8)-(9):
Lv. (s,a)/ 1T (Qn a) Vi)(,
Lq. = 12 (,,,, [((1 - (s)) + max max()(()), (,()), Qn((, - [))],
where Levv (u) = IT- I1u > 0)]u², For T E (0.5, 1), this asymmetric loss diminishes the impact of Qn values that exceed Vn. placing greater emphasis on smaller values instead. Subsequently, can substitute V/, (s') for min.' Qn aa') within the feasible Bellman operator. By minimizing Eq. (8) and Eq. (9), we can predetermine the approximated largest feasible region, shown in Figure
3.2 FRASIBILITT- OPTIMIZATION OBJECTIVE
Given the feasible policy set y and the predetermined largest feasible region Sy. we can instantiate the feasibility objective in Eq. (4) as follows, with Ve changed to V₁2-
Feasible: max E. (s) -Less]
Infeasible: max E, [- 1. (s) Tugs))
s.t.
5,
s.t. (T] S
(10)
(TT(T S
Eq. (10) is similar to the objective in the safe online RL method, RCRL (Yu al., 2022a). However, RCRL suffers from severe training instability due to its coupling training procedure. In contrast, predetermine Sj from the offline data. However, the training of V,T. V and is still coupling together, leading to potential training instability. To solve this, we show in Lemma 1 that these interrelated elements can be fully decoupled. Based on this insight, we devise a new surrogate learning objective that disentangles the intricate dependencies among these elements.
Lemma 1.
The optimization objectives and safety constraints Eq. (10) can be achieved by sepa- rate optimization objectives and constraints as follows (see Appendix B.2 for proof):
Optimization objective in the feasible region: max, Ensa [A, a)] max, E, [V7T
Optimization objective in the infeasible region: max, Eavz | a)l max, E, |- V% (s)).
Safety constraint in the feasible region: [(a10) T (-Is)da-
< 0.
Lemma 1 shows that the maximization objectives in Eq. (10) can be achieved by finding the optimal advantages A7,, Aj first, and then optimizing the policy т. The safety constraints in Eq. (10) can also be enforeed by using the predetermined Qi, without coupling to other networks. Inspired by this, we propose surrogate objective in Eq. (11) that fully decouples the intricate interrelations between the optimizations of V,", VØ and æ. We dub this FISOR (Feaslbility- guided Safe Offline RI)::
Feasible: max Eant [A)(s,a) Tugoss]]
Infeasible: max E As, Img(osoo]
s.t.
S (1))))))))
<als)daa 1,
T(1s)da== s.t.
(11)
(TT|TT]) <
DRa Se
We present an intuitive illustration in Figure 2 to demonstrate the characteristics of Eq. (11). In feasible regions, FISOR favors actions that yield high rewards while ensuring that the agent remains within feasible regions. The largest feasible region grants the policy with more freedom to maxi- mize reward, allowing for enhanced performance. As depicted Figure 2, agents originating from feasible regions can effectively find direct and safe path to the goal.
In infeasible regions, FISOR chooses actions that minimize constraint violations as much as possi- ble. Figure 2 shows that the agents starting from infeasible regions initially prioritize escaping from infeasible regions, and then navigate towards the destination along safe paths.
5

---

Published as a conference paper at ICLR 2024
The decoupled optimization objective in Eq. (11) allows us to directly obtain the closed-form solution as shown in Theorem 1 below (sce Appendix B.3 for detailed proof):
Start with the same velocity and yaw
dearal 0.8
Theorem 1. The optimal solution for Eq. (11) satisfies:
x (als) x TT(aas)) w(s,a),
(12)
where w(s a) is the weight function, and on, 02 are tempera- tures that control the behavior regularization strength:
-088
-1.6
exp (a1 (s, a)) [ogg(s.))
(s) 0
w(s a)
(13)
-24
exp (=p((-aAi
VI (s) > 0
1556 2//
Trajectory start from feusible region Trajectory start from infessible region
The optimal solution in Eq. (13) is similar to weighted behav- ior cloning. but adopts distinet weighting schemes feasible and infeasible regions. Using Eq. (13), we can easily extract the optimal policy in a remarkably simple and stable way, while
Figure 2:
Trajectories induced by FISOR from different start points.
achieving balance between safety assurance, reward maximization, and behavior regularization.
3.3 GUIDED DIFFUSION POLICY LEARNING WITHOUT TIME-DEPENDEN CLASSIFIER
Theorem shows that the optimal policy ne needs choose drastically different behaviors based on the current state's feasibility. We use diffusion model to parameterize policies due to its strong abil- ity in modeling complex distributions (Ho et al., 2020; Wang et al., 2023c). To obtain the weighted distribution of x² in Eq. (12), a simple way is to use classifier dussifier-guidance methods to guide the diffu- sion sampling procedures (Lin al., 2023; Lu et al., 2023). However, such methods require training an additional complicated dime-dependent classifier, inevitably increasing the overall complexity. Instead, we demonstrate in Theorem that the exact n in Eq. (12) can in fact be obtained without the training of time-dependent classifier, thus greatly reducing training difficulty.
Theorem (Weighted regression as exact energy guidance). We can sample a x (a)s) by op- timizing the weighted regression loss in Eq. (14) and solving the diffusion ODEs/SDES given the obtained zg. (see Appendix C for proof and thorough discussions)
Specifically, the exact nT can be obtained by modifying the diffusion training loss for the behavior policy Tgt ming (s,(s.a). [||= zg (az, s, t) [)] via simply augmenting the weight function w(s a) in Eq. (13) with the following weighted loss:
min (0) = min 8
(w(s,a) a) -|=- (ar, 8, 011)
(14)
0
where at = aga + 0,2 is the noisy action that satisfies the forward transition distribution ((a loya, T) in the diffusion model, and at, dg are human designed noise schedules. After obtaining 6, we can sample from the approximated optimal policy ~ =a(ass by solving the diffusion ODEs/SDEs (Song et al., 2021) as stated in Theorem 2. Eq. (14) bypasses the training of the complicated time-dependent classifier in typical guided diffusion models (Lin et al., 2023; Lu et al., 2023) and forms a weighted regression objective, which has proved stable against directly maximizing the value functions (Kostrikov et al., 2022). This weighted loss form has been used in recent studies (Hstscn-Fstruuch et al., 2023; Kang et al., 2023), but they did not provide theoretical investigation in its inherent connections to exact energy guidance for diffusion policies (Lu et al., 2023) (see Appendix C.3 for comparison with other guided sampling methods).
3.4 PRACTICAL IMPLEMENTATION
Theorem 1 offers a decoupled solution for extracting the optimal policy from the dataset. The remaining challenge is how to obtain the optimal advantages A in advance. In-sample learning methods, such as IQL (Kostrikov et al., 2022), SQL (Xu et al., 2023) and XQL. (Garg et al., 2023), decouple policy learning from value function learning and allows the approximation of A = Q V." without explicit policy, offering improved training stability. We use IQL to learn the optimal value function Q and V,* in our method with (T(u) |T I[u < 0)) u2. E 10.
Lv, = [( [T.T (Qr a) V,(s))),
(15)
Lq. ((,,,,,)) (r + 7V,(ss) -
(16)

---

Published as a conference paper at ICLR 2024
Table 1: Normalized DSRL (Liu et al., 2023a) benchmark results. t means the higher the better, & means the lower the better. Each value is averaged over 20 evaluation episodes and seeds. Gray: Unsafe agents. Bold: Safe agents whose normalized cost is smaller than 1. Blue: Safe agents with the highest reward.
BC
CDT
BCQ-Lag
CPQ
COpiDICE
TREBI
FISOR (ours)
Tisk reward
cost
reward
cost
reward
cost
reward
cost
reward
cost
reward
cOSE
reward
cost
CarBuitonl
0.01
6.19
0.17
7.05
0.13
6.68
0.22
40.05
-0.16
4.63
0:007
3.75
-0.02
0.26
CarButton2
-0.10
4.47
0.22
12.87
-0.04
4.63
0.05
19,03
-0.17
3.40
-0.03
0.97
0.01
0.58
CarPushi
0.21
CarPush2
0.11
3.89
2:78
0:10
4.55
0:12
CarGoall
0.35
1.54
0.00
3:15
0.44
2.76
0.33
4,93
0.43
2.81
0.41
1.16
0.49
0.83
Carfool22
0.22
3.30
0:45
6.05
0.34
4.73
0.10
6.31
0.19
2.83
0.13
1.16
0.06
0.33
AntVal
0.99
12.19
0.98
0.91
0.85
18.54
-1.01
0.00
1.00
10.29
0.31
0.00
0.89
0.00
HaliChontaVel
0.97
17.93
0.97
0.55
1.04
57.05
0.08
2.56
0.43
0
0.87
0.23
0.89
0.00
SwammerVel
0.38
2.98
0.67
0.29
23.64
SyfedyGym
Average
0.35
11.38
AniRon
0.73
11:73
0.65
0..00
0.00
0.62
0.63
0.13
BailRun
0.67
11.38
0.32
0.45
0:43
6.25
0.85
13.67
0.55
11.32
0.29
4,24
0.18
0.00
CarRun
0.96
1.88
0.99
1.10
0.84
2.51
1.06
10.49
0.92
0,00
0.97
1.01
0.73
0.14
Droaekun
0.55
5:21
0.58
0.30
0.850
17.58
01,02
7:95
0.72
13.77
0.59
1.41
0.30
0.55
AntCrele
0.65
19.45
0:48
7:44
0.67
19.13
11.00
0,00
0.18
13.41
0.37
2.50
0.20
0.00
BallCiille
0.68
CarCircle
DroneCirele
0.82
13.78
0.55
1.29
0.95
18.56
-0:27
1.29
0.24
2.19
0.54
2.36
0.48
0.00
BulletGyym
Average
0.72
10.58
0.63
2.00
0.71
10.63
0.32
5.28
0.55
7.64
0.56
2.45
0.39
0.10
easysparse
0.32
4.73
0.86
0.10
0.99
14.00
-0.016
0.34
0.94
18.21
0.26
6.22
0.38
0.53
essymean
0.22
2.68
0.27
0.24
0.54
10.35
-0.166
11.34
0.74
14.81
0:19
4.85
0.38
0.25
easydense
0.20
1.70
0:43
2:31
0.40
6.61
-0.06
0.29
11.27
0.26
5.81
0.36
0.25
modiumquuae
mediummean
2.94
mediunddense
0.65
3.79
0.29
-0.08
0.20
0.91
9.52
0:03
haddspasee
0.28
1.98
0.17
0.47
0.48
7.52
.0.04
0.28
0.34
7.34
0.00
0.82
0.30
0.01
hundmam
0.34
3.76
0.78
3.52
0.51
6.06
-8.05
0.24
0.36
7.51
0.16
4.91
0.26
0.19
harddense
0.40
5.57
0.24
1.49
0.39
5.11
-0.04
0.24
0.42
8,11
0.02
1,21
0.30
0.34
MetaDrive
Average
3.21
0:25
-0.06
0.63
0.36
0.25
So far, we have transformed the original tightly-couledd safety-constraained offline RL problem into three decoupled simple supervised objectives: 1) Offline identification of the largest feasible region (Eq. (8-9)); 2) Optimal advantage learning (Eq. (15-16)); and 3) Optimal policy extraction via guided diffusion model (Eq. (14)). This disentanglement of learning objectives provides a highly stable and conducive solution to safe offline RL for practical applications. To further enhance safety, we sample N action candidates from the diffusion policy and select the safest one (i.e., the lowest Qi value) as the final output as in other safe RL methods (Dalal et al., 2018; Thananjeyan et al., 2021). Please see Appendix D.3 and D.4 for implementation details and algorithm pseudocode. Code is available at: https: : / /i thub. com/ / Y nann AIR/FAIRRROSSSA
4 EXPERIMENTS
Evaluation Setups. We conduct extensive evaluations on Seeett-Gymmaiium (Ray et al., 2019; Ji et al., 2023), Bullet-Safety Safety-Gym (Gronauer, 2022) and MetaDrive (Li et al., 2022) tasks DSRL benchmark (Liu et al., 2023a) to evaluate FISOR against other SOTA safe offline RL methods. We use normalized return and normalized cost as the evaluation metrics, where a normalized cost below 1 indicates safety. As stated in DSRL, we take safety as the primary criterion for evaluations, and pursue higher rewards on the basis of meeting safety requirements. In order to mimic safety- critical application scenarios, we set more stringent safety requirements, where the more difficult task anfety-Gymmasium has its cost limit set to 10, and other environments are set to 5.
Baselines. We compare FISOR with the following baselines: 1) BC: Behavior cloning that imitates the whole datasets. 2) CDT (Liu al., 2023b): A Decision Transformer based method that considers safety constraints. 3) BCQ-Lag: A Lagrangian-basse -baed method that considers safety constraints on the basis of BCQ (Fujimoto et al., 2019). 4) CPQ (Xu et al., 2022b): Treat the OOD action as an unsafe action and update the Q-value function with safe state-action. 5) COptiDICE (Lee et al., 2022): A DICE (distribution correction estimation) based safe offline RL method that builds on OptiDICE (Lee et al., 2021). 6) TREBI (Lin et al., 2023): A diffusion- based method using classifier-guidance to sample safe trajectory. See Appendix D for more experimental details.
Main Results. Evaluation results presented in Table 1. FISOR is the only method that achieves satisfactory safety performance in all tasks and obtains the highest return in most tasks, demon- strating its superiority in achieving both safety and high rewards. Other methods either suffer from
7

---

Published as a conference paper at ICLR 2024
CarPush1
AntClrcle
mediummen
Noralized Rttum
0.9
0.9
0.2
0.0
(.0
CDT COpIDICE (PQ DCQ-Lag TREBI F1908
CDT O0piDICE CPQ BOQ-Lag TREDI FISOR
CDT COPIDCC CPQ BCQ-aa; TREII FISOR
1.50
Cost
01
COPiDICE CPQ BEQ-Lag TREGI FISOR
CUT OQpiDICE
BOQ-Lag TREDI FISOK
CUT COgiDDCC CPQ BCQ TREMI TREII FISOR
1=
=0
=
FISOR (ours)
FISOR wlo construint
Figure 3: Soft constraint sensitivity experiments for cost limit / in three environments.
Normalized Return
Normalized Cost
Normalized Return
Normalized Cost
0.4
0.50
0.2
6
0.25
4
0.0
2
0.22
-0.25
0.00
0.25
0.30
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00)
0.00
0.25
0.50
0.75
1.00
Train Steps
le6
Train Steps
le6
Train Steps
lc6
Train Steps
1e6
r-0.95
r-0.9 (ours)
r-0.8
r-0.7
N-256
N-64
N-16 (ours)
N-4
(a) Ablation reverse expectile parameter
(b) Ablation on the number of action candidates
Figure 4: Ablations on hyperparameter choices CarPushl.
severe constraint violations or suboptimal returns. BCQ-Lag tries to balance performance and safety using the Lagrangian method, but the coupled training procedure makes it unstable and difficuul to find the right balance, leading to inferior safety and reward results. CPQ takes a more conservative approach by only updating the value function safe state-action pairs, which can result in very low returns when constraints are enforced. CDT and TREBI use complex network architectures, which meet the constraints and achieve high returns on some tasks, but their overall safety performance is not very impressive since they use soft constraints.
4.1 ABLATION STUDY AND ANALYSIS
Soft Constraint Under Different Cost Limits. We evaluate the sensitivity of cost limit selection | for poft-constraint-bacee methods to demonstrate the effectiveness of hard constraint. We evaluate three cost limits and present the results in Figure FISOR w/o constraint refers to our algorithm without safety constraints, focusing solely maximizing rewards. Figure 3 shows that most soft- constraint-brat methods are highly sensitive to the value of cost limit. In some cases, choosing a small cost limit even leads methods to select the right cost limit to achieve the best performance, which requires task-specific tuning. In contrast, and achieves superior results using only one group of hyperparameters.
increase in the final cost. This shows that it is difficult for these
algorithm, which considers hard constraints, does not encounter this issue
Hyperparameter Choices. We sweep two hyperparameters: the expectile value and the number of sampled action can- didates N during evaluation. Figure 4 shows that and N primarily affect conservatism level, but do not significantly impact safety satisfaction. A larger and smaller N result in more aggressive policy, whereas a smaller 7 and larger N lead to a more conservative behavior. We find that setting 0.9 and N 16 consistently yields good results in terms of both safety and returns. Hence, we evaluate FISOR on all 26 tasks using these same hyperparameters. Please refer to Appendix D.5 for detailed hyperparameter setups.
Table 2: Ablations on reachability,
w/o
FISOR
Task
reward
reward
AmtRn
0.30
0.44
0.45
0.03
BallRan
0.08
0.14
0.18
0.00
CarRun
-0.33
0.00
0.73
0.14
DroneRun
-0.11
5.12
0.30
0.55
AntCiscle
0.00
0.00
0.20
0.00
BallCircle
0.02
0.81
0.34
0.00
CarCircle
0.01
2.16
0.11
Ablations Each Design Choice.
We demonstrate the effectiveness of individual components of our method: reachability, feasibility objective, and diffusion model. 1) To verify the advantage of
DroneCircle
-0.21
0.26
0.48
0.00
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablations infeasible objective and diffuusion policies (normalized cost).
Return
Cost
250
40
200
Tisk
w/o infeasible wio diffusion
FISOR
30
150
CarButton 4.61
0.72
0.26
20
100
CarButton2
6.53
1.73
0.58
CarPushl 0.86
1.21
0.28
50
10
CarPush2 3.10
4.45
0.89
medium-s medium-m medium-d
modium-s medium-m medium-d
CarGoall
4.00
2.71
0.83
BC
BC.P
BCLL
FISOR (ours)
CarGoal2
2.82
Figure 5: Safe offline results in MetaDrive.
using HJ reachability, ablate the choice of determining the largest feasible region using cost value function instead of feasible value function. This variant is denoted as w/o HJ. The results are summarized in Figure and Table 2. For the variant w/o HJ, the presence of approximation errors renders the estimated infeasible region pretty large and inaccurate, which leads to severe policy con- servatism and hurt final returns. By contrast, the feasible value function in FISOR can accurately identify the largest feasible region, which greatly increases optimized returns. 2) We train FISOR by setting the weight for the infeasible region in Eq. (13) to 0 (w/o infeasible) to examine the neces- sity of minimizing constraint violation in infeasible regions. 3) We also ablate on replacing diffusion policy with conventional Gaussian policy (w/o diffissio). The results are presented in Table 3, where w/o infeasible suffers from severe constraint violations without learning recovery behavior in infea- sible regions. Mcanwhile, w/o diffusion struggles to effectively fit the data distribution both inside and outside the feasible region, resulting in high constraint violations.
Extension to Safe Offline Imitation Learning. We also show the versatiliiyy of FISOR by extending it to the safe offline Imitation Learning (IL) setting, where aim to learn safe policy given both safe and unsafe high-reward trajectories without reward labels. In comparison to 1) BC, 2) BC with fixed safety penalty (BC-P), and 3) Lagrangian penalty (BC-L), FISOR can imitate expert behavior and meanwhile avoid unsafe outcomes, as illustrated in Figure 5. See Appendix D.6 for details.
RELATED WORK
Most existing safe RL studies focus on the online setting, which typically use the Lagrangian method to solve the constrained problem (Chow et al., 2017; Tessler et al., 2018; Ding et al., 2020), thus sulting in an intertwined problem with stability issues (Saxena & Cao, 2021). CPO (Achiam et al., 2017) utilizes trust region technique and theoretically guarantees safety during training. However, all these methods cannot guarantee safety during the entire online training phase. By contrast, safe offline RL learns policies from offline datasets without risky online interactions. CPQ (Xu et al., 2022b) is the first practical safe offline RL method that assigns high-cost values to both OOD and unsafe actions, which distorts the value function and may cause poor generalizability (Li et al., 2023b). COptiDICE (Lee et al., 2022) is a DICE-based method (Lee et al., 2021) with safety con- straints, but the residual learning of DICE may lead to inferior results (Baird, 1995; Mao et al., 2023). Recently, some methods (Liu et al., 2023b; Lin et al., 2023) incorporate safety into Decision Transformer (Chen al., 2021) or Diffuser (Janner et al., 2022) architecture, but are highly com- putational inefficient. Moreover, all these safe offline RL methods only consider soft constraint that enforces constraint violations in expectation, which lacks strict safety assurance. Some studies (Choi et al., 2020; Yang et al., 2023b; Wabersich et al., 2023) utilize safety certificates from control theory, such as control barrier function (CBF) (Ames et al., 2019; Lee et al., 2023; Kim et al., 2023) and HJ reachability (Bansal et al., 2017; Fisac et al., 2019; Yu et al., 2022a), to ensure state- wise zero violations (hard constraint), but only applicable to online RI.
6 CONCLUSION AND FUTURE WORK
We propose FISOR, which enables safe offline policy learning with hard safety constraints. We introduce an offline version of the optimal feasible value function in HJ reachability to characterize the feasibility of states. This leads to a feasibllit-ddeeendett objective which can be further solved using three simple decoupled learning objectives. Experiments show that FISOR can achieve su- perior performance and stability while guaranteeing safety requirements. It can also be extended to safe offline IL problems. One caveat is that limited offline data size could hurt the algorithm's performance. Therefore, extending FISOR to a safe affline-to-onlin RL framework can be a viable future direction improve performance with online interactions, while retaining safety.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by National Key Research and Development Program of China under Grant (202YFBB2502904), and funding from Al.mo.AI.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference machine learning, pp. 22-31. PMLR, 2017.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. 2019 18th European control conference (ECC), 1420-3431. IEEE, 2019.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learming Proceedings 1995, pp. 30-37. Elsevier, 1995.
Andrea Bajesy, Somil Bansal, Eli Bronstein, Varun Tolani, and Claire J Tomlin. An efficient reachability- framework for provably safe autonomous navigation in unknown environ- ments. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 1758-1765. IEEE, 2019.
Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlln. Hamilton-jacohi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pp. 2242-2253. IEEE, 2017.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milhe et al. Jax: composable transformations of python+ numpy programs. 2018.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference Learning Representations, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang. Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficiens offline rl. In Thirty-seventth Conference on Neural Information Processing Systems, 2023.
Jason Choi, Fernando Castañeda, Claire Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control Iyapunov functions and control barrier functions. In Robotics: Science and Systems (RSS), 2020.
Jason J Choi, Donggun Lee, Koushil Sreenath, Claire J Tomlin, and Sylvia Herbert. Robust control barrier-value functions for safety-critical control. In 2021 60th IEFE Conference on Decision and Control (CDC), pp. 6814-6821. IEEE, 1021.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6000-1100. 2017.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv: 101.07,.01.08777 2018.
10

---

Published as a conference paper at ICLR 2024
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient
primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 33:8378- 88980, 2020.
Jaime Fisac, Neil Lugovoy, Vicenç Robi, Shromona Ghosh, and Claire Tomlin. Bridg- ing hamilton-jaee jacooii safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8550 8556. IEEE, 2019.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20155, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023.
Sven Gronauer. Bullet-safety-gyn gym: A framework for constrained reinforcement learning. 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.1033:
2022.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning arXiv:2304.10577 10573, 2023.
actor-critic method with diffusion policies. arXiv preprint
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising difffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Michael Janner, Yilun Du, Joshua Tenenbaum, Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, pp. 9902-9915. PMLR, 2022.
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09334, 2023.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference Machine Learning, pp. 267-274, 2002.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. arXiv preprint arXiv:2305.20082 2023.
Jeeseop Kim, Jaemin Lee, and Aaron D Ames. Safety-critical coordination of legged robots via layered controllers and forward reachable set based control barrier functions. arXiv preprint arXiv:23/2.08863 2023.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab. Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. JEEE Transactions Intelligent Transportation Systems, (3(6):490944922 2021.

---

Published as a conference paper at ICLR 2024
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774--837.. PMLR, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1199 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Jaemin Lee, Jeeseop Kim, and Aaron Ames. A data-driven method for safety-critical control: Designing control barrier functions from state constraints. arXiv preprint arXiv:2312.07786, 2023.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMLR, 2021.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305. 15669, 2023a.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023b.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive:
Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions pattern analysis and machine intelligence, 4563):3461-3455, 2022.
Shengbo Eben Li. Reinforcement learning for sequential decision and optimal control. Springer, 2023.
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning, 2023.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wen- hao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.0m 093033 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Con- ference on Machine Learning, 2023b.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact macrgy-guided diffusion sampling in offline reinforeement learning. In Inter- national Conference Machine Learning. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2024
Jiafei Lyu. Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Haitong Ma, Yang Guan, Shegnbo Eben Li, Xiangteng Zhang, Sifa Zheng, and Jianyu Chen. Feasi- ble actor-critic: Constrained reinforcement learning for ensuring statewise safety, arXiv preprint arXiv:2105. 10682, 2021a.
Haitong Ma. Changliu Liu, Shengbo Eben Li, Sifa Zheng, and Jianyu Chen. Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning. In Learning for Dynamics and Control Conference, pp. 97-109. PMLR, 2022.
Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:2110.09796. 2021b.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Revealing the mystery of distribution correction estimation via aaaooooonl-grrdiieen update. In The Twelfth International Conference Learning Reprexentations, 2023.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et When trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning, Advances in Neural Information Processing Systems, 35:36599-36612, 2022.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Alexander Robey, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V Dimarogonas, Stephen Tu, and Nikolai Matni. Learning control barrier functions from expert demonstrations. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3717-3724. IEEE, 2020.
Divya Saxena and Jiannong Cao. Generative adversarial networks (gans) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), 54(3):1-42, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmenta- tion. In International Conference on Machine Learning, pp. 20423-20444 PMLR, 2022.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2018
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEFE Robotics and Automation Letters, (3):4915-4922, 2021.
Kim P Wabersich, Andrew J Taylor, Jason J Choi, Koushil Sreenath, Claire Tomlin, Aaron D Ames, and Melanie N Zeilinger. Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 4375):137-177, 2023.
13

---

Published as a conference paper at ICLR 2024
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision pro- cesses. In Interaatoonal Conference on Machine Learning, pp. 9797-9806 PMLR, 2020.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023a.
Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang. Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference Machine Learning, pp. 36593-36604. PMLR, 2023b.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023c.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample soft- max for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085- 4098, 2022a.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu, Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8753-8760, 8760, 2022b.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang. Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regulariza- tion. In The Eleventh International Conference on Learning Representations, 2023.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 15:23851-23888 2022.
Tsung-Yen Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations, 2020.
Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe rein- forcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 13): 2023a.
Yujie Yang. Zhilong Zheng, and Shengbo Eben Li. 008-0585408885, 2023b.
Feasible policy iteration. arXiv preprint
Dongjie Yu, Haitong Ma, Shengho Li, and Jianyu Chen. Reachability constrained reinforcement learning. International Conference on Machine Learning, pp. 25636-25653 PMLR, 2022a.
Dongjie Yu, Wenjun Zou, Yujie Yang. Haitong Ma, Shengbo Eben Li, Jingliang Duan, and Jianyu Chen. Safe model-based reinforcement learning with an uncertainty-awarr reachability certificate. arXiv preprint 1rX0v22200.07553, 2022b.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning- In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4680-4688, 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. 2020.
Advances in Neural Information Processing Systems,
Weiye Zhao, Tairan He, and Changliu Liu. Model-free safe control for zero-violation reinforcement learning. In 5th Annual Conference on Robot Learning, 2021.
Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: A survey. arXiv preprint 022 2023.
14

---

Published as a conference paper at ICLR 2024
A
EXTENDED DISCUSSIONS ON RELATED WORKS
A.1 OFFLINE RL
To address the issue of distributional shift when learning RL policies solely from offline datasets, straightforward approach is include policy constraints that enforce the learned policy to remain close to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019; Fujimoto & Gu, 2021; Li et al., 2023a) or generalizable regions in data (Li ct al., 2023b; Cheng et al., 2023). Alternatively, value regularization methods do not impose direct policy constraints; instead, they penalize the value OOD actions to mitigate distributional shift (Kumar et al., 2020; Kostrikov et al., 2021; Niu al., 2022; Yang et al., 2022; Lyu al., 2022). These methods require jointly training actor and critic networks, which can be unstable practice. Recently, in-sample learning methods (Kostrikov al., 2022; Xu et al., 2023; Wang al., 2023a; Garg et al., 2023; Xu et al., 2022a; Ma et al., 2021b) exclusively leverage state-action pairs from the dataset for value and policy learning with decoupled losses, greatly enhancing stability. In addition, some recent studies employ expressive diffusion models as policies to capture complex distributions (Janner et al., 2022; Lu et al., 2023; Hansen- -atsen-Estruch et al., 2023), also achieving impressive performances.
A.2 ADDITIONAL DISCUSSION ON HARD CONSTRAINT AND SOFT CONSTRAINT
Based on different safety requirements, we divide existing works into two categories: soft constraint and hard constraint.
Hard Constraint. The hard constraint requires the policy to achieve state-wise zero constraint violation:
(sx)) 50,a~m,Wenn
can also be represented by cost function:
c(s:) = 00 w,
Soft Constraint. The most commonly used soft constraint restricts the expected cumulative costs below the cost limit
E
S1, E1, - c(())) 11
There are bunch of works focus on the state-wise constraint (Ma et al., 2021a; Zhao et al., 2023):
c(st) S w, a म. Vt N.
Since this type of problem still allows for a certain degree of constraint violation, we classify it as a soft constraint.
Discussions on Related Works (Hard Constraint). There a group of safe online RL works con- sidering hard constraints that have inspired Wang et al. (2023b) proposed method for learning safe policy with joint soft barrier function learning, generative modeling, and policy optimization. However, the intertwined learning process is not suitable for offline scenarios. SNO-MDP (Wachi & Sui, 2020) primarily focuses on the expansion of safe regions in online exploration. Additionally, the Gaussian process modeling approach may lead to considerable computational expense. In contrast, FISOR can learn the feasible region directly from the dataset without policy evaluation. ISSA (Zhao al., 2021) can achieve zero constraint violations, but its need for task-specific prior knowledge to design safety constraints makes it unsuitable for general cases. RCRI (Yu et al., 2022a) and RE- SPO (Ganai et al., 2024) employ reachability theory in an online setting, but their use of Lagrangian iterative solving methods hinders algorithm stability. FISOR achieves stable training across multiple tasks through its decoupled optimization objectives.
Algorithm Classification In order to better distinguish the safety constraints used by existing meth- ods and consider their training methods (offline or online), we summarize them as shown in Table 4. Many safe-control works pretrain safety certificates using offline methods (Robey et al., 2020; Zhao et al., 2021), but use them for online control without an offline trained policy. These methods are not considered safe offline RI. algorithms.
15

---

Published as a conference paper at ICLR 2024
Table 4: Detailed comparisons with related safe RL methods.
Online RL
Offline RL
CPO (Achiam et al., 2017)
CBPL (Le et al., 2019)
RCRL (Chow et al., 2017)
CPQ (Xu et al., 2022b)
RCPO (Tessler et al., 2018)
COptiDICE (Lee et al., 2022)
Soft Constraint
FOCOPS (Zhang et al., 2020)
CDT (Liu et al., 2023b)
PCPO (Yang et al., 2020)
TREBI (Lin et al., 2023)
Sauté (Sootla et al., 2022)
RI-CBF-CLF-OP (Choi al., 2021)
ISSA (Zhao et al., 2021)
FAC-SIS (Ma al., 2022)
RCRL (Yu al., 2022a)
FISOR (ours)
Hard Constraint
SRLNBC (Yang et al., 2023a)
SNO-MDP (Wachi & Sui, 2020)
Wang et al. (2023b)
RESPO (Ganai et al., 2024)
B THEORETICAL ANALYSIS
We first analyze the relationship between the feasible-dependent optimization problem in Eq. (4) and the commonly used safe RL optimization objective Eq. (1). And we provide the proof of key Theorems.
B. FEASIBILITY- DEELLIDDT OPTIMIZATION
This section analyzes the relationship between feasibiltt-depennent optimization and commonly used safe RL optimization objectives from safety perspective. Therefore, overlook the KL divergence constraint. The widely used safe RL optimization objective can be written as:
max Eg [V (s)
s.t. E. [V (s)) 1.
(17)
When consider the hard constraint situation, we set the cost limit to zero and drop the expectation in the constraints:
max E, [V. s.t. V. (s) S..
(18)
First, we divide the state space into two parts: the feasible region and the infeasible region. In the feasible region, where states denoted as s Sy, can find policy that satisfies hard constraint in Eq. (18). Conversely, we cannot find policy that satisfies hard constraints in the infeasible region & Sy. Based on above notations, we introduce Lemma (Proposition 4.2 in Yu al. (2022a)): Lemma 2. Eq. (18) is equivalent to:
max E, V (8) ThE, - (8).
(19)
V < 0, Sy.
For practical implementation, V (s) < 0 is extremely hard to satisfy due to the approximation error of neural network as shown in the toy case in Figure 1. So hope to replace V (s) < with new safe policy set (s) that is conductive for practical solving. After adding the KL divergence constraint back, got the feasibility-dependen optimization objective shown in Eq. (4).
16

---

Published as a conference paper at ICLR 2024
B.2 PROOF OF LEMMA
Proof. Due the involvement of two different optimal policies, for maximizing rewards (max, Ex [Vr((s) (+)) and the other for minimizing constraint violations (max; -V," (s)), in the proof process, in order to avoid confusion, we will use x, and ny represent them respectively:
म arg E, [V. (s))
(20)
k arg max (s),
Next, will proceed with the proofs of the three subproblems separately.
Optimization Objective in the Feasible Region. The expected return of policy x in terms of the advantage over another policy म can be expressed (Schulman et al., 2015; Kakade & Langford, 2002):
(T) = mf+) + Egndd (s) BaNa (s,
(21)
where (TT)) = E, (s) and d'(ss) (s) is the discounted visitation frequencies. Considering the case where it is the optimal policy म. Eq. (21) can be rewritten as:
(( = ((क)) + Esndy Es Buw [s)
(22)
According to Eq. (22), we can deduce:
max Ens (A, a)l
$ max E Re (8) Baan (A, a)]
$ max n(T,)) + Essar (4) Baur (A, a)]
1 max VTTT)
E max E. (7)
Transformation (i) holds since the optimal policy max, Eava [A](s, a)] guarantees the maxi- mization of expected advantage in any state and thus for the states from the state visitation distribu- tion d".. It does not affect the optimality of taking the expected value over states. Transformation (ii) is achieved by adding n,m"), which is unrelated the optimization variable Transformation (iii) holds by using Eq. (22). Transformation (iv) holds by using the definition of 17()).
Optimization Objective in the Infeasible Region.
max Ensa |-A
4 min Eavx (s, a)]
- min Bawx
max h(ss),, = 8,00 0,441 m 8442)) BEN
jii max (s)
max E, (s)
Transformation (i) is achieved by using the definition of (s, = (2,s, - (s), where (s) is unrelated to the action. Transformation (ii) is achieved by using the Defi- (s, a) is that in the first step, action is taken, and subse-
nition 1. The definition quently, the safest policy is followed. Transformation (iii): According to the definition of ni. Eg-x [max,e h(st), = s, ag = a, 0.41 ~ (- 8t41)] is minimized and only if π is the opti- mal policy ni.
17

---

Published as a conference paper at ICLR 2024
Safety Constraint in the Feasible Region.
Jusertennsso (-Is)da== =1 =1
(()(), 5
(1 50
Transformation (i): Jta Q7, (7,(0))) r-|ssdda means that all possible output action a of the policy need to satisfy Qi, a) S O. Specifically, for action a, if (als) greater than O, then Q" must be less than equal to O. For actions with probability density of O,
longer require specific value for Q mcaning that (T ( a (a)())))))))))))))))))){))))({ < O. Transformation (ii) can be derived by mathematical induction. In details, ()) I implies that for the transition a) 8', V*(s') I holds according to Definition 1. This means the next state is also a feasible state. Then, for this feasible s', we can also obtain feasible state (s', a') constraining Q (s', a') (a'(s')) I O. Then, we can derive that all states visited by starting from feasible region still locate in feasible region, so inductions.
s" with V*(s") I by
S Vg (s) S by mathematical
B.3 PROOF OF THEOREM 1
Proof. We first consider the situation when states within feasible region, i.c., VA 0. The analytic solution for the constrained optimization problem can be obtained by enforcing the KKT conditions. Then construct the Lagrange function as:
[1 Å1, p1) Earn (4)(s, A2 (Aro x(-ss)da
-1) 1) 1/ (DxL(F|(TTB) c)
Differentiating with respect to π gives:
θL (, a) - ii log logss(a(s) + log=(als) + μn -
where λς =}1- (s.0))) Setting
ah1 to zero and solving for gives the closed-form solution: On
1
nt (a|s)
(a(s))) exp (an ()),aa) (T
Z
where a1 1/p, Z1 normalizing constant to make that x is valid distribution.
For states the infeasible region, i.e., V₁ > 0, we can use the same method to derivation the closed-form solution. The Lagrange function of the infeasible part is:
L2( X2, p2) Ea-s |- (s, a)] A2 (((())daa
(T((TT E)
Setting
aLe to zero and solving for gives the closed form solution: dx
(alss)
Zz no exp)))) ()
where a2 = 1/p2. the effect of Zg is equivalent to that of Z
C GUIDED DIFFUSION POLICY LEARNING WITHOUT CLASSIFIER
We first briefly review diffusion models, then present the inherent connections between weighted regression and exact energy guided diffusion sampling (Lu et al., 2023).
18

---

Published as a conference paper at ICLR 2024
C.I A RECAP ON DIFFUSION MODEL
Diffusion models (Ho et al., 2020; Song et al., 2021) are powerful generative models that show ex- pressive ability modeling complex distributions. Following the most notations in (Lu al., 2023), given an dataset D = { D with D samples Bo from an unknown data distribution (2)))), dif- fusion models aim find an approximation of qo (2o) and then generate samples from this approx- imated distribution.
Diffusion models consist of two processes: forward noising process and reverse denoising process. During the forward nosing process, diffusion models gradually add Gaussian noise to the original data samples Zy from time stamp 0 to T > O, resulting in the noisy samples ær. The forward noising process from 2p to 2т satisfies the following transition distribution (r (2o)) E [0, 7]:
(2 (20)) (x, (,, I) E [0,7],
(23)
where are, Jy > 0 are fixed noise schedules, which are numan-designe to meet the requirements that qr (IT (z)) = qr (TT) ((xT 00, x² I) for some ā that independent of æg. such as Variance Preserving (VP) (Song et al., 2021: Ho et al., 2020) or Prrrance-Explodingg (VE) schedules (Song & Ermon, 2019; Song et al., 2021). Given the forward noising process, one can start from the marginal distribution at @r (xT)), i.c. æT ~ (xTT ō²1) and then reverse the forward process to recover the original data æy ru go (20). This reverse denoising process can be equivalently solved by solving a diffusion ODE or SDE (Song et al., 2021):
(Diffusion ODE) dr.- = f(t)a= 1 (t) v. log a (mx)) dt
(24)
2 [f(t)==- 922 log (x;)) (2x) dt + g(t)dw
(Diffusion SDE) dæ.-
(25)
where ft) dloggs (t) dr 2 2d log of are determined by the fixed noise schedules a,, ot. di
do?
in is standard Wiener process when time flows backwards from 7 to O. The only unknown element in Eq. (24-25) is the score function Va,, log ge (2))), which can be simply approximated by a neural network 20(2;; t) via minimizing MSE loss (Ho et al., 2020; Song al., 2021):
min t=2(00 [|] (0 t) oe V 2, log q (x))
(26)
+ min E. 1- (071)) [|| - 2,( [
0
where zo (mo). a. = ag + o. Given unlimited model capacity and data, the esitimated satisfies 20(x;,t) t) & or V Va, log qe (x;). Then, zy can be substituted into Eq. (24-25) and solves the ODEs/SDES to sample wo qu (Zo).
C.2 CONNECTIONS BETWEEN WEIGHTED REGRESSION AND EXACT ENERGY GUIDANCE
In our paper, we aim to extract the optimal policy ग' in Eq. (12), which behaves as a weighted distribution form as:
म" (als) & TT(a)))w(ss,))
(27)
where T. is the behavior policy that generates the offline dataset D and w(s a) is a feasibility- dependent weight function that assigns high values safe and high-reward (s, a) pairs. This can be abstracted as a standard energy guided sampling problem in diffusion models (Lu et al., 2023):
(w( x (x (E(o))
(28)
where po(x)) is the interested distribution which we want to sample from, (o(x)) is parameter- ized by pretrained diffusion model trained by Eq. (26), E(xo) is any form of energy function that encodes human preferences (c.g. cumulative rewards, cost value functions), f(x) 2 can be any non-negative function. To sample Ey Po(x))) with the pretrained (o())) and the energy func- tion E, previous works typically train a separate time-dependent classifier & and then calculating a modified score function (Lu et al., 2023; Janner et al., 2022; Lin et al., 2023):
V an lar.) = Vn, log (xx)
(29)
19

---

Published as a conference paper at ICLR 2024
where f*(x) is determined by the choice of fx)). Then, sample Zo -o(2)) by solving modified ODEs/SDEs via replacing Væ, log (x())) with Va. log Pr (xi) in Eq. (24-25). This kind of guided sampling method requires the training of additional time-dependent classifier &,, which is typically hard to train (Lu et al., 2023) and introduces additional training errors.
Weighted Regression the energy function into Eq. (26) and forming a weighted regression loss in Eq. (30), we can obtain the Vx, log Pr (ax+) in Eq. (29) without the training of any time-dependent classifier, which greatly simplifies the training difficulty:
Exact Energy Guidance. In this paper, find that by simply augmenting
min E (11.), [f(E(=)))lz (f 20(x,,t)12
(30)
Theorem 3. We can sample zu po(xu) by optimizing the weighted regression loss in Eq. (30)
and substituting the obtained zg into the diffusion ODEs/SDEs in Eq. (24-25).
Proof. We denote q: (21) and Pa (xx)) the marginal distribution of the forward noising process at time L starting from (aro) and po (2ro, respectively. We let the forward noising processes of Loo (z, (o)) and Pio (rr (r))) share the same transition distribution, that
4ro (x) (=)(x)/20)) (N (ar0,, (o0.0,oo((1)) of [0,2].
(31)
Then, we have
pr( = poolamlooo)oooo0o))la
(32)
= g g0 (0(())))))))))))))))))) (E((r))) dxy
Z
where the first equation holds for the definition of marginal distribution, the second equation holds for the definition of Po(2)) x 40 (2))) (E ((x))) and Z = ((())) is a normalization constant make Po valid distribution.
Then, taking the derivation w.r.t æz on the logarithm of each side of Eq. (32), we have
Va, (a..)( = V m, (
/ V æ, 9t0 ((
Qoo (x, (
(33)
V a, 91o ( Do )oo (mr).
pr (x,)
f (xx (xu) (xv) (Ex)) V Va, log qeo (x)/xx))dz
Pe (x,))
where the first equation holds since Z is constant w.r.t x; and thus can be dropped. To approximate the score function V., log p. (z,), we can resort to denoising score matching (Song & Ermon, 2019) via optimizing the following objective:
20

---

Published as a conference paper at ICLR 2024
min
(x()))
| Ft, t) + os vw, log (zx) [1]] dx;
0
A min
I p((2z)
z0 z0(0, t) + or (0000))200000(0000))
1o0 (x )
dx
p((2),
I mi(oo))
( 1)122 + 20, (2ze 1) S q:o (20 ((0))
log gro ( )
4 min
da
Pe (xx)))
i11 < min
ʃ 90 (0(20)) (1l + Van, log goo (m ( [20) dægdtdz
ss min Ex 8
(((())))) =1Ø]
(34)
where Te 0gz0 + o, Transformation (i) holds by substituting Eq. (33) into Væ, log (x Transformation (ii) holds by expanding the L2 norm according to its definition and drop the terms that are independent to 0. Transformation (iii) holds by rearranging (ii). Transformation (iv) holds since qeo (2+ mn) = Na (0 I). which is re-parameterized as 2; = 0gZo + 0z 2 ~ (O, f). Therefore, o, va, log 4:0 (20 (20
Eq. (34) is exactly the weighed regression loss in Eq. (30), meaning that the exact Po(2)) can be obtained by training this simple weighted regression loss and completes the proof.
Theorem 2 can be easily obtained via replacing Po with .², replacing qo with Tg and replacing f(E) with w. Some recent works also mention the weighted regression form (Hansen- -Hansen-Estruch et al., 2023; Kang al., 2023), but they do not identify these inherent connections between weighted regression and exact energy guidance. Based on Theorem 3, we show that we can extract n² using the weighted regression loss Eq. (14).
C.3 COMPARISONS WITH OTHER GUIDED SAMPLING METHOSS
Diffusion models are powerful generative models. The guided sampling methods in diffusion mod- els allow humans to embed their preferences to guide the diffusion models to generate desired out- puts (Lu et al., 2023; Ajay et al., 2023; Janner et al., 2022; Ho & Salimans, 2021; Dhariwal & Nichol, 2021), 1), such the policy that obtains the highest rewards and adheres to zero constraint violations.
Classifier-Free Guidance, One of the most popular guided sampling methods is the classifier- free guided sampling method (Ajay et al., 2023; Ho & Salimans, 2021), which trains the diffusion model with the condition information assistance to implicitly achieve classifier guidance. However, the conditioned variables may not be always easy to acquire. In contrast, it may be easy for us obtain a scalar function that encodes our preferences such as the cost value or reward value functions.
Classifier Guided Sampling. The other most popular guided sampling method is classifier guided sampling (Lu et al., 2023; Janner et al., 2022; Dhariwal & Nichol, 2021), which trains a separate time-dependent classifier and uses its gradient modify the score function during sampling like Eq. (29) does. This guided sampling method requires the training of an additional time-dependent classifier, which is typically hard to train and may suffer from large approximation errors (Lu al., 2023). In contrast, we smartly encode the classifier information into the diffusion model in the training stage through the weighted regression loss Eq, (30), which avoids the training of the complicated time-dependent classifier, greatly simplifying the training.
Direct Guided Sampling. Some diffusion-based offline RL methods directly use diffusion models to parameterize the policy and update diffusion models to directly maximize the Q value (Wang et al., 2023c). This requires the gradient backward through the entire reverse denoising process, inevitably introducing tremendous computational costs.
Sample-based Guided Sampling. Some diffusion- based offline RL methods directly use diffusion models to fit the behavior policy Ty and then sample a lot of action candidates from the approximated T, and then select the best one based on the classifier signal such as value functions (Hansen-Estrucd) Estruch et al., 2023; Chen et al., 2023). Although its simplicity, these methods show surprisingly good
21

---

Published as a conference paper at ICLR 2024
performances. Inspired by this, also generate some action candidates and select the safest one to enhance safety.
D EXPERIMENTAL DETAILS
This section outlines the experimental details to reproduce the main results in our papers.
D.1 DETAILS ON TOY CASE EXPERIMENTS
The toy case experiments this paper primarily focus on the reach-avoid control task (Yang et al., 2023a; Ma et al., 2021a), where the agent is required to navigate to the target location while avoiding hazards. The state space the agent can be represented as: S == (ær, y. U, O), where x and correspond to the agent's coordinates, represents velocity, and represents the yaw angle. The action space of the agent be represented as: A (é, 0). where corresponds the acceleration, 0 represents angular acceleration. The reward function is defined as the difference between the distance to the target position at the previous time step and the distance at the current time step. A larger change in distance implies higher velocity and results in higher reward. The constraint violation function h is defined as:
h = Raazard min {dhazardl dhazard2]
where Rhazard is the radius of the hazard. dnazardl, dhazard2 are the distances between the agent and two hazards. When h O, the agent collides with the hazard, otherwise the agent is in a safe state. Correspondingly, e is defined as : max(h, 0).
To obtain the ground truth largest feasible region, based on the current agent's state, we use the safest policy (maximum deceleration, maximum angular acceleration away from hazards). If no collision occurs, the current state is within the largest feasible region.
We use the converged DRPO (Yu et al., 2022b) algorithm to collect 50k of interactive data, and use a randomly initialized policy collect 50k of data, totaling 100k, for FISOR training. The data distribution is illustrated in Figure 1 (b).
In the toy case environment, we also conduct ablation experiments focusing on the selection of parameter 7 in Eq. (8) and visualized the feasible region in Figure 6. See from the results that with a small т value, the learned feasible region will become smaller and more conservative. This is acceptable since smaller feasible region will not likely induce false negative infeasible regions and beneficial to induce safe policies.
Offline learned feasible region under different t
V, (leamed))
Optimal feasible value funetion
0.0
0.8
1.66
2244
O Region with unsafe states
( Infensible region (ground truth)
Infensible region (leamed)
(a) 0.95
(b) 0.9
0.8
(c)
0.7
Figure 6: Visualization of feasible region under different T.
D.2 TASK DESCRIPTION
Gadtty-Gymmnsiim (Ray et al., 2019; Ji al., 2023). Environments based on the Mujoco physics
simulator. For the agent Car there three tasks: Button, Push, Goal. 1 and 2 are used to
22

---

Published as a conference paper at ICLR 2024
represent task difficulty. In these tasks, agents need to reach the goal while avoiding hazards. The tasks are named as (Task (Diffiiuttl Safety-Gymnasium also provides three velocity constraint tasks based on Ant, Half FaleCtae,a,, and Swimmer. Figure 7 visualizes the tasks in Safety-Gyymnsium.
(a) CarButtonl
(b) CarButton2
(c) CarPushl
(d) CarPush2
(e) CarGoall
(I) CarGoal2
AntVel
(d) HalfCheetahVel (d) SwimmerVel
Figure 7: Visualization of the Safety-Gymnasiun environments.
Bullet-Safety-Gynn (Gronauer, 2022). Environments based on the PyBullet physics simulator. There are four types of agents: Ball, Car, Drone, Ant, and two types of tasks: Circle, Run. The tasks are named as (Agent)
Figure (a) visualizes the tasks Bullet-Safety-Gynn
MetaDrive (Li et al., 2022). Environments based on the Panda3I) game engine simulate real-world driving scenarios. The tasks are named as (Road) (Vehicle). Road includes three different levels for self-driving cars: easy, medium, hard. Vehic includes four different levels of surrounding traffic: sparse, mean, dense. Figure (b) visualizes the tasks in MetaDrive.
(a) Sallet-SSfety--iimm
(b) MetaDrive
Figure 8: Visualization of the Bullet-Safety-Gye and MetaDrive environments.
D.3 ADDITIONAL EXPERIMENTAL DETAILS
Evaluation metrics. Our evaluation protocol strictly follows the recent safe offline RL benchmark DSRL (Liu et al., 2023a), we use the normalized reward return and the normalized cost returns as
https: http://wwww.f....eee//t www.saaffftt-aaaaaaa ctteci///////aattt//////////// https: ://gtthgbbbtt./// com/ wwwp//uuwxx///uuleeeeeee.eeeeeee Satett---eettyyaall https: http///gtttttt.t//t/ htt....cccm//iuuun/i///saaaaa
23

---

Published as a conference paper at ICLR 2024
the evaluation metrics: Anormalized Rx Tiiin C normalized Ca where Rx E, rr is the Finax, "Tmin" policy's 's reward return and Cg E, Ct its cost return. In all the benchmark tasks, the per-step cost G = 0 if the state is safe and Cr > 0 for unsafe states (Ray et al., 2019; Gronauer, 2022; Li al., 2022). Tmin and mmax are the minimum and maximum reward return in the offline data, respectively. I is suman-defined cost limit. c is positive number to ensure numerical stability if I = Following the DSRL benchmark, the task considered safe when C₂ does not exceed the cost limit /, in other words Cnormalised is no larger than 1.
Feasible Value Function Learning. According the loss function the feasible value functions in Eq. (9), we need to perform function learning using the constraint violation function h(s). However, the DSRL (Liu et al., 2023a) benchmark used in the experiment does not provide it, but only provides cost function c(s). Therefore, we simply employ sparse h(s) function design. Specifically, when c(s) = 0, the state safe, h(+) = -1; when e > O, the state unsafe, h = (M > 0). For optimal observe that it works best when the predicted mean of the feasible value function is around O. Based on this, we consistently chose M = 25 across all experiments. Note that determining the value of M doesn't need online evaluations for hyperparameter selection since we only need to monitor the mean value of Vi2 without any online interactions.
algorithm performance,
Network Architecture and Training Details. We implement the feasible value functions and value functions with 2-layer MLPs with ReLU activation functions and 256 hidden units for all networks. During the training, we set the for expectile regression in Eq. (8) and Eq. (15) 0.9. And use clipped double Q-learning (Fujimoto et al., 2018), taking minimum of two Q and maximum of two QN- We update the target network with a 0.001. Following Kostrikov al. (2022), we clip exponential advantages to (-o0, 100) in feasible part and ( (-o0, 150] in infeasible part. And we set the temperatures in Eq. (13) with or and 022 5. our paper, we use the diffusion model in IDQL (Hansen-Estee Estruch et al., 2023), which is implemented by JAXRI.
D.4 PSEUDOCODE AND COMPUTATIONAL COST
The pscudocode of FISOR is presented in Algorithm 1. We implement approach using the JAX framework (Bradbury et al., 2018). On a single RTX 3090 GPU, we can perform 1 million gradient steps in approximately 45 minutes for all tasks.
Algorithm Fuuuib Safe Offline RL (FISOR)
Initialize networks Qn. Vz. Qr. Ve, zg- Optimal feasible value function learning:
for each gradient step do
Update Vn, using Eq. (8) Update Qn using Eq. (9) end for
Optimal value function learning (IQL): for each gradient step do
Update V, using Eq. (15)
Update Q, using Eq. (16)
end for
Guided diffusion policy learning:
for each gradient step do
Update zy using Eq. (14)
end for
D.5 HYPERPARAMETERS
We use Adam optimizer with a learning rate -4 for all networks. The batch size is set to 256 for value networks and 2048 for the diffusion model. We report the detailed setup in Table 5.
https://gtt thub. com www.o//iksttrkoooo/ooooooo
24

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameters of FISOR
Parameter
Value
Optimizer
Adam
Learning rate
3e-4
Value function batch size
256
Diffusion model batch size
2048
Number hidden layers (value function)
2
Number neurons in hidden layer
256
Activation function
ReLU
Expectile
0.9
Feasible temperature @n
3
Infeasible temperature Q2
5
Discount factor y
0.99
Soft update
0.001
Exponential advantages clip (feasible)
(-o0, 100
Exponential advantages clip (infeasible)
(-o0, 150]
Number of times Gaussian noise is added T
5
Number of action candidates N
16
Training steps
le6
D.6 DETAILS ON SAFE OFFLINE IMITATION LEARNING
We can also extend FISOR to the domain of safe offline imitation learning (IL), showcasing the strong adaptability the algorithm.
Task Description. We test FISOR in three MetaDrive (Li et al., 2022) environments: mediumsparse mediumme ummean, medi umdense. Based on the dataset provided by DSRL (Liu et al., 2023a), select trajectories with high rewards that larger than 220 (expert dataset), in- cluding both safe and unsafe trajectories, as shown in Figure 9, The dataset does not include reward information but the cost function is retained.
maliandane
200
150
Kotum
150
In0
100
100
50
50
50
Original dataset
Expect dataset
Cost
Cost
Cost
Figure Visualization of expert dataset trajectories on the cost-return space.
Baselines. To apply FISOR in the safe offline IL setting, it suffices to modify Eq. (13) to:
V (s) S 0 (Feasible)
wr)(s,a) Ax
V (s) > 0 (Infeasible)
25

---

Published as a conference paper at ICLR 2024
In the IL experiment, adjust the batch size of the diffusion model to 1024, set the constraint Violation scale M to 1, and keep the rest consistent with Table 5.
BC-P adds fixed safety penalty top of BC (akin to TD3+BC (Fujimoto & Gu, 2021)), and its loss can be written as:
BC-P:
min L. (0) min B(e,a)DD (TTO()) a)² + k: Q (x.
0
0
where k is constant used to control the level of punishment and is set as A = 5.
Compared BC-P, BC-L replaces k with an adaptive PID-based Lagrangian multiplier (Stooke et al., 2020), and introduces a cost limit 1 = 10. Its loss can be written as:
BC-L: min L. = min (s,a)>> (TO (s) - a)² + A (s, -1
Each experiment result is averaged over 20 evaluation episodes and seeds.
D.7 EXPERIMENTS OF EXTENDING SAFE ONLINE RL TO OFFLINE SETTINGS
Besides the safe offline RL baselines, we adapted some safe online RL methods offline settings. Specifically, we consider Sauté (Sootla et al., 2022) and RCRL (Yu et al., 2022a), as Sauté has good versatility and RCRL also utilizes HJ Reachability enforce hard constraints.
However, directly applying safe online RL methods in offline settings faces the distributional shift challenge. Morcover, it is also difficult strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regulariza- tion offline settings. Considering these challenges, we added behavior regularization term like TD3+BC (Fujimoto & Gu, 2021) into Sauté and RCRL to combat the distributional shift. Also, we carefully tuned the conservatism strength to find a good balance that can obtain good results.
In particular, Sauté (Sootla et al., 2022) handles the safety constraints by integrating them into the state space and modifying the objective accordingly, which can easily applied to both online RI. methods. We incorporated this method into the state-of-the-an offline RL algorithm, TD3+BC (Fu- jimoto & Gu, 2021) (Sauté-TD:BC) The policy loss can be written as:
Sauté-TD3BC. min (+) = min E (a))DD (TO(S)) 1000-a33) -
where Q is the Sautéd action-value function.
RCRL (Yu et al., 2022a), based on reachability theory (hard constraint) and using Lagrangian it- erative solving, can be combined with TD3+BC to mitigate potential distributional shift issues in offline settings (RC-TD3BC), Its loss can be written as:
RC-TD33C min (+) = min B (TT(()) - a)² - (s, + (2())
8
We test the Sauté-TD3B3 and RC-TD3BC on DSRL datasets and carefully tune the hyper- parameters to find good results, the results presented in Table 6. However, these methods fail to obtain good results even with carefully swept hyper-parameters due to the intricate coupling objectives including reward maximization, safety satisfaction, and distributional shift. In detail, Sauté-TD3B) still permits some constraint violations, failing to effectively balance maximizing re- wards and meeting safety constraints. RC-TD3BC approach tends to make the algorithm overly conservative. Despite adopting the TD3+BC solving method, RC-TD3BC still faces distributional shift issues. Therefore, naively extending safe online RL algorithms directly to offline settings can lead to reduced algorithm performance and issues like distributional shifts.
hard constraints, but its coupled solving
D.8 DATA QUANTITY SENSITIVITY EXPERIMENTS.
We select some competitive baselines that achieve relatively good safety and reward performances in Table and train them with 1/2 and 1/10 of the data volume. Figure 10 shows that all basclines fail miserably under the small data setting that only contains 1/10 of the original data. FISOR,
26

---

Published as a conference paper at ICLR 2024
Table 6: Results of extending safe online RL offline settings. 1 means the higher the better. means the lower the better. Each value is averaged over 20 evaluation episodes and 3 seeds. Gray: Unsafe agents. Bold: Safe agents whose normalized cost is smaller than 1. Safe agent with the highest reward.
Sauté-TD3BC TD3BC
RC-TD3BC
FISOR (ours)
Task
reward
cost
reward
cost
reward t
cost
I
-0.03
3.48
-0.10
3.89
-0.02
0.26
CarButton2
-0.06
5.48
-0.12
2.30
0.01
0.58
CarPushl
0.19
0.97
0.24
1.81
0.28
0.28
CarPush2
0.05
3.03
0.16
3,48
0.14
0.89
CarGoall
0.34
1.56
0.44
3.84
0.49
0.83
CarGoal2
0.23
3.36
0.33
4.18
0.06
0.33
AntVel
0.80
1.12
-1.01
0.00
0.89
0.00
HalfCheetah Vel
0.95
11.87
-0.37
0.21
0.89
0.00
SwimmerVel
0.44
8.95
-0.03
1.09
-0.04
0.00
SafetyGym
0.32
4.42
-0.05
2.31
0.30
0.35
Average
AntRun
0.48
1.56
0.02
0.00
0.45
0.03
BallRun
0.24
3,43
-0.08
16.55
0.18
0.00
CarRun
0.87
1.50
-0.24
33.35
0.73
0.14
DroneRun
0.42
13.38
-0.19
9.8
0.30
0.55
AntCircle
0.22
17.57
0.00
0.00
0.20
0.00
BallCircle
0.61
1.13
0.10
29.18
0.34
0.00
CarCircle
0.61
3.81
0.01
45.11
0.40
0.11
DroneCircle
-0.11
1.22
-0,25
3.04
0.48
0.00
BulletGym
0.42
5,45
-0.08
17.13
0.39
0.10
Average
easysparse
0.01
6.19
-0.06
0.24
0.38
0.53
easymean
0.31
3.61
-0.06
0.20
0.38
0.25
easydense
-0.03
0.49
-0.05
0.12
0.36
0.25
mediumsparse
0.37
2.25
-0.08
0.33
0.42
0.22
mediummean
0.01
6.19
-0.07
0.10
0.39
0.08
mediumdense.
0.11
1.62
-0.07
0.20
0.49
0.44
hardsparse
0.26
4.37
-0.04
0.22
0.30
0.01
hardmean
0.11
2.00
-0.03
0.56
0.26
0.09
harddense
-0.02
0.42
-0.03
0.48
0.30
0.34
MetaDrive
0.13
3.02
-0.05
0.27
0.36
0.25
Average
however, still meets safety requirements and demonstrates more stable performance compared to other methods, although a reduction in data volume weakens FISOR's safety little. We believe FISOR enjoys such good stability as it decouples the intricate training processes of safe offline RL, which greatly enhances training performances.
E LIMITATIONS AND DISCUSSIONS
A limitation of FISOR is its requirement for more hyper-parameter tuning, such as the for- ward/reverse expectile and policy extraction temperatures. However, our experiments demonstrate that the algorithm's performance is not sensitive to hyper-parameter variations. Using only single set of hyper-pane pareemear,m,,ers, FISOR consistently outperforms baselines and obtains great safety/reward performances 26 tasks. Moreover, safety constraints with disturbances probabilistic constraint can challenging for FISOR. These constraints may affect network estimations and impact the algorithm's 's performance.
27

---

Published as a conference paper at ICLR 2024
Vell
CarCirele
hardsparse
Nomalized Retum
0.4
0.2
0.0
0.0
CDT
FISOR
TREBI
FISOR
FISOR
Nornalized Cost
2.
CDT
FISOR
TREBI
FISOR
CPQ
FISOR
Cost Limit
1.0
0.5
0.1
Figure 10: Data quantity sensitivity experiment results
While FISOR performs well on most tasks, still struggles achieve zero constraint violations in some cases. We attempt to analyze this issue and identify the following potential reasons:
We use Q7 to construct safety constraint, which theory can transform to V as stated in Ap- pendix B.2. However, in the offline setting, tion dD induced by the offline dataset rather than the whole state space. Therefore, the transfer
only conduct constraints the state distribu-
(*) from V. to Q7 requires another assumption on the data coverage that 0 < [ < C since Q% (s, a)dd(s))))))))) (x) = Q7 a)ddd)))((((())))))))) (s)=(als) only when the offline data has a good coverage on the policy distribution d= Strictly enforcing the policy distribution d staying within the support of offline data distribution, however, remains a longstanding challenge for offline RL (Lee et al., 2021; Levine al., 2020). One can use V enforce safety constraints. However, training the VØ is coupled with its policy training, which hinders algorithm stability as discussed before.
the offline setting, it is difficult to obtain the true value of V22 and V." Instead, we only have access near-optimal value function within the data distribution. Learning the optimal solution with limited data also remains a challenging problem (Kostrikov et al., 2022; Xu et al., 2023; Garg al., 2023; Xiao al., 2023).
Despite the inherent challenges of offline policy learning, to the best of our knowledge, our work represents a pioncering effort considering hard constraints within the offline setting. Moreover, we disentangle and separately address the complex objectives of reward maximization, safety constraint adherence, and behavior regularization for safe offline RI. This approach involves training these objectives independently, which enhances training stability and makes it particularly suitable for offline scenarios. our empirical evaluations, we demonstrate that, compared to soft constraints, our approach consistently achieves the lowest constraint violations without the need for selecting cost limit and attains the highest returns across most tasks, showcasing the promise of applying hard constraints and decoupled training in safe offline RL.
LEARNING CURVES
We test FISOR with the same set of parameters in Table 5 in 26 tasks of DSRL (Liu et al., 2023a), and the learning curve is shown in Figure 11.
28

---

Published as a conference paper at ICLR 2024
CarBuieel
CarRutten]
Carluntion2
Carluttinn)
Nemalloed Returs
Narualized Cost
7.5
0.00
5.0
Norall aed
0.1
+0.2
0.00
0:25
0.50
0.75
1.0
0.00
1025
0.50
0.75
0.00
1.00
Training Stape (le5)
Training Steps (leps (1eb)
Training Staps (166)
Training Steps (1eb)
CurPush!
CarPush!
CarPush?2
CarPiss2
Nomallba Cost
Nommelized lizd
Normllzze
600
0.00
Training Sieps (1o6)
Traming Steps (1e5)
Triiinn Steps (1e6)
Training (leb)
CarCkoll
CarCzell
CarCcel2
0,075
Namallzz Cost.
Nomaltzal
Nomalirod
0,000
Nommllied
0.2
0,25
8.00
0.00
0.30
6.00
0.25
1.00
0.00
0.55
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Steps (1e6)
Traniin Steps
Training
Training Staps (1:6)
AntVel
AntVel
Half' heetth
HalftCheetah
Nommaizzd Retum
Normallzed Cost
0.5
Nomaitrzd
055
Normalize
6,5
0.0
0.0
0.00
0.25
8,30
075
1.0
0.00
0.25
0.30
0.75
1.00
0.00
0.35
0.30
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Steps (len)
Training Steps (lo6)
Training Strpe (1e6)
Training Sieps (1eb)
Swimme/el
Swimmerk()
AntRun
Antuun
8:00
Normaliedd Cost
Narmaliral
Nomallizd
Nomms
02
0.25
1.00
0,50
0.60
0.35
0.30
6.75
1.00
0.00
01:55
0.30
0.73
1.00
Training Steps (leb)
Training Shepe (le6)
Training Strpe (leb)
Training Soeps (1e6)
BallRun
Ballitus
CaiRun
CurBunn
Nomallzed Ream
Nommalzed leed Cost
Nommilaed Cosl
0.0
20
0.00
0.25
0.30
0.15
120
0.00
[1.5
1150
8:75
1.00
0.00
0.23
[130
Li0
0.00
0.25
0.50
6.75
1.0
Training Stepe (106)
Tranning Stops (lef)
Toining Staps (lef)
Training Strps (1o6)
DeancRin
DroneRun
AmCCrck
AntC AntCircle
Nermaltze Retum
3.4
Normllzzd Con
10
Normalood Renmt
Normilizel Cos
0.2
0.5
010
0.30
1.00
0.00
0.00
0:25
0.50
0125
0.75
1.00
Training Steps (1e6)
Training Steps (1e6)
Training Steps (1e6)
Training Stops (136)
CarCirok
CarCincle
Necnalized Retum
04
Normsliee Coos.
Normdlzeed
0.5
02
0,01
0.0
Sleps
Steps
Trinnnn
Stops
DroseRun
Dronekun
easysparse
ensysparse
Nornalized Betum
Normalzzed Cost
Nocmllzze Cos
02
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
3.00
0.25
0.50
0.75
1.00
Training Steps (lef)
Training Steps (1o6)
Thinnngg Stepe (166)
Training Siqps (1ef)
29

---

Published as a conference paper at ICLR 2024
eHymiein
cunveesse
eisyderee
Nomzlized Reeurn
Normzalieed Cont
0.4
Normtatzad
0.2
0.0
0.0
0.00
0.25
8,50
170
0.0
633
0.30
0.75
1.0
0.00
0.33
0.30
6.1%
1.00
0.00
0.25
0.50
0.3
1.0
Training Steps (1e6)
Traming Steps (leb)
Training Stepp (leb)
Training Steps (1e6)
medfummspare
mediurmmemn
mediummenn
0.75
Noraalized Cost
0.4
Vommllged Cosl
0.50
Nortaliee
0.25
0.5
02
0.5
0.01
0.0
0.25
0.00
0125
0.50
100
0725
0.50
0.75
100
Training Steps (1e6)
Tramning Sleps (1e6)
Training Stepo (leb)
Training Steps (1c6)
masSumnkeee
meliunddene
harchgame:
hartlegoms
Normaloed Desure
0.75
Veraliize Cost
Nomulized Cost
0.50
0.2
8.5
0.20
0.1
0.00
0.0
011
Nomalizel Return
Nornalbad Cost
0.4
Normalize Cost
Vamaitral
0.0
Training Steps (1e6)
Tramnin Steps
Training Stape
Training Stops
Figure 11: The learning for the 26 tasks in benchmark (Liuu et al., 2023a).
30


Reviews:
[{"review_id": "SoT8fuiweg", "reviewer": "Reviewer_5fk7", "summary": "The paper introduces a method for safe offline policy learning with hard constraints, called FISOR. The method works through three dedicated learning objects that separate the feasible and infeasible region plus satisfaction of safety constraints. Policy learning is performed via a guided diffusion model and experiments show that FISOR is both effective int terms of high rewards and safety.", "questions": "Table 1: What is the criterion for an agent to be safe? Is it formulated as a threshold on the cost or do you count actual safety violations and only indicate those?\n\nWhat are other limitations? \n\nAre there safety constraints that cannot be handled or that are substantially more difficult?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Safety in RL is an important and relevant topic and this paper is a nice addition to the existing body of work. The approach to introduce safety via the three learning objectives is well motivated and substantially explained (although I did not follow all details). The usage of the diffusion model for policy learning is interesting. \n\nExtensive experiments including ablation studies are performed on well-known environments.\nWhile FISOR does not achieve the highest rewards in these environments, it consistently is safe and has lower cost than the baseline methods, except for CPQ in the metadrive environment.", "weaknesses": "The presentation of the results in Table 1 is a bit confusing. What is the criterion for an agent to be safe? Is it formulated as a threshold on the cost or do you count actual safety violations and only indicate those?\n\nI would like a more thorough discussions of limitations of the method. In the conclusion the limited availability of offline RL data is mentioned, but it is (if I didn't miss something) not well motivated or shown how this limitation affects the method or if it is more a general concern.\nWhat are other limitations? Are there safety constraints that cannot be handled or that are substantially more difficult?\n\nThe figures are a bit blurry on my reader, maybe they can be converted to a different format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8aQjR96y9n", "reviewer": "Reviewer_FU8i", "summary": "Safe offline reinforcement learning aims to develop safe policies without risky online interactions, but most existing methods risk potential safety violations by only enforcing soft constraints. This work presents a new approach, FISOR (FeasIbility-guided Safe Offline RL), utilizing reachability analysis from safe-control theory to enforce hard safety constraints, by maximizing rewards within a feasible region determined by the offline dataset and minimizing risks elsewhere. FISOR decouples the process into safety adherence and reward maximization. The result outperforms baselines in safety satisfaction and return performance in the DSRL benchmark for safe offline RL.", "questions": "- How do you guarantee the gap between the ground-truth infeasible regions and the learnt infeasible regions?\n- How do you measure the normalized cost in the evaluation? Why is it not zero for cases that are 100% safe?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of including reachability analysis into the learning for safety satification is interesting.\n- The paper is well-written.", "weaknesses": "- There seems to be an inconsistency between the goal and the actual approach. The paper aims for 100% guarantee in safety. However, the approach still need to estimate the feasible areas and then to yield the value function threshold. There could be estimation error and this estimation requires very thorough dataset provided, which may not be practical in reality.\n- The learnt infeasible states may be false negative. How do you assure that the infeasible states are really unsolvable instead of you able not able to solve it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46qs7esPo2", "reviewer": "Reviewer_3zVo", "summary": "This paper studies safe offline reinforcement learning by introducing feasibility into the RL problem for safety-critical systems. The authors convert the safe offline RL problem into a feasibility-dependent optimization problem, i.e., when feasible, maximize reward, when infeasible reduce the safety violation as much as possible.", "questions": "1. because this work uses learning to learn feasible regions by HJ reachability, how \"accurate\" the learned regions are? How does the learning region error affect the optimization problem and safety violations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is novel and the idea is interesting. It scales the HJ reachability to the high-dimensional RL problem and leverages it to offline compute feasible regions, which helps build feasibility-dependent RL problems to encode hard constraints. \n2. The paper is well-motivated, well-organized, and in general easy to follow. \n3. The paper is sound, at least from the writing logic, given the fact I didn't check the math details in the appendix. \n4. The experiments are comprehensive and the results are quite significant compared to the existing work.", "weaknesses": "1. The authors may need to add more references to cover \"safe RL with hard constraints\" to better motivate this work and claim the differences. A quick search gives me several recent papers, such as\na. Wang, Yixuan, et al. \"Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments.\" International Conference on Machine Learning. PMLR, 2023.\nAnd those in \nb. Zhao, Weiye, et al. \"State-wise safe reinforcement learning: A survey.\" arXiv preprint arXiv:2302.03122 (2023). \n\n2. It seems that the baselines in the experiments all consider soft safety constraints. The reviewer would like to see the comparison with baselines belonging to safe offline RL with hard constraints if there is such work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJhN8Sv5JM", "reviewer": "Reviewer_axZi", "summary": "This paper proposes an algorithm for safe offline RL problems. Unlike most of the existing studies that deals with expected form of safety constraints, this paper deals with hard safety constraint (i.e., probability-1 constraints). The authors discover that the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset, which leads to a feasibility-dependent objective which can be further solved using three simple decoupled learning objectives. The authors compare their proposed algorithm called FISOR against baselines on DSRL benchmark for safe offline RL.", "questions": "[Q1] I consider that there are several safe online RL algorithms with hard constraints. And some of them can be easily extended to offline RL settings. Could you give me some comments about my thoughts?\n\n[Q2] In general, the performance of an offline RL algorithm largely depends on the data coverage of the offline dataset. Could you explain the (empirical) relationship of the performance if FISOR and the data coverage? Is FISOR more efficient in terms of the data coverage compared to other baselines?\n\n[Q3] Why only CPQ satisfies the safety constraints within the baseline methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow. The problem settings are motivated and I think it is an important direction for safe RL research.\n- The proposed method is solid and looks sound to me.\n- The empirical evaluation are conducted in various benchmark tasks.", "weaknesses": "### Related Work\n\nHard constraints in safe RL have been studied in online RL settings. There are several important missing references as being represented by:\n\n- Sootla, Aivar, et al. \"Sauté rl: Almost surely safe reinforcement learning using state augmentation.\" International Conference on Machine Learning. PMLR, 2022.\n- Wang, Yixuan, et al. \"Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments.\" International Conference on Machine Learning. PMLR, 2023.\n- Wachi, Akifumi, and Yanan Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" International Conference on Machine Learning. PMLR, 2020.\n\nI agree with the authors that there is litte literature on safe **offline** RL and know that there is Table 4 in the Appendix. However, there are existing studies that are directly related to this paper. For example, Sootla et al. (2022) can be easily extended to the offline RL settings since all we have to do is to augment the state and provide a large penalty for violating the hard safety constraint.\n\n### Experiments\nWhile I would like to thank the authors for conducing good experiments, I am not fully convinced by the experimental results. First, all the baselines are based on the soft safety constraint and it seems unfair to simply compare their safety performances (Note that I do not say that such baseline methods should not be compared given there is few exising paper with the same problem settings). I personally consider that, if safe online RL algorithm can be extended to offline settings, such algorithms should be additionally included as a baseline. Second, I want the authors to discuss the experimental results on the reward performance. I understand sinec FISOR is enforced a hard constraint, it is natural that reward performance is deteriorated. However, under the condition that most of the baseline methods do not satisfy the safety constraint (because safety requirements are different by nature), I feel that it is rather arbitrary to compare the reward peformance **within** safe agents. Though the authors say that \" FISOR is the only method that can\nguarantee safety satisfaction in all tasks, while achieving top returns in most tasks\", I personally feel that this claim is overstated. I think the experimental results are interesting; hence, I would like the authors to discuss more on the **Main Results** paragraph for future readers (e.g., why CPQ satisfy the safety constraints, why FISOR does not perform well in Safety Gym tasks, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)