Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning
Authors: Zijian Guo
Weichao Zhou
Shengao Wang
Wenchao Li
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Safe Reinforcement Learning
Constraint-conditioned Actor-Critic
Data Generation
Out-of-distribution Detection
Zero-shot Adaptation
Primary area: reinforcement learning

Abstract: Offline safe reinforcement learning (OSRL) aims to learn policies with high rewards while satisfying safety constraints solely from data collected offline. However, the learned policies often struggle to handle states and actions that are not present or out-of-distribution (OOD) from the offline dataset, which can result in violation of the safety constraints or overly conservative behaviors during their online deployment. Moreover, many existing methods are unable to learn policies that can adapt to varying constraint thresholds. To address these challenges, we propose constraint-conditioned actor-critic (CCAC), a novel OSRL method that models the relationship between state-action distributions and safety constraints, and leverages this relationship to regularize critics and policy learning. CCAC learns policies that can effectively handle OOD data and adapt to varying constraint thresholds. Empirical evaluations on the $\texttt{DSRL}$ benchmarks show that CCAC significantly outperforms existing methods for learning adaptive, safe, and high-reward policies.

Published as a conference paper at ICLR 2025
CONSTRAINT-CONDTIINNED ACTOR-CRITIC FOR OFFLINE SAFE REINFORCEMENT LEARNING
Zijian Guo', Weichao Zhou², Shengao Wang', Wenchao Li!
Division of Systems Engineering, Boston University
2 Department of Electrical and Computer Engineering, Boston University {z jguo, zwc662, wsashawn, wenchao @ebu.duu
ABSTRACT
Offline safe reinforcement learning (OSRL) aims learn policies with high re- wards while satisfying safety constraints solely from data collected offline. How- ever, the learned policies often struggle to handle states and actions that are not present or out-of-distribution (OOD) from the offline dataset, which can result in violation of the safety constraints or overly conservative behaviors during their online deployment. Moreover, many existing methods are unable to learn poli- cies that can adapt to varying constraint thresholds. To address these challenges, we propose constraint-conditionedd actor-critic (CCAC), novel OSRL method that models the relationship between state-action distributions and safety con- straints, and leverages this relationship to regularize crities and policy learning. CCAC learns policies that can effectively handle OOD data and adapt to varying constraint thresholds. Empirical evaluations the DSRL benchmarks show that CCAC significantly outperforms existing methods for learning adaptive, safe, and high-reward policies. The code is available at https:/gittab.com/88DDEEEEP Lab/CCAC
INTRODUCTION
Offline safe reinforcement learning (OSRL) seeks to learn policies that maximize rewards while adhering to safety constraints solely based on offline datasets. OSRL brings together key aspects of safe RI. (uu et al. 2024) and offline RI. (Levine et al.) 2020 Prudencio et al. 2023). It adds
constraints ensure that the learned policy meets safety requirements (Lin et al. 2024 Cao al.,
2024
Guo et al.) 2024). while also learning from offline datasets to avoid expensive and potentially
hazardous online environment interactions [Bharadhwaj et al. 2020 Shi et al. 2021 I Kondrup et al.
2023
Zhou et al. 2023). However, it also inherits the challenges from these two areas.
major challenge in offline RL is distribution shifts (Levine et al., 2020): the behavior pol- icy trained offline can become unpredicatble when encountering states and actions unseen out- of-distribution (OOD) from the offline datasets during deployment. To mitigate this issuc, many existing offline reinforcement learning methods limit the policy's deviation from the (unknown) be- havior policy that generated the offline dataset or underestimate the critic's value. (Lyu et al.) [022] Kostrikov et [222]] Garg et al., [2023] Kumar et al. 2020). However, such approaches often result in overly conservative policies. Additionally, the theoretical foundation of many offline RL methods based the assumption that the offline dataset is collected with a single behavior policy, How- ever, in real-world tasks, dataset is more likely to be collected from multiple and diverse behavior policies. For example, the training data for self-driving vehicles are often collected by multiple human drivers whose policies are different from one another.
As for safe RI., key challenge is the lack of adaptability to varying constraint thresholds after train- ing (Liu et al.) 2022). Here, "varying" means that during deployment, trained agents are assigned a constraint threshold that may differ for each rollout or can change dynamically over time. This limitation stems from the common adoption of constrained optimization formulation (Xu et al.) 2022 Hong et al.) 2024 Guan et al. 2024, Zheng et al. 2023). which typically involves a single, fixed constraint threshold during training. However, in many real-world safety-critica applications, the ability to adapt to different or even unseen constraint thresholds essential. For example, a self-driving vehicle must adjust its behavior to meet varying safety requirements, such as altering

---

Published as a conference paper at ICLR 2025
its speed in response to different speed limits the road. While some existing methods tackle the problem of varying constraint thresholds using supervised learning (Liu et al. 2023b; Zhang et al. 2023b) or trajectory optimization (Lin et al. 2023), they are limited in their abilities to generalize to potential unseen constraint thresholds between training and deployment.
In this work, tackle these challenges simultaneously by learning safe, high-reward policy from offline datasets that can adapt to varying constraint thresholds while effectively managing distribu- tion shifts. Our key idea is to model the distribution of states and actions and uncover the relation- ships between behaviors and cost constraints from the offline dataset. We introduce a novel method called munstrain-coonitione actor-critic (CCAC), which combines tional autoencoder (CVAE) and a constraint-conditoon
punstrint-conddiionn classifier to learn this relationship. The CVAE facilitates data augmentation by generating new training samples that align with the learned behavior-constran relationship across different constraint thresholds. Meanwhile, the constraint- conditioned classifier identifies OOD data within the generated samples, allowing for a selectively conservative update of the critics and actor. This structure enables CCAC to train policy that can effectively handle OOD data and adapt to varying constraint thresholds during online deployment. Our main contributions are summarized as follows.
varia-
We propose a novel approach for learning adaptive, safe, and high- policies in OSRL. To our knowledge, this is the first value-based method that can achieve zero-shot adaptation to varying constraints in the OSRL setting.
We introduce novel constraint-condittoneed generative model and binary classifier for gen- erating state-action pairs and detecting OOD data, respectively. We demonstrate that these components can be used to effectively regularize the learning of both the crities and the actor
We conduct comprehensive experiments to show that (i) CCAC outperforms state-of of-the-art baselines both in safety and task performance by large margin, and (ii) CCAC can achieve high rewards while generalizing to varying constraint thresholds without re-training the policy.
2 RELATED WORK
Constraint satisfaction in RL. Many methods across different paradigms have been proposed to achieve constraint satisfaction. In offline safe RL, most of the works formulate the problem con- strained optimization problem, including primal-dual methods that optimize iteratively the primal and dual problem (Xu et al., 2022; Hong et al., (024) and stationary stationary-distriuuteee cesstributiinncorrrttiooo methods that train policies via importance sampling (Polosky et al. 2022 Lee et al. 22022). (Zheng al. 2023) breaks down the offline safe RL. problem into separate components and solves them individually. However, all of these methods require a fixed constraint threshold. Several works con- vert the offline safe RL problem into sequential modeling and use transformers to learn conditioned policies (Liu et al., 2023b; Zhang et al. (023b). Although they can generalize to different cost thresholds, they are sensitive to the rewards and costs they are conditioned on, e.g., combinations of high rewards and low costs that do not exist in the dataset can degrade the performance. (Lin) ctt al. (203)) solves the offline safe RL problem from a trajectory optimization perspective, but assumes single behavior policy and constrains its policy to stay close to the behavior policy. For online safe RI. (Zhang et al.) 2021 (Sootla et al., 2022) learns policies by maximizing the reward with constraint-related information included in the states. (Yao al. 2024b) also learns constraint- conditioned policies, but they consider constant thresholds. Moreover, it is nontrivial to tackle the issue of distribution shifts when switching from an online an offline setting.
Distribution shifts in offline (safe) RL. In the offline setting, a major challenge is to address the distribution shifts caused by OOD states and actions. Various methods have been proposed to pe- nalize the divergence between the learned policy and the behavior policy, using metrics such as Maximum Mean Discrepancy (MMD) (Kumar et al. 2019, Fisher divergence (Kostrikov et al. 2021) and KI. divergence (Nair et al., 2020 Wu et al. 2019). Generative models, e.g., VAES and GANs, are frequently used to generate actions based on states (Kumar et al. 2019 Chen et al.,
2022 Wu et al. 2022 Zhang et al. 2023a Liu et al. 2024) in order to regularize critics and policy learning and similar techniques are adopted when considering safety (Xu et al. 2022). However, the dependence of generating actions on the constraints is rarely explored. Although (aao et al. 2024a) consider distribution shift based on constraints, they still focus on a single constraint threshold.
While model-based RI. methods can leverage learned dynamics models for prediction and planning
2

---

Published as a conference paper at ICLR 2025
to address state distribution shifts (Kidambi al., 2021 Yu et al. 2020; Diehl al. 2022 Cho
jet al., 2024), the mitigation of OOD states effects is rarely explored in model-free methods. By conditioning on the constraint, are able to address the distribution shifts of both OOD states and actions in model-free manner.
3 PRELIMINARIES
CMDP and offline safe RL. We consider the Constrained Markov Decision Process (CMDP) model (Altman 1998) defined by the tuple (S,A, r.r.,rr..... c, where is the state space, A is the action space, P S x A x S + [0. 1] is the (unknown) transition function, r S x A 7 [- Rmany Rman] the reward function with Rman the maximum reward, e: A - [0, Cman! is the cost function with Cman the maximum cost, 7 is the discounted factor, and S [0, is the initial state distribution. Let : x A - [0, 1] denote the policy; T (5t, de Ct Ty rr)==1 denote
trajectory with maximum episode length T'; R((T) = Ei= 7'r, denote the reward return of the tra- jectory 5: and C(T) Σi= 7'0, denote the cost return. The goal is to find policy that maximizes the reward return while limiting the cost return threshold from fixed dataset D {Tr)
max E,~ [R(T)), s.t.
Enywa [C(T))
Actor-critic methods. The actor-critic methods (Haarnoja et al. 2018, Stooke et al. 2020) solve Eq. by maintaining two critics Qr, Qe to respectively approximate the the Q-value for the reward (5. = E, En [R(T)] and Q-value for the cost (E a) = (e<x<(C(T)) with 5 {s, a....... The critics Q. Qu and actor æ updated in alternating manner.
min E. anD (Q)((,)) a) - QT E (r.c) (policy evaluation)
4,
max DD [2, (s,a)) s.t. E a)] Se (policy improvement)
x
The critics are updated via policy evaluation, by iterating the Bellman operator: TTQ = 7P a), where f {r. c) and pa is the transition matrix associated with the policy: pTQ a) = P/aa a')]. The policy is improved by updating it towards actions that maximize the expected Qr valuee and whose Qe-values satisfy the constraint threshold.
4 CONSTRAINT-CONDITONN
ACTOR-CRITU
In this section, we introduce our proposed method as illustrated Figure D Section we argue that the offline dataset can be viewed collected by a behavior policy that chooses different actions depending on the constraint threshold. Building on this perspective, we focus on learning the dependence of the state-action pairs on the constraint thresholds within the offline dataset. In particular, we treat the state-action pairs that violate cost constraints as OOD. Section 4.2 we introduce the constraint-conditoone state-action distribution. In Sections 4.3 and 4.4 we provide details on how we apply the learned OOD distribution regularize the learning of constraint-condiioone critics and actor produce safe and high-reward policies.
VAE and constraint-conditioned classifier to model the OOD
(3.0, k(,,.)
Qe(a.alk)
Cost cirtic
Offline dataset
(s)
CVAR
Classifier
(8.0) a(s)
Actor
==(a|s, R)
(.,0,)
Q+(a,alk)
Reward critic
Figure I: CCAC overview. During training, we sample (s, a, s' r, k) from the dataset and use (s,0,k) x) to train the CVAE and Classifier, which aim learn a constraint-conddiioned distribution of OOD state-action pairs, v(s, a/xs). Specifically, we generate pairs (8, â) by sampling from the CVAE, and the pairs with a classifier score above 0.5 are considered OOD. Then these OOD pairs, along with the sampled (s, a, s', ,, R), are used to update the cost critic, Qe(s, a/k), while the sampled (s, a, s', r, ι) are used update the reward critic, (7 a/ss). Finally, the critics are employed to update the policy (a)ls,)).
3

---

Published as a conference paper at ICLR 2025
4.1 RETHINKING THE OFFLINE DATASET: A CONSTRAINT- CONDITTOONOD PERSPECTIVE
As mentioned in Section 2 many metrics have been proposed to measure the distribution shifts of states and actions in offline RL. However, those metries often overlook the correlation between the constraint thresholds and the state-action distribution. We argue that this correlation is a significant aspect of offline datasets that can be exploited not only for characterizing the distribution shift but also for learning adaptive policies. In our method, we define the varying constraint threshold at each time-step t as the cost budget, denoted as Ki-- Specifically, for each trajectory in the offline
dataset, we set the cumulative cost of that trajectory as its initial cost budget, i.e., K1 = Σi= t= Ct- The cost budget is then updated at each time-step based the incurred cost, i.c., Kt+1 Ky Ct. for t= 1,2,...... 1-1. 1. This formulation is similar to the notions of return-to-go (Chen et al. [2021] and cost-to-go (Liu al. 2023b). Compared with existing safe RL methods that a constant threshold, varying constraint thresholds provides better flexibility for the agent to adjust its behaviors, e.g., opting for more conservative actions as the cost budget decreases.
T
Considering the cost budget as the underlying cause of the behaviors in the dataset, the state-action distribution in the offline dataset D= - {T. with = {st, dt Cty Tts (e, Ke) can be factorized as d== "T (s) ()))) (a)s,)) k), where TD (apaass, K) is some
behavior policy, d (1:) and d= (s(rt) represent the distributions of encountered cost budgets and visited states by following TD. respectively. Given a cost budget ic we consider state-action pair (s,a) to be OOD taking the action a at the state 8 will lead to exceeding the cost budget Ki and otherwise (s, a) is considered to be in-distribution (IND). Note that the OOD issue when encountering states and actions unseen in the dataset- - is more nuanced in the offline safe RL setting due to the need to ensure safety in addition to policy performance. Our treatment of regarding state-action pairs that exceed the cost budget as OOD provides a means of incorporating safety constraints into the policy optimization process. This enables the learned policy to avoid unsafe regions of the state-action space even if they are within the dataset distribution, while also adapting to unseen and varying cost budgets. As we will show in Sectior44.2 and Section 4.3] this is done through novel data augmen- tation and filtering process by constraint-on constraint-condiiooee VAE and classifer, and an overestimation of the cost critic respectively. In Section 4.4 we offer further insights on how this overestimation can achieve a similar policy regularization effect as an underestimation of reward critic (Kumar et al., Lyu al. 2022) to mitigate the OOD issue in offline RL.
2020
4.2
DATA GENERATION AND CLASSIFICATION
We aim to learn constraint-conditoonn policy I [a]ss, k) that can satisfy the varying cost budget by favoring the IND alateacction pairs while avoiding the OOD ones. Given a dataset, we can readily classify the data OOD not based cost budget k. However, empirical results in Section 5.3 indicate that directly learning such a policy from the offline dataset is often limited by the size of the dataset. To overcome this, we propose learn the OOD state-action distribution, which we denote as v(s, aa [s)), and use it to augment the dataset. Specifically, we train a generative model to generate diverse state-action pairs and train a binary classifier to identify OOD samples within the generated data. For the generative model, we propose a constraint-conditoone variational autoencoder (CVAE) adapted from VAE (Kingma & Welling 2013] (ohn et al. 2015), which models the distribution by transforming an underlying latent manifold. The objective is to optimize the following evidence lower bound (ELBO):
max E. ( [Z [plss, (p(s, k)] 3Dx (a(x)s, (k)p((((kk)) P.g
where q(21s,, a, k) is the encoder that maps the tuple of state, action, and cost budget into the latent space, p(8, al22, x) is the decoder that reconstructs the vector in latent space back into state-action pair, p(=[<) is the prior distribution, typically set to (0, 7), and /3 a hyperparameter. The first term represents the reconstruction loss, and the second term is the KL-divergence between the en- coder output and the conditional prior of 2. Previous works (Fujimoto et al.) [2018] Kumar et al.) 2019 Xu et al. 2022) also use conditional VAEs, i.e., VAEs conditioned states, but they use them for sampling actions solely and do not consider constraints.
To generate state-action pairs, we randomly sample from the prior distribution p(2[k) and pass them through the decoder. However, to determine which generated state-action pairs are OOD, additional steps needed. We propose train a constraint-conditoons binary classifier h(s,alls)
4

---

Published as a conference paper at ICLR 2025
to distinguish IND and OOD state-action pairs using the binary-cross-entropy loss:
min B, [-y log (h(s, alk)) ((1 y) log h(s, [[s)
(4)
where y is the label of the tuple (s, a, k). We label a state-action pair sampled from datasets by checking its cost budget. For example, during each training iteration, we randomly sample batch of (s, a, k) and a cost budget R from D. We then assign labels: y = 0 (IND) if x S R and y = (OOD) otherwise. Additionally, we include reconstructed state-action pairs (š, â) obtained from the CVAE in the training of the classifier, assigning them the same labels as their original counterparts (s, a, k). This is based on two mild assumptions: (i) the reconstructed data should closely resemble the real data, which is the goal of CVAE, and (ii) taking similar actions in similar states leads to a similar number of constraint violations. The reconstructed data can be seen as small perturbations of the original data. Consequently, the batch data for classifier training is augmented to include both (s, a, R. y) and (š, û, R. y), enhancing the effectiveness of the training process. Considering h(s, alk) as the probability of (s, a, k) being OOD, the OOD state-action pair distribution v(ss, alk) can be proportional to Ezwp(=le) [p(s, a 2, (),
4.3 CONSTRAINT- CRITICS
cunstraint-oonditoone cost critic. To achieve safety, aim to overestimate the cost critic of OOD state-action pairs within bounds. Our choice is to learn a constraint-conditoons cost critic that encourages the expected Qe-values under particular distribution of state-action pairs, i.e., the distribution the OOD state-action pairs v(ss, a[k), to be no less than a specified threshold e while minimizing the Bellman error, which yields the following objective:
min E ED [[Qe(s, a|x) - T Qe(s, alx))²] s.t. Be.avn,==Dlee66 a|r)] 2
(5)
Qe
Intuitively, Qe-values of the state-action pairs that are well-supported by the data distribution T (s,, allx), will be pushed down to comply with the Bellman backup by the Bellman error term. Conversely, those OOD and unsafe state-action pairs' Qa-val will be pushed up. The overestima- tion can extend to IND state-action pairs that are near the boundary of IND and OOD, causing their Qe-values be higher than their true values. However, empirically, this does not show noticeable impact on performance as it tends to lead to a mildly conservative policy that avoids risky actions.
In order to solve 5). we use the primal-dual method (Chow et al.) 2018 Stooke et al.) 2020) by introducing a dual variable, i.e., the Lagrangian multiplier Xc. and transforming it into an uncon- strained optimization problem:
max min Es,,,<D [Qcc(s,)kk) T T" - B.(BB,.r..x-Dl..(s..l - c) (6) 2.20 Qu
Now, we show that we can make Qa-vel of the OOD state-action pairs greater than e with appro- priate A when updating the cost critic by 55: Theorem 4.1. For any v(s, a/k) with supp v C supp TD. VK E D. (8) E Dood. by iterating 5). the cost critic obtained Q alpc satisfes::
de
v(s, alk): [1I- d alk))
(3))))) = O7 (s. 2
where de is the weight, Dood is the set that contains all the OOD data generated by v(s, alk)) d= (8, alx) (8,a)k) = d (s)x)) is the marginal state-action distribution, and if we choose No Z max(2 maxx,a.. dd ( ( Qe(s, (l)) - ?P")). 0} then we can get Q): (s, 2c.
M(s.a1))
The proof can be found in Appendix A Note that I 1--7 7F is the inverse of the state occupancy matrix with non-negative entries and TD (8, alxs) > 0 since the behavior policies can be sub-optimal, e.g., possible to violate constraints. One can set the constraint threshold to be greater than the maximum value of н in the offline datasets to ensure that the -alues of OOD state-action pairs sufficiently overestimated.
constraint-oodiitoon reward critic. We learn a constraint reward critic by mini- mizing the Bellman error without any additional terms:
min E, Es alk) (=))))
5

---

Published as a conference paper at ICLR 2025
Although directly applying the Bellman operator is known to suffer from OOD actions, overestimat- ing the valuues can avoid this issue when learning the policy.
4.4
ACTOR
To maintain consistency with the reward and cost critics, train constraint-coondiioned actor by maximizing the reward while satisfying the constraints as follows.
max E Dx,, [s, a|*)] s.t. (.s.Derr((aa [Qc a/x)] S K. Vk E D
(8)
Without conditioning 8 Eq. simplifies to the standard policy improvement in E and the constraint regarding the minimum in will dominate the other constraints. Although our definition of OOD differs from the typical definition of OOD (unseen state-actions in the dataset), our approach can still effectively mitigate this issue via the overestimation of Qe (s, als) to achieve policy regularization. By overestimating the Qe(s, a|xx), we can naturally avoid high-cost actions as they will violate the constraint in Eq. (). Additionally, this overestimation induces policy regulariza- tion, which is common approach in offline RL to address the OOD issue by penalizing deviations of the learned policy from the behavior policy (Kumar et al.] 2020 Lyu et al., 2022 Kostrikov et al. 2021). For example, CQL (Kumar al.] (2200) applies a conservative (underestimation of) (s. to regularize the learned policy to stay close to the behavior policy. It is proved in Dual RL (Sikchi al.) 2023) that adding such a constraint on value functions is equivalent to adding f-divergence between the learned policy and behavior policy. Leveraging the dual relationship between (s, a and Qc(s.a/x)) (Paternai et al. (019)), our method achieves similar policy regularization effect by overestimating Q= x.). thereby keeping the learned policy close to the behavior policy. Thus, when learning the Q, a alms), our method does not need an additional ex- plicit regularization. Empirically, we validate this in Appendix D.4 replacing the reward function learning part in CCAC with an existing explicit regularization method.
Similar to solving Eq 5). we also use the primal-dual method to solve Eq, 8
min max ED, [Q,(s, A,20
Σ (x( (Qc(s, - -x) KED
where Aa (R) are the Lagrangian multipliers, which are parameterized by networks and tuned auto- matically. The overall method is summarized in Algorithm inn Appendix C.I The training of OOD generation and detection, and that of the RL components can be performed separately. We start by training the CVAE and classifier till convergence. Then, we fix them and proceed to train the critics and actor. Training all components simultaneously can cause instability, as inadequately trained OOD generation and detection module may produce incorrect OOD state-action pairs, ultimately leading to the failure of the critics and actor training. During evaluation, the initial cost budget is set arbitrarily for each rollout and updated at every time-step based on the costs incurred on the fly. A well-trained policy expected to adjust its behavior accordingly.
EXPERIMENTS
In this section, we evaluate our method in multiple tasks to answer the following questions:
Q1 Can CCAC learn safe and high-reward policies from offline datasets?
Q2 Can CCAC achieve zero-shot adaption to different cost budgets?
Q3 What is the importance of the OOD detection component in CCAC?
Tasks. The BellettSSaeety-Gy (Gronauer 2022 and Sa fety--ymnas um (J et al. 2023) are public benchmarks that include a variety of continuous robot locomotion control tasks commonly used in previous works (Chow et al. 2019 Zheng et al. [2023 Liu et al. 2023b) and DSRL (Liu et al.) 2023a), a comprehensive benchmark specialized for offline safe RL, provides the offline datasets. We consider three tasks: Run, Circle, and Velocity and multiple types of robots: Ant, rewards
Ball, Car, Drone, Hopper, and HalfCheetah. In the Run task, agents for achieving high speeds between two boundaries but face penalties if they cross the boundaries or exceed agent-specific velocity threshold. In the Ve Velocit by task, agents also receive rewards for

---

Published as a conference paper at ICLR 2025
Taks
Metric
BC-safe
CQL-Saute
BOQ-Lag
BEAR-Lag
CPQ
COPEDICE
VOCE
CDT
TREB1
FISOR
Ours
Reward ? 0.36ua32
1.95
0,3
2.07azas
0.89
1.76an24
2.1111
0.91.uu
0.78ua.5
0.76.aas
0.97.am
Ball-Run
Cost 1
2,08-3ss
3:26
0.35-1
16.23
2,00
11.59
12.74354
0.92-001
2.19
0.0.0
0.27:419
Reward
0,96
0.93:am4
0.84amm
1.0soe
0,97.m
0.95
0.98
1.0:a
0.97amss
0.73e215
0.95.aM
Car Run
Cost 4
1,06-3
0.67-121
2.634.8
6.474553
0.0-aa
0.0-na
9.41
0.91-mm
2.54
2.574385
0.19-435
Reward f. 0.56-333
0.2
1.09
0.04
0.35v03
0.82-009
0:11:a15
0.16-44
1.01-33
Ant- Circla
Coer g
3.545%
19.00
17.064964
15.55-11
0.0-aa
675-49
7.95-1144
0.0--0
0.55-1.4
Ball-Cirele
Reward
0.62
0.3
1.08
0,93
0.92
0.0:om
0.91:0m
0,73ec
0.26::a14
0.57:444
1.6741.5
0.02..18
8.95
11.22
2.25
9.14185
12.96a8.2
2.11at0
1.84119
0.0..0
0.0..0
Car-Circle
Reward f
0,38-a27
0.54
0.00
1.064208
0,00
0.59-a1
0.19.a1
0.88-006
0.52.415
0.25-aas
Coar 2
0.58-1.
2.84+x
11.74-2
15,044374
7.57
10.95
1.564344
1.59-190
1.124990
0.0--0
(L73-144
Drone-Circle Reward f. 0,82a116
0.15amd
1.38um
1.27a2es
0.83a8a
0.59umee
0.0una
0.94anxo
0.68aa14
0.62:112
0.52.A11
J 1,32.1.0
2.
19.07413
14.89
3.661444
2.62
0.0.aa
1.4400s
5.89
0.06:131
0.07.as4
Am-Velocity Reward ?
0,98
.0.09.030
0.93un3:
-1.6.aa
-0.3-a4
1005e00
-1.02aaa
1.02.au
0.59uass
0.78:aat
0.9.ass
0,53.au
1.0
15.9
0.0wm
0.9
11.09
0.0.na
0.59.033
0.9
0.0
0.58.a1s
HalfCheetthh Reward f
0.9aam
0.54a234
1.08.0s
1.06am
1.79
0.69
4.00.auu
L.lano
0.46.ass
0.86
0.96.au
Velucily
1 0,66au
1.28ac43
59.17423
29
8241874
97.81
0.0.na
0.0.na
L18.0s
0.05.a13
0.79
Hopper-
Reward
0:25
0,51a33
0.5
0.39
0.15
0.15ama
0,58
0.61at3
0.16
0.89:
Velacity
1
3,02
13.53
18.9
15.3
5,99
2,07.31
1.05.m
8,52
0.0
0.32.12s
Reward
0,662135
0.55
0.89
0.92044
0.66
0.78
0.32
0.9
0.62n
0.51:03
0.91:am1
Averago
1
16.72-14
14.4-100s
14.4-104
7.43
4.
1.84-95)
0.27-131
Table I: Evaluation results of the normalized reward and cost. The agents are trained using the whole dataset. The cost threshold is I (the cost budget is for Run and Ci Circle tasks and 10 for Velocity ty task). The percentages of safe trajectories in the datasets are as follows: 4.9%. 28.6%, 5.9%. 5.5%, 6.8%,9.1% %, 9.3%, 9.7%, 5.8% for the tasks listed from top to bottom. 1: the higher the reward, the better. J: the lower the cost (up 1), the better. Each value is averaged over 20 episodes and 3 seeds. Bold: Safe agents. Gray: Unsafe agents. Blue: Safe agent with the highest reward.
moving forward but incur costs if they exceed a specified velocity threshold. In the Circle task, agents are rewarded for moving gion smaller than the radius of the target circle. This setup of conflicting rewards and costs requires the agents to balance their trade-off (Liu et al.) 2022 Guo et al. 2024). which is challenging since pursuing high rewards can lead to constraint violations, while conservative policies yield safety but lower rewards. More details of the environments can be found in Appendix B
clockwise circular pattern but are constrained within a safe re-
Metrics. Our evaluation metrics include normalized cumulative reward and cost:
R₂ Rminn,x
Cx
Raommalized R Rmax,N Rmin,
Cnormalzzdd
where R2 and C₂ are the evaluated cumulative reward and cost of policy ग, respectively, and Rmax, and Rmin, are the maximum and minimum cumulative reward of the trajectories that sat- isfy the cost budget k in the offline dataset D. For convenience, we will abbreviate normalized cumulative reward as reward and cumulative cost as cost. A policy is safe if Civemalized S 1.
Baselines. We compare CCAC with the following baselines: 1) BC-safe: behavior cloning that imitates only safe trajectories in the datasets. 2) CQL-Sauté: a state-augment adapt from Sauté (Sootla al. 2022) and CQL (Kumar al., 2020). 3, 4) BCQ-Lagrangian BEARRLaag) Lagrangian-based methods built upon BCQ (Fu-
method that
(BCQ-Lagg and SEARR-Lagrangia jimoto et al. 2018) and BEAR (Kumar et al. 2019) respectively. 5) CPQ (Xu al. 2022 a Q- lcarning-based method that treats OOD actions as unsafe and learns policy from safe actions. 6) Cop- tIDICE (Lee al. 2022): ; stationary distribution correction based method. 7) VOCE Guan all 2024 a probabilistic-imfeee
based method that learns conservative critics. 8) CDT (Liu et al.
2023b); Decision- Transffrmmr based method that learns conditioned policies. 9) FISOR (Zheng et al. 2023): diffusion- method that learns feasibility-guidee policy. 10) TREBI (Lin et al.) method that uses trajectory optimization. See Appendix for implemen- tation details. For CQL-Sauté, CDT, TREBI, and our method, an initial cost budget is assigned for each rollout and then updated based on the incurred costs. For the rest of the basclines, we use the cost budget to refer to the constant cost threshold used for training and evaluation.
) a diffusion-basee
5.1 CAN CCAC LEARN SAFE AND HIGH-REWARD POLICIES FROM OFFLINE DATASETS?
The evaluation results for different trained policies are presented in Table D The training curves included in Appendix D.I Our method demonstrates the best performance compared to the
7

---

Published as a conference paper at ICLR 2025
baselines in terms of reaching the highest reward in most of the tasks while maintaining safery. We can also observe that high rewards usually come with high costs and vice versa, which highlights their trade-off. The baseline methods either suffer from significant constraint violations yield sub-optimal returns. The results of BC-safe and CQL-Sauté indicate that, due to distribution shifts, merely mimicking the safe data maximizing the reward critic of cost-augmented states cannot achieve safety. The Lagrangian- based baselines fail to behave safely on most tasks, which suggests that directly applying the primal-dual method in the offline setting can hardly work well without regularizing the OOD state-action pairs. Though CPQ considers the OOD actions during training, the OOD states at test time can still result in constraint violations. COptiDICE and VOCE oscillate between overly conservative and overly aggressive since the trajectories in the dataset are collected from diverse behavior policies, potentially leading to biased estimations of the crities. FISOR ex- hibits conservative behaviors as it considers hard constraint instead of soft constraint. TREBI strug- gles to maintain safety as it restricts its learned policy to remain close to the behavior policy, despite the behavior policy being suboptimal.
Distribution shifts. To assess the effect of OOD states and actions, we use different percentages of data to train policies and then evaluate their performance. Less training data makes it more chal- lenging to train the policies and implies a larger distribution shift during evaluation. The evaluation results shown in Figure Appendix Ba.udixxBBBaaae also includes visualization of the offline datasets. We can observe that most methods exhibit an increase in costs and a decrease in rewards as the percent- age of training data decreases. Our method. on the other hand, continues to satisfy the constraints and achieve high rewards, showcasing its robustness against distribution shifts.
BC-safe
CQL-Sauté
FISOR
CCAC(ours)
Cost threshold
Eval reward
1.0
Eval cost
p-2.25
Data percentage
Figure 2: Evaluation results of reward and cost in Run and Circle tasks with different percentages of datasets being used for training. The dashed line represents the normalized cost threshold.
CQL-Sauté
CDT
TREBI
CCAC(ours)
Eval cost ==
Cost budget
Ball-Run
Car-Run
Ant-Circle
Drone-Circle
HalfCheetah--el
Hopper-Vel
Eval cost Eval reward
3000
2000
Cost budget
Figure 3: Results of alignment with different cost budgets. The agents trained using the whole dataset. The top-row plots show the evaluated reward and the bottom-row plots show the evaluated cost. The solid line and the light shade area represent the mean and mean = standard deviation. The dashed line represents the scenario where the evaluated cost matches the cost budget.
5.2 CAN CCAC ACHIEVE ZERO SHOT ADAPTION TO DIFFERENT COST BUDGETS?
One major advantage of our method is the ability to adapt varying cost budgets without the need for re-trnnni To illustrate this, we set different cost budgets for evaluation rollouts to obtain the results in Figure 3 We compare
method with CQL-Sauté, CDT, and TREBI, since all other baselines cannot adapt to different cost budgets without re-training. The results reveal strong cor- relation between the actual and target cost for our method. We also observe that the actual
8

---

Published as a conference paper at ICLR 2025
cost of our method is consistently below the cost budget even when the budget is small, whereas CDT fails to ensure safety despite achieving higher rewards in some cases. Moreover, our method achieves better zero-shot adaptation by optimizing towards safe actions using constraint-conddtione critics and actor built with simple MIP models, unlike CDT and TREBI, which relies on the sequen- tial modeling abilities of transformer architectures and planning capacities of the diffusion models, respectively, to mimic the trajectories in the dataset.
CQL-Sauté
CDT
CCAC(ours)
max cost budget
Ball-Run
Car-Run
Ant-Circle
Drone-Circle
HaifCheetah-ve
Hopper-Vel
Cml. cost Cml, reward
3000
NOX
DAX
LON
180
130
145
26
60
120
200
1000 780
time-step
Figure 4: Results of cumulative reward and cost over time. The cost budget is 5 (increased to 10 at t = 2)2) for Run and Circle tasks and 10 (increased to 20 at t = T/2) for velocity tasks.
To further evaluate the adaptability to manage unseen cost budgets, we construct two more challeng- ing scenarios: (1) we manually assign an increased cost budget certain time-step during eval- uation and examine if the agents can respond to the change accordingly. Note that all the training data encounters a gradually decreasing cost budget as the cost budget is updated as Kt+1 = Nr Gt- Figure 4 shows the cumulative reward and cost over time. We can observe that our method can adapt to changes in the cost budget without compromising rewards or safety satisfaction, while other base- lines either violate constraints suffer a notable drop in rewards. (2) We build modified dataset by removing all the safe trajectories whose cumulative cost is smaller than a certain threshold. We re-train the agents and evaluate them using unseen cost budget that is set to be half of this thresh- old. The average evaluation results are presented in Table 2 with the detailed results for each task provided in Appendix D33 Additionally, for CQL-Sauté, CDT, and our method, we evaluate differ- ent additional unseen cost budgets that are all smaller than the threshold, and the adaptation results shown in Figure TO The results show that our method consistently attains high rewards while maintaining safety, whereas all basclines either fail to ensure safety or exhibit overly conservative behavior, highlighting our method's better adaptability and robustness to distribution shifts. On the other hand, the adaptability to unseen cost budgets is also linked to the so-called "stitching" abil- ity (Kumar et al., 2022). Despite the removal of safe trajectories, certain sub-trajectories within the unsafe ones can still be safe. The results suggest that our method is able to exploit these better than the existing techniques.
Tisks
Merni
CQL-Smit
BOQ-Lag
BEAR-1ag
CPQ
CDPIDICE
VOCE
CDT
TREBI
FISOR
OCACiours)
Reward
0.55ame:
1.51amm
1.88.100
0.95
1.38ass
1.65sun
0.99
0.82nnu
0.74
0.96:nax
Cust
5:23
19.85
23.56
1.92
6.276k2
9.35
1.34a004
2.12
0,54:100
0.23-927
Reward
0.40m
1.11483
1.13-431
0.64in
0.63108
0.0655
091--077
0.431005
0.27
0.79.a31
Circle
Cost
6.88
15.61ak
13.19
3.73
9,82
10.05
2.58
3.53
a.17
0.17.a79
Velocity
Roward
0.44
0.784857
0.18:04
0.411135
0.65a0
0041:044
0.88:03s
0.45:026
0.47:6a6
0.86:93
2.08
27.26.
20.1
184486
6.941m
0,0mm
2.1.22777
1.07a1.0
Reward
0.45 .044
1.091256
0.98..0
0.634475
0.81807
0.261506
(.91
0.51
0.44
0.85 azi
Average
Cost
1.9
20.Ma
17.8a1t44
13.81
8.07
6.54a121
24112448
2/41as0
0.22:11e
0.25:a34
Table 2: Average evaluation results of the normalized reward and cost of agents trained datasets with safe trajectories removed. The cost threshold is 1 (the cost budget is 5 for Run and Circle tasks and 10 for Velocity tasks). Each value is averaged over 20 episodes and 3 seeds. Bold: Safe agents. Gray: Unsafe agents. Blue: Safe agent with the highest reward.
5.3 WHAT IS THE IMPORTANCE OF THE OOD DETECTION COMPONENT IN CCAC?

---

Published as a conference paper at ICLR 2025
To assess the importance of the OOD detection part in method, i.e., the CVAE and the classifier, we first conduct an ablation study by constructing three variants of our method as shown in Figure [] where "w/ CVAE" and "w/ offline data" mean that the OOD data used to train the cost critic are either sampled from the CVAE and identified by the classifier, or ob- tained from the offline datasets based-on cost budgets; "w/o
CCAC(w/ CVAE offiine data)
CCAC(w/ CVAE & offline data)
CCAC(w/ CVAE & w/ offiine data)
CCAC(w/ CVAE) & w/o offline data)
cost threshold
4.0
3.5
CVAE" and "w/o offline data" mean the absence of these el- ements. Note that CVAE(w/ CVAE & w/o offline data) the CVAE(ours) in the remaining tables and figures. We can ob- serve that when using both the generated OOD data and OOD offline data, the performance does not change much since the OOD distribution v(ss, a ()) that the CVAE aims to approxi- mate covers the OOD offline data. The result also shows that the limited OOD offline data only offers limited benefits in learning the crities and policies. The CVAE and classifier essential components as their removal leads to substantial in- creases in cost and moderate drops in rewards. The CVAE enables the generation of diverse data not present in the offline dataset, and the classifier can determine which of these generated samples are OOD.
Eval reward
0.6
2.5
1.5
1.0
0.2
0.0
0.0
Figure 5: Ablation study; average performance of CCAC and its vari- ants in Run and Ci Circle tasks.
IND Q
OOD Q
Qc threshold
Ball-Run
Car-Run
Ant-Circle
Ball-Circle
Car-Circle
Drone-Circle
G-vallue
Cost budget
Figure 6: Ablation study: Qe-vals plots. The solid line and the light shade area represent the mean and mean + standard deviation, respectively. The dashed line denotes the -alva thresholds.
We also evaluate the Qe-values for both the IND and OOD state-action pairs. For cost budget k, we sample state-action pairs whose cost-to-go satisfies the threshold from the offline dataset and compute E. (s, a/k) a/{x] as IND Qe- Next, we generate state-action pairs using the CVAE, identify the OOD samples using the classifier, and compute Ls a/k) as OOD Q.. The results are shown in Figure We can observe that the IND Qe satisfies the threshold while OOD Qe exceeds the threshold. The results are consistent with Eq. which aims to overestimate the Qe- values of OOD state-action pairs while minimizing the Qa-val of IND state-action pairs. This indicates the effectiveness of the data generation and detection components in our method.
CONCLUSION
We propose novel method called CCAC that leverages the constraint information to guide policy learning and deployment with the support of constraint-related data generation and OOD detection. Empirical results demonstrate that CCAC excels in learning a safe and high-reward policy across multiple offline safe RL tasks, whereas prior works either show overly conservative behaviors or struggle to meet safety requirements. Notably, CCAC can adapt to different or even unseen cost budgets without re-training and consistently maintain safety and achieve high rewards. The ablation study highlights the critical role data generation and OOD detection in our method. A bottleneck of our method could be the data generation and OOD detection component when the dimension of state-action increases. However, we show a feasible direction by integrating such techniques to regularize critics and policy training. Future works will investigate the incorporation of more advanced data generation and OOD detection techniques and the possibility of offline-to-online finetuning further improve policy performance since the offline dataset can be sub-optimal.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors thank the anonymous reviewers for their invaluable feedback and constructive sug- gestions. This work was supported part by the U.S. National Science Foundation under grant CCF-2340776.
ETHICS STATEMENT
This work does not present any significant ethical concerns. All experiments carried out in simulated environments, preventing any risk to real-world systems. The focus on offline datasets ensures the avoidance of potentially dangerous and unsafe interactions during policy learning, pro- moting safety and responsible practices in reinforcement learning.
REPRODUCIBILITY STATEMENT
The datasets used are provided from a publicly available benchmark that uses simulated dynamical control environments, ensuring that the experimental setup can be reproduced. Details about experimental setup, including parameters, models, evaluation metrics, and implementations, are provided in the appendix and supplementary materials to facilitate the reproduction of our findings.
REFERENCES
Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48(3):387-417, 1998.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010.144997 2020.
Chenyang Cao, Zichen Yan, Renhao Lu, Junbo Tan, and Xueqian Wang. Offline goal- conditioned reinforcement learning for safety-critical tasks with recovery policy. arXiv:2403.01733 2024.
arXiv preprint
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Xi Chen, Ali Ghadirzadch, Tianhe Yu, Yuan Gao, Jianhao Wang. Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent- variable waaatage-weighhee policy optimization for offline rl. arXiv preprint 2rXiv:2033.08442 2022.
Minjae Cho, Jonathan P How, and Chuangchuang Sun. Out-of-distribuite adaptation in offline rl: Counterfactual reasoning via causal normalizing flows. arXiv preprint arXiv:2405.03892 2024.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167): 1-51, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Buenez-Guzman and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint 1901 2019.
Christopher Dichl, Timo Sievernich, Martin Krüger, Frank Hoffmaann, and Torsten Bertram. Um- brella: Uncertainty-aware model-based offline reinforcement learning leveraging planning, 2022.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. corr abs/18 12.02900 (2018). arXiv preprint arXiv:1812.02900 2018.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent without entropy. arXiv preprint arXiv:2301.02328 2023.

---

Published as a conference paper at ICLR 2025
Sven Gronauer. Bullet-safety-gnn gym:: A framework for constrained reinforcement learning. 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review safe reinforcement learning: Methods, theory and applications, 2024.
Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Zijian Guo, Weichao Zhou, and Wenchao Li. Temporal logic specifcaaion-condiiioned decision transformer for offline safe reinforcement learning. In Proceedings of the 41 International Con- ference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp- 17003-17019, PMLR, 21-27 Jul 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming. pp. 1861-1870. PMLR, 2018.
Kihyuk Hong, Yuhang Li, and Ambuj Tewari. A primal-dual-crite algorithm for offline constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp- 280-288. PMLR, 2024.
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304, 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel Model-
based offline reinforcement learning. 2021.
Diederik P Kingma and Max Welling. 13Xiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Flemming Kondrup, Thomas Jiralerspong, Elaine Lau, Nathan de Lara, Jacob Shkrob, My Duc Tran, Doina Precup, and Sumana Basu. Towards safe mechanical ventilation treatment using deep offline reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15696-15702. 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
off- policy q-
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 1191 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline rein- forcement learning over behavioral cloning? arXiv preprint arXiv:2204.05611 2022.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim. and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint (rxXi:2204.089533 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164) 2020.
Haohong Lin, Wenhao Ding, Zuxin Liu, Yarı Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters, 2024.
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference Machine Learning, pp. 2122-22155, PML.R 2023.
12

---

Published as a conference paper at ICLR 2025
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive udvantage-guide guided policy regularization for offline reinforcement learning. arXiv preprint arXiv:2405. 19909, 2024.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205. 14691, 2022.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wen- hao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306. 09303, 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Con- ference on Machine Learning, pp. 21611-21630. PMLR, 2023b.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09330 2020.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained re- inforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019.
Nicholas Polosky, Bruno C Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained of- fline policy optimization. In International Conference Machine Learning, pp. 17801-17810. PMLR, 2022.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions Neural Networks and Learning Systems, 2023.
Tianyu Shi, Dong Chen, Kaian Chen, and Zhaojian Li. Offline reinforcement learning for au- tonomous driving with safety and exploration enhancement, 2021.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.08866 2023.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Aivar Sootla, Alexander I Cowen-Rivery Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmenta- tion. In International Conference on Machine Learning, pp. 10423-20444 PMLR, 2022.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278- 1291. 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
arXiv preprint 1rXiv:1911.111
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022.
Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. arXiv preprint 200.:2207..46533 2024a.
13

---

Published as a conference paper at ICLR 2025
Yihang Yao, Zuxin Liu, Zhepeng Cen, Jiacheng Zhu, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constraint-condition policy optimization for versatile safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020.
Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with ex- plicit behavior density for offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: conditional sequence modeling approach to offline safe reinforcement learning, 2023b.
Yang Zhang, Bo Tang, Qingyu Yang, Dou An, Hongyin Tang, Chenyang Xi, Xueying LI, and Feiyu Xiong. BCORLE(S)) (ambdaS); An offline reinforcement learning and evaluation framework for coupons allocation in e-commerce market. Advances Neural Information Processing Sys- tems, 2021.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Feasibility-guider safe offline reinforcement learning. The Twelfth International Confer- Learning Representations, 2023.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 7176-7183, 2023.
14

---

Published as a conference paper at ICLR 2025
A PROOF OF THEOREM 4.1
Proof. Let Q denote the true Q-function iteration & and Q denote the k-th Q-function obtained by solving 5). Q++1 is related to the previous Q - T=Q². By introducing the La-
grangian multiplier A.. Eq. becomes (q. then take the derivative Eq.(6) w.r.t. Qo and set to O, we obtain the following expression for Qe in terms of in terms of Q::
(s, alk) TQ. + de v(s, Ak
(10)
2 TD(s,
Note that we only sample y once from D. so the ~ DD in the objective and constraint in Eq are the same and will cancel out in (10). Since v(ss, alk) 0, b 2 0, TD (s, k)) 0, each iteration
2 (s, Now, let's examine the fixed point Eq.10 10 - TQ de a(s,a(k)
TD(s,
= de v(s,alk)
TD(s,a(k)
de v(s,alk) TD(s,
=]](s,]])(( - )P*) + 7P 7P= (T,
Ae a()
= (7(s,a)k) + y PT (s, (s,
+
2 TD(s,a())
+ de (I- (I-~P")-- a(s
(s. =
2
TD(s,a/k)
We set Ae 2 0 since we don't want to underestimate the Q.-values. When (5, alk) > k. means that the stte-acttio pairs OOD, e.g., unsafe to take the actions in the states, then want to overestimate Q: (s, alw), the choice of Xe that guarantee Q% (s, als) 2 is given by:
WD (8, a)))
de 2 2(E (z, - 7P=)
m(s, a [x)
TD (s, ak)
+ Ae 2 2 (E Q: ()))
a(s
We show that the Qo-values of OOD state-action pairs can be overestimated exceed the threshold.
B ENVIRONMENTS DETAILS
B. REWARD AND COST FUNCTIONS
We use the Bullet-san salett-ffeetygyym (Gronauer] 2022) and Sa Safeyy-yynnsii um (Ji et al. 2023) for this set of experiments. We consider three tasks: Run, Circle, and Velocity ty and multiple types robots: Ant, Ball, Car, Drone, Hopper, and HalfCheetah.
In the Run task, agents receive rewards for high-speed movement between two safety boundaries. However, they incur penalties when they either cross these boundaries or surpass a velocity threshold that is specific to different types of robots. In the Velocity task, agents also receive rewards for moving forward but incur costs they exceed a specified velocity threshold. The reward and cost function are defined as:
r(st) = || 1 1-122 - |x+ - 8//2
c(s.) 1(y) Mim ||v.l12 > vhi
where 8; = [ yes Uz, vy]. r is the radius of the circle, and Fim specifies the range of the safety region. In the Circle task, agents gain rewards for circular motion in clockwise direction but are required to remain inside a designated safe area, which is smaller than the circumference of the intended circle. The reward and cost functions are defined as: - + Zy
r(st) =
1+ || | 2 - 1( Fimm
c(st)
15

---

Published as a conference paper at ICLR 2025
where ylm is the safety boundary and Vim is the velocity limit. Since the cost function is the indicator function so the cost-to-go represents the number of constraint violations.
B.2 OFFLINE DATASET VISUALIZATION
The dataset cost-reward plots for tasks Ball-Run, Car-Run, Ant Ant-Circle, Ball-Circle, Car-Ci rrle e, and Drone- rcle, are shown in Figure Analyzing the figures provided, can generally discern an increasing trend for the reward in relation to the cost. In other words, as cost increases, too might the reward return, underscoring the inherent trade-off between reward and cost. This phenomenon aligns with findings discussed in previous works (Liu et al. 2023b 2022 Guo et al. 2024). We utilize the data density filter in Liu et al. (2023a) to evaluate the algorithms ability to perform under different levels of data availability, e.g., less training data means a larger distribution shift during evaluation, and assess their generalization capabílities. We also use the partial data filter to filter out some specific trajectories, as shown in Figure & tasks, the safe trajectories whose cumulative cost smaller than 10 are removed. For Ve Velocity tasks, the safe trajectories whose cumulative cost is smaller than 20 are removed.
For Run and Circle
Ball-Run
Car-Run
Ant-Circle
Ball-Circle
Car-Circle
Reward
30
30C 150
26
25
Cost
Figure 7: Illustration of the offline dataset. Each row shows reward versus cost for varying data percentages. Each column represents an environment. Each point denotes collected trajectory with corresponding episodic reward and cost values.
C IMPLEMENTATION DETAILS
C.1 OUR METHOD
We present the full algorithm of our method in Algorithm There two sets of components that need training: the first includes the RL components, i.e., the critics and the policies; the second in- cludes the data generation components, i.e., the CVAE and the classifier. Directly training all these components simultaneously would make it difficult to converge since, at the beginning of training, the poor-quality OOD data generated by the CVAE and weak classifier that cannot correctly iden- tify OOD samples would lead to an erroneous update of the critics and policies, especially the cost critic. Therefore, we train these two sets of components separately. We first train the CVAE and classifier for M iterations until convergence. Then we perform policy training which follows the standard training procedure for deep RL methods: at each iteration, we sample a batch of data, up- date critics, update the actor, update the target critics, and so on. To create OOD samples, randomly in the latent space, 2, and use the CVAE decoder to produce state-action pairs. These gen- erated pairs then fed into the classifier, which outputs the probability. h(s,, a [t), of being OOD. We classify the pair as OOD if h(s,, a/и) exceeds a threshold of 0.5. For evaluation, we first specify an initial cost budget K1, and then this cost budget is updated autoregressively based constraint
sample
16

---

Published as a conference paper at ICLR 2025
removed data
remaining data
Ball-Run
Car-Run
Ant-Circle
1250
993
933
433
253
U/5
180
Ball-Circle
Car-Circle
Drone-Circle
Reward
1000
430
400
Ant-Velocity
Hopper-Velocity
HalfCheetah-Velocity
2030
1033
300
Cost
Figure 8: Illustration of the offline dataset with partial data removed. Each point denotes collected trajectory with corresponding episodic reward and cost values.
violations K4+1 K2 Ct. Then, we compute the cumulative cost Ex_ Cr and compare it with the initial cost budget. smaller cumulative cost means safety is achieved.
Algorithm Cost-Conditioned Actor-Critic (CCAC)
Require: offline data D
1: // CVAE and Classifier Training
2: Initialize encoder pe, decoder 4p, and classifier hip
3: for t = 0, ... M do
4: Sample a batch of data (s, k) ~ D and cost threshold R
5: Update encoder P.e and decoder 9, by Eq.(3)
6: Get reconstructed state-action pairs (s, à) via CVAE
7: Assign the labels y for each state-action pair based and
8: Update classifier he by Eq.
9: end for
10: // Policy Training
Initialize reward critic ensemble (Qr, (s, alw)) and cost critic ensemble (Qe, (s, akx)] actor Lagrangian multiplier d and (x(K), target networks (0r )n and ( in with
S/ +o', o', and di d' for 0, .... N do
12:
13:
Sample a batch as data (s, a, K, 8') ~ D
14:
(Optional) Perform one iteration of CVAE and classifier training
15:
Sample the same size of 2 p(z[k) conditioned on the x in the batch
16:
Generate state-action pairs (5, â) ps,ss, a|2,
17:
Assign labels to (8, a) using the classifier
Update cost critics and Ae by Eq. 6 and reward critics by Eq.
19:
Update actor and Xa (K) by Eq.
Update target cost eritic: d' -- aΦ. + (1 a)σ'
21:
Update target reward critic: of + aΦ, + (1 - a'o'
end for
17

---

Published as a conference paper at ICLR 2025
C.2 HYPERPARAMETERS
For BCQ-Lag, BEAR-Lag, CPQ, COPáDICE and CDT, we use the OSRIL implementation and we implement method in this framework, We adopt the CQL-Sauté from this CQL implementa- tion For VOCE FISOR and TREBI" we use their official implementations. For value-based baselines, we use Gaussian policies with mean vectors given as the outputs of neural networks, and with variances that are separate learnable parameters. The policy networks and Q networks for all experiments have two hidden layers with ReLU activation functions. The Kr, K and KD are the PID parameters Stooke et al. (2020) that control the Lagrangian multiplier for the BCQ-Lag and BEAR-Lag. For each task, the update steps is the same for all the methods. For fair comparisons, we use the same model structure for the critics and policies for all the value-based methods. However, CDT and FISOR take advantage of more expressive transformers and diffusion models, making difficult to make fair comparisons: CDT uses Transformer model to take a sequence of states and actions as input, whereas our method only takes in current state; FISOR uses a ResNet-based diffu- sion model its policy network, whereas ours is simple MI.. When switching from the diffusion model to an MLP architecture, FISOR results in significant performance degradation. Therefore, we report results based on their original design and default hyperparameters. BCQ-Lag and BEAR-Lag employ VAE to generate actions, while our method uses CVAE to generate state-action pairs. The constraint threshold 5 across all the tasks. The hyperparameters that are not mentioned are their default value for baselines. The hyperparameters used in the experiments are shown in Table B
Parameter
Ball-Run
Car-Run
Ant-Circle
Ball-Circle Car-Circle Drone-Cirele
Velocity
Actor hidden size
[256, 256]
Critic hidden size
[256, 256]
VADCVVA hidden size
[512, 512, 64, 512,
Episode length
100
200
500
200
300
300
1000
[Kp. KG, Kp)
10.1, 0.003,
0.001]
BCQ-Lag, BEAR-Lag
Batch size
512
1024
512
512
512
512
2048
Training steps
2e5
le5
2c5
2e5
2e5
3e5
2e5
0.99
Actor learning
le-4
Critic learning
le-3
VAE/CVAE learning
lc-3
Critic ensemble
Table 3: Hyperparameters for value-based methods (including our method).
D MORE EXPERIMENT RESULTS
D.1 TRAINING CURVES
The training curves of our method and the baseline methods are presented in Figure 9 the methods at regular training intervals, which is optional, and the interaction data is not added to the offline dataset used to train the methods. These curves follow the trends observed in online model-free safe RL methods, where costs typically start high and decrease over time, while rewards begin low and increase. the offline setting, data collection is not needed and training safety is ensured due to the offline datasets. After convergence, our method consistently remains below the cost threshold and achieves high rewards, while other baselines either struggle to satisfy constraints or suffer from conservatism. Note that for our method, the constraint threshold is only used during evaluation; during training, the method is unaware of the threshold under which it will be evaluated. As mentioned in section 5 consider the scenarios where pursuing rewards aggressively will lead to significant costs and excessive demand for safety will lead to low rewards. In our experiments, for Circle e task, traveling along the circle will get high rewards but will enter unsafe regions; for Run task, driving as far as possible will get high rewards but will exceed the speed limits. On the other hand, the agents can simply take no actions and stand still achieve safety. The training curves reveal such trade-off' between rewards and costs. For example, BCQ-Lag, BEAR-Lag and
We evaluate
https://iihub.ccm///uzzus////
huttss//gthubbco///uuaaavvveeeee
18

---

Published as a conference paper at ICLR 2025
BC-safe
BCQ-Lag BEAR-Lag
CPQ COptiDICE
VOCE
FISOR CCAC(ours)
CQL-Sauté
CDT
Ball-Run
Car-Run
Ant-Circle
Eval reward
502
253
-100
Eval cost
150
03
43
50-
140
Ball-Circle
Car-Circle
Drone-Circle
500
Eval reward
400
300
602
200
200
100
200
Eval cost
102
1.0
1.5
55
15
78
3.
Ant-Velocity
Hogper-Velocity
HallCheetah-Velocity
Eval reward
1500
500
01
200
1000
Eval cost
207
000
150
600
202
202
000
1.0
2.0
3.5
0.5
1.5
2.5
Training steps
Figure 9: Training curves. The dashed line represents the cost threshold, which is for Run and Circle tasks and 10 for Velocity tasks. The solid line and the light shade area represent the mean and standardd deviation. The interaction data is not added to the offline dataset.
COptiDICE often attain high rewards across various tasks, but this is accompanied by elevated costs. While FISOR ensures safety, it yields low rewards, achieving only about half of what our method achieve. In contrast, method can balance the rewards and costs
analyzed in sections 5
Sample efficiency. We can also observe that certain environments, such as Car-Run and Drone- Circle, CCAC takes more steps to converge. However, it is important to note that we working in the offline safe RL setting where training occurs solely on a fixed offline dataset Taking more steps to converge does not imply additional safety violations there is interaction with the environment during training. Besides, evaluation during training is optional, and Figure 9 is provided solely as reference to illustrate the training progress. Furthermore, while sample efficiency is valuable, that
method that achieves efficiency at the expense of safety may be less meaningful. although some baselines converge faster, such FISOR Car-Run and BCQ-Lag Drone-Cireh they either fail to achieve high rewards or cannot maintain safety after convergence. In contrast, our method consistently achieves both high reward and safety after convergence.
19

---

Published as a conference paper at ICLR 2025
D.2 COMPLETE RESULTS FOR DIFFERENT DATA PERCENTAGES
The detailed evaluation results for different dataset percentages are presented in Table 4 For base- lines, we focus on BC-safe, CQL-Sauté, and FISOR. BC-safe helps gauge the difficulty of task. If mimicking the safe actions in the datasets achieves both safety and high rewards, then the task relatively easy. CQL-Sauté is adopted from (Sootla et al. 2022) by combining the conservatism estimation technique of (Kumar et al., 2020) into the offline setting. This baseline also employs the cost-to-go, like our method, but focuses solely on maximizing rewards. FISOR is the only baseline method evaluated that ensures safety, The results show that our method and FISOR consistently stay within the cost threshold across all tasks, with our method displaying higher rewards in all tasks. We can also observe that in most tasks, rewards decrease, costs increase, and variance rises as the percentage of data being used to train the methods decreases, as less training data typically leads to greater distribution shifts during testing. FISOR achieves this through its conservative approach while we achieve this by data augmentation and OOD detection.
Arr-Cirde
Hall-Circle
Car-Cincle
Circl
Avergge
Methods
Rewadd 1
Cost J.
Reward 1
Cost J
Reward
Cost 1
Reward
Cost
Reward
Cost 1
Reward 1
Clos
Reward
Closs
Ous (p- 2.0)
0.97.0ML
0.27.440
0.95.KM4
0.19uxz
1.01:s2s
0.55
.M00
0.0.a
0,85 Loue
0.73.185
0,82.aa1
0,87.444
6.911a
0.3
(p- 0.75)
0.97.041
0.27.419
0.95xxc
0.13:k1
1.0.433
0:66144
6.87
0.0.8o
0.84
0.85
0.79uoas
0.0:00
0.9:817
Our (
0.96
0.33:
0.97:xm
0,3:47
0.9:4
0.83
0.88
0,0
0.85
0,96
0,76:0a9
0.68:1m
0,89
0,52.130
Ouzs (p- 0.25)
0.94
0.39..2
0.93.a01
0.36.a
0.83aa0
0.93
0.82
0.22
0.87
1.34235
0.72.aa
0.56.4.3
0.84
0,66 .122
FISOR (p- 1.00
0.67.a.
0.0uae
0.78
0.0.0a
0.15
0.25
0.0..o
0.25 2007
0.0.a0
0,63.a09
0.0.00
0.45
0.8.
FISOR (170)
0,68.033
0.0.00
0.83.008
0.0.0a
0.161814
0.0.ca
0.37.am1
0.0.
0,21
0.0.a0
0.68.021
0.0.00
0.89.
0.0.
FISOR 0.50
0.88
0.00
0.72
0.0.8a
0.16:a12
0.0.0a
0.39
0.0
0,31
0.0:a0
0,69una1
0.0.00
0,47
0,0180
FISOR (== (125))
0,88
0.0cas
0.74
0.02:am
6.16:am
0.0.aa
0.41
0.0:aa
0.25 0039
0.0:a0
0.74:0x2
0.0
0.68:a34
0.0:a0
BC-sfi (y
2.08-1
0,566
0.67
0,80
1/7k5
BC safe (p 0,71
0,46u029
3.66.44
0.94
0.11:0.
0,574
4.25
0,59uau
1.68.133
0.43133
1.53,334
0,78Ln04
1.46.191
0.63
2.12.346
BC sale (p - 0.53
0.6uaee
3.63.113
0.84.K15
0.44.am
0.571a22
4.60u519
0.59ua2
2.32.119
0.42
1,08
0,777
2,00.:170
0.63
2,36.4
BC safe (p- 0.2M
0.61
5.09.11
0,76.a3
0.31:a
0,50835
4.37u547
0,54axa2
1.33at85
0,52
1.29
0.677o0a
0,57a043
2.08
CQL-Santé
1:985oo::
9,266s03
0.675124
0.2
0.0
0.68
0,02
0,540nas
2,8485a
0.15
2.499mm
0.74
2:55
CQL. Stuné (u- 0.75)
L/BHuna
6.56us3
0.97.am
0.85...8
0.21:am
0.0.aa
0.69
0.04
0.51uxx
1.92
0.20aax
4.02.6.0
0.68aass
2.23.445
CQL Saaté (p- - 3.5)
L/63uess
8,47
0.951104
1.842212
0.2.414
0.32.125
0.67
0.02.009
0.43
2.31834
0.17.x0
2.82...4
0.68a0as
2.61444
CQL Saut
1.97.1899
12.44.44
0.95.000
0.1.a1
0.21
0.8...0
0.67.000
1.31azas
0,52.0as
0,87.331
0,26.oas
3.211731
0/75 3,09
Table 4: Evaluation results of the normalized reward and cost under different data percentages. The cost threshold is 1. p 1.0/0.75/0.5/0.222
means 100%, 75%, 50%, and 25% of the offline data is used during training respectively. 1: the higher the reward, the better. J: the lower the cost (up to the threshold 1). the better. Each value is averaged over 20 episodes and 3 seeds. Bold: Safe agents whose normalized cost is smaller than 1. Gray: Unsafe agents.
D.3 ADAPTATION RESULTS OF UNSEEN COST BUDGETS
We train the agents using the dataset shown in Figure g with safe trajectories being removed. During evaluation, the cost budget is set be [3, 4, 5, 6, 7, 7,8,9 9] for Run and Circle tasks and [3, 6, 9, 12, 15, 18] for Velocity tasks. The adaptation results are shown in Figure 10 We can observe that our method is capable of adapting to different unseen cost budgets while the baselines either struggle to maintain safety or display overly conservative behaviors, resulting in low rewards.
Table 5 shows the detailed results of Table 2 The agents are also trained using the dataset shown in Figure S For evaluation, the cost budget is 5 for Run and Circle tasks and 10 for Velocity tasks. is important to note that for CQL-Sauté, CDT and our method, during training, they have no access to the cost budget that will be used to evaluate them. In contrast, for the rest of the baselines, although they are trained using the same cost budget as evaluation, the safe trajectories are removed, requiring them to display some sort of adaptability to learn safe policies from unsafe datasets. Com- pared with the results in Table I where the agents are trained using the whole dataset, we can observe that the number of safe agents decreases, and most of the methods achieve higher costs since the remaining dataset consists of unsafe trajectories with higher costs. However, our method still achieves the best performance among all the methods in terms of safety and high rewards.
20

---

Published as a conference paper at ICLR 2025
CQL-Sauté
CDT
CCAC(ours)
Eval cost
Cost budget
Ball-Run
Car-Run
Ant-Circle
Drone-Circle
HalfCheetah-Vel
Hopper-Vel
Eval reward
300
104
50
120
Eval cost
Cost budget
Figure 10: Results of adaptation to different unseen cost budgets. The agents are trained on a dataset where safe trajectories with cumulative costs smaller than certain threshold removed. The top- row plots show the evaluated reward and the bottom-row plots show the evaluated cost. The solid line and the light shade represent the mean and standat deviation. The dashed line represents the scenario where the evaluated cost matches the cost budget.
Taks
Metric
OQL-Saute
BCQ-Lag
BEAR-Lag
COpiDICE
VOCB
CDT
FISOR
CCAC(our))
Reward
0.18aa3s
1,98111
2.88asus
0.96stm
1.87a0
2.37ati
0,98stic
0.74:auu
0.99
Ball-Run
Cost 1
5.93.139
10.36
16.82s125
2.73aA.0
12.55an8i
12.55
1.59ptai
0.34:1.0
0.47:au1
Reward 1
0.93
1.04stic
0.89
0.95sm02
0.89.002
0.93a002
0.99.m0
0.74
0.93-am
Car Run
Cost L
4.53-2
29.34
30,29
1.12
0.0:ae
6.15e785
1,0996d5
0.74:2.3
Reward
0.13-am
1.274400
1.14
0.01
0.36+037
0.0-an
0.87-46s
0.09.
0.76-4.45
Ant- CireCir
Cost 1
0.84 255
19.98 lor
12.03
0.0
15.17
0.0 a.
5.32 1.
0.0.na
0.19 au
Reward
0.68=a4s
0.99
1.13+a.1
1.06.11
0.94
0.0s001
0.96st04
0.31:613
0.86:1.00
Ball-Circle
Cost
9.97-3.9
11,42:381
11,24
10.2e1.44
32.56+13.0
2.0g05
0.0.0a
0.0.s0
Reward
0.68-931
0.8144088
1.0
0.96-o07
0.62:005
0,261004
0.85-oom
0.26
0.8-084
Car-Cirele
Cost L
10.36
11,68
10.1
9:21 21
11.831571
7.65.4
1.54
0.0 nu
0.45 1.30
Reward
0.44
Droae-Cirele
Cost
10.34
19.38
14.39.174
0.73.am
2.08.1.9
0.0.aa
1,45.e7v
0.70.1.39
0.03.a22
Reward
0.1-a16
0,7-034
-0.72aa
-0.96-a144
1.05400l
-1.02.0a
0.99-ne
0.47
0.71
Ant- Vnt-Velocity
Cost )
0.41 155
17.82
0.0
0.0-ne
10.83:517
0.0 an
1.14
0.0..s
0.39
Reward
0.98.com
0.96.uu
HallChootab-Velecity
Cost
0.71
52.22
54.14.11.w
97,88
0.0.ae
0.0.ap
2:28.1125
0.0.ua
0.3:
Reward
0.2-0.1
0.544635
0.174em
0.35
4.21
0.01
0.65-453
0.15:amc
0.91-414
Hopper-Velocity
Cost $
5.11-4m
11.73
6.161125
7.66179
9.98
0.0 ap
5.32
0.25
0.44 am
Roward 1
0.45en4
1.09
0.98
0.634075
0.84047
0,26
0.91=019
0.44
0.85:111
Average
Cost
4.91
20.44:1933
17.8
13
13.81
8.07:8.s
6.54x121
2,41
0.22-146
0.25
Table Complete evaluation results the normalized reward and cost of agents trained on datasets with partial data removed. The cost threshold is 1 (the cost budget is 5 for Run and Ci rcle tasks and 10 for Ve Velocite ty task). t: the higher the reward, the better. J: the lower the cost (up to I), the better. Each value is averaged over 20 episodes and 3 seeds. Bold: Safe agents. Gray; Unsafe agents, Blue: Safe agent with the highest reward.
D.4 EVALUATING POLICY REGULARIZATION VIA IQL INTEGRATION
To evaluate whether our method achieves a similar policy regularization effect, we construct the following experiment. We adopt existing method, IQL (Kostrikov et al. 2021). which learns Qr,s, a) and ((s) by using expectile regression with a SARSA-style objective to avoid sampling OOD actions; thus, the leamned (s, a.) represents the value of the behavior policy. Then replace (q. with IQL for learning (s, [k). while keeping all other components unchanged. The results in demonsstett that integrating IQL. has little impact performance, which provides further
21

---

Published as a conference paper at ICLR 2025
support that the overestimation of Qe(s, al)) discourages OOD actions in manner similar to policy regularization (without the direct regularization of (,,s,, alk)).
Task
Metric
CCAC(in Table 1)
CCACIIQ
Reward
0,97 1 0.01
0.98 1 0.03
Ball-Run
Cost +
0.27 + 0.19
0.19
Reward
0.87 1:
0.02
Ball-Circle
Cost J
0.0 1 0.0
0.0 0.0
Reward
0.85 + 0.04
0.8
Car-Circle
Cost I
0.73 + 1.95
+ 0.96
0.96
Reward
0.9 = 0.05
Ant- Vel-Velocity
Cost
0.58 1 0.15
0.11
Reward
0.96 0.04
0.99 + 0.02
HalfCheetah- Velocity
Cost J
0.79 1 0.2
0.3 0.23
Reward
0.91 0.07
0.9 + 0.08
Average
Cost J
0.47 1 0.93
0.43 1 1.07
Table 6: Evaluation results of the normalized reward and cost. f: the higher the reward, the better. J: the lower the cost (up 1), the better. Each value is averaged over 20 episodes and seeds.
22


Reviews:
[{"review_id": "ObjQ5VMNMR", "reviewer": "Reviewer_3i9u", "summary": "The paper presents a constraint-conditioned actor-critic method, which combines a constraint-conditioned variational autoencoder (CVAE) and a constraint-conditioned classifier to learn this relationship.", "questions": "1. The training process of CCAC appears to separate the training of the CVAE and the policy (optional once). Why not consider continuing the training of the CVAE during the policy training process?\n\n2. CCAC demonstrates zero-shot adaptation to different cost budgets. In addition to empirical evidence, could you provide some intuitive explanations for this capability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The visualization of the offline datasets is quite intuitive, and the experimental design, along with the ablation study, adds significant value to the research.\n\n3. The method can learn safe and high-reward policies from offline datasets, demonstrating better adaptability and robustness to distribution shifts.", "weaknesses": "1. The experimental environments are relatively simple and do not include more complex tasks, such as humanoid. In the experimental section, the authors use only three seeds, which I believe is insufficient. If possible, could you provide additional results using more seeds (at least five)?\n\n2. Although the algorithm demonstrates performance compared to the baselines while maintaining safety, Figure 9 suggests that the algorithm's sample efficiency is not good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iMcivgEEsu", "reviewer": "Reviewer_REk7", "summary": "In the context of offline safe reinforcement learning, this paper implements penalties for out-of-distribution (OOD) actions using a generative model, such as a conditional variational auto-encoder (CVAE). \nThen, the authors propose a method to train a policy that conditionally adapts to different cost budgets using overestimated cost Q functions.", "questions": "- It appears that the policy is trained through Equation 9. In traditional offline reinforcement learning methods, a penalty term is added to a policy loss to ensure that the actions of the policy do not deviate from those in the dataset. However, the proposed method addresses this by solely using the overestimated $Q_C$. Is this correct?\n\n- In Table 1, COptiDICE shows significant constraint violations, but in its original paper, COptiDICE reported low violations in similar tasks. \nAn explanation for this discrepancy is needed. \nIn line 323, the authors mentioned that this is due to biased estimation. Still, this explanation is not entirely convincing since COptiDICE itself includes a component that trains the policy by overestimating the cost of OOD data.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to read.\n- The proposed method shows the best performance.\n- Through various ablation studies, the paper effectively shows that the proposed method is robust to OOD scenarios and can generate adaptive behaviors across different cost budgets.\n- Section 5.3 provides experimental justification for the usage of CVAE and good visualization of the overestimated $Q_C$.", "weaknesses": "- In lines 46-53, the need for a varying threshold is highlighted, but this issue has been addressed in several studies [1,2]. \nIt is crucial to motivate the proposed method by identifying shortcomings or potential improvements in the existing techniques on varying thresholds.\n\n- In line 98 of Related Work, the statement “these methods overlook the dependence of actions on the constraints” seems to need to be toned down a bit. This seems obvious since the works mentioned by the authors are related to offline RL, not offline safe RL.\n\n- The overview in Figure 1 needs to be further elaborated. It is difficult to understand what is being illustrated in the figure, as it does not clearly describe the inputs and outputs of each model and does not explain the learning flow. Also, the caption should include a description of each module and arrow.\n\n- In lines 203-204, there is a need for a detailed explanation of the mechanism for generating out-of-distribution (OOD) data. According to the attached code, it appears that the $(s, a)$ pairs are generated through the CVAE using various Gaussian noise vectors $z$. Pairs, where the value of $h$ is less than $0.5$, are designated as OOD data. This process should be clearly explained in the paper.\n\n- The TREBI paper [2] also deals with various cost budgets in offline environments. Unfortunately, the authors do not compare CCAC with this method, which deals with the same problem as the proposed method.\n\n## References\n- [1] Liu, Zuxin, et al. \"Constrained decision transformer for offline safe reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n- [2] Lin, Qian, et al. \"Safe offline reinforcement learning with real-time budget constraints.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eeBi3ktFC", "reviewer": "Reviewer_pD1A", "summary": "This paper focused on offline safe RL. The authors proposed CCAC that can learn with adaptive constraint budgets. CCAC first trained a binary classifier to distinguish OOD state-action pairs with constraint budgets acting as labels. The OOD distribution was then used for constrained optimization for the actor and critic. The authors provided theoretical justification and comprehensive empirical investigation.", "questions": "Please refer to the above for questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written in general. The idea is interesting and technical details easy to follow. The authors conducted extensive experiments showing CCAC performed favorably against the existing methds.", "weaknesses": "My concern is that the paper seems to blur the boundary between out-of-distribution and out-of-budget (OOB) state-action pairs. If we assume all OOD state-action pairs are unsafe, then it means the algorithm cannot generalize to unseen state-action pairs since their costs are lower bounded by $\\epsilon$. This seems at odds with the standard offline RL setting which seeks to generalize and improve upon the behavior policy and beyond the dataset. \nIf we recall that the key issue of offline RL lies in that incorrect action values of unseen state-action pairs cannot be corrected, then in my opinion mixing OOD and OOB runs the risk of not generalizing beyond the dataset $\\mathcal{D}$ since safe unseen $(s,a)$ always have $Q_c \\geq \\epsilon$. If this is true, then later analysis like Theorem 4.1 is not very meaningful (which itself is an variant of CQL derivation imo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8y787Dx08", "reviewer": "Reviewer_GJjh", "summary": "This paper addresses the challenge of safe offline reinforcement learning (RL) and introduces a novel approach termed Constraint Conditioned Actor Critic (CCAC). CCAC is designed to facilitate adaptive policy learning under various constraint limits while ensuring high rewards and minimizing distributional shifts. To achieve this, CCAC employs a cost-to-goal conditional Variational Autoencoder (CVAE) to model the adaptive behavior across different safety constraint intensities. Additionally, a discriminator is utilized to autonomously evaluate the safety levels. By integrating the discriminator with the CVAE, CCAC establishes an Out-of-Distribution (OOD) detector, \\( v(s,a|k) \\), that assesses both distributional shift and safety performance concurrently. Utilizing this framework, CCAC executes standard actor-critic updates to develop a constraint-aware critic and policy. The effectiveness of CCAC has been demonstrated through evaluations on the DSRL benchmark and extensive ablation studies.", "questions": "See weaknesses for details.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept of CCAC is intuitively straightforward and easy to implement.\n\n2. This paper is well-written, clearly highlighting the challenges, motivations and contributions.\n\n3. CCAC achieves SOTA performance in several DSRL envs.\n\n4. The ablation studies are comprehensive, clearly demonstrating the advantages of CCAC over other baseline methods. Notably, CCAC exhibits constraint-aware behavior that guarantees strong performance (high rewards and low costs below the constraint limit). Furthermore, CCAC demonstrates robustness across varying quantities of data.", "weaknesses": "Several minor weaknesses are present in the study. Addressing these concerns could significantly improve my evaluation of the manuscript:\n\n1. Limited evaluation environments. The DSRL benchmark offers over 30 environments, but this paper only conducts experiments on solely 9 envs. Including more scenarios can substantially enhance the paper quality.\n\n2. Some assumption. CCAC utilizes the trained CVAE to augment the dataset and annotates these data with their original cost/reward values. While this approach is acceptable, it may introduce some errors if the reward/cost functions are not smooth across the spatial space, which then will introduce some annotating errors.\n\n3. The overconservative behavior of FISOR is primarily attributed to the use of hard constraint, rather than its use of the largest feasible region as claimed by the authors in line 323. In fact, using the largest feasible region will include more actionable actions for policy optimization and so can reduce the overconservatism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)