Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
Authors: Prajwal Koirala
Zhanhong Jiang
Soumik Sarkar
Cody Fleming
Venue: ICLR 2025 Poster (2025)
Keywords: Safe RL
Offline RL
Variational Autoencoders
Latent Safety Constraints
Primary area: reinforcement learning
TL;DR: This study proposes learning safety constraints in the latent space for safe offline RL, simplifying constraint imposition within the Constrained MDP framework.

Abstract: In safe offline reinforcement learning, the objective is to develop a policy that maximizes cumulative rewards while strictly adhering to safety constraints, utilizing only offline data. Traditional methods often face difficulties in balancing these constraints, leading to either diminished performance or increased safety risks. We address these issues with a novel approach that begins by learning a conservatively safe policy through the use of Conditional Variational Autoencoders, which model the latent safety constraints. Subsequently, we frame this as a Constrained Reward-Return Maximization problem, wherein the policy aims to optimize rewards while complying with the inferred latent safety constraints. This is achieved by training an encoder with a reward-Advantage Weighted Regression objective within the latent constraint space. Our methodology is supported by theoretical analysis, including bounds on policy performance and sample complexity. Extensive empirical evaluation on benchmark datasets, including challenging autonomous driving scenarios, demonstrates that our approach not only maintains safety compliance but also excels in cumulative reward optimization, surpassing existing methods. Additional visualizations provide further insights into the effectiveness and underlying mechanisms of our approach.

Published as a conference paper at ICLR 2025
LATENT SAFETY-CONSTRAINE POLICY APPROACH FOR SAFE OFFLINE REINFORCEMENT LEARNING
Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar & Cody Fleming Iowa State University
Ames, Iowa, USA jwra, zh ang, soumiss, fleminge ) clastate.ed
ABSTRACT
In safe offline reinforcement learning (RL), the objective is develop a policy that maximizes cumulative rewards while strictly adhering to safety constraints, utiliz- ing only offline data. Traditional methods often face difficulties in balancing these constraints, leading to either diminished performance or increased safety risks. We address these issues with novel approach that begins by learning conservatively safe policy through the use of Conditional Variational Autoencoders, which model the latent safety constraints. Subsequently, we frame this as a Constrained Reward- Return Maximization problem, wherein the policy aims to optimize rewards while complying with the inferred latent safety constraints. This is achieved by training an encoder with a reward-Advantage Weighted Regression objective within the latent constraint space. Our methodology is supported by theoretical analysis, including bounds on policy performance and sample complexity. Extensive empiri- cal evaluation on benchmark datasets, including challenging autonomous driving scenarios, demonstrates that our approach not only maintains safety compliance but also excels in cumulative reward optimization, surpassing existing methods. Addi- tional visualizations provide further insights into the effectiveness and underlying mechanisms of our approach. The code is available (here
INTRODUCTION
Although Reinforcement learning (RL) popular approach for decision-making and control applications across various domains, its deployment in industrial contexts is limited by safcty concerns during the training phase. traditional online RI, agents learn optimal policies through trial and error, interacting with their environments to maximize cumulative rewards. This process inherently involves exploration, which can lead to the agent encountering unsafe states and/or taking unsafe actions, posing substantial risks in industrial applications such autonomous driving, robotics, and manufacturing systems (Gareıa & Fernández 2015) Gu et al. 2022] Moldovan Abbeel 2012 Shen al. 2014 Yang et al. 2020p. The primary challenge lies in ensuring that the agent's 's learning process does not compromise safety, as failures during training can result in costly damages, operational
disruptions, even endanger human lives (Achiam et al. [017]] Stooke et al.) (020)) To address these challenges, researchers have explored several approaches aimed at minimizing safety risks while maintaining the efficacy of RL algorithms.
One effective method to mitigate safety risks associated with training an agent is offline RL. In this paradigm, the focus shifts from active interaction with the environment to learning policies from static dataset. This dataset comprises trajectory rollouts generated by an arbitrary behavior policy or multiple policies, collected beforehand. By leveraging this fixed dataset, offline RL eliminates the need for real-time data collection, thereby significantly reducing the risk of actually encountering unsafe states during the learning process. Training an agent with Offline RI., however, presents unique set of challenges, primarily due to the issue of distribution shift (Levine et al.) 2020 Tarasov ct al. 2024 Fu et al.) 2020)). The static dataset may not fully represent the range of scenarios the agent will encounter in the real world, leading to potential mismatches between the training data and the learned policy. This discrepancy can result in suboptimal policy performance when deployed in real-world settings. Despite these challenges, offline RL remains powerful tool for safely training RL agents, as allows for policy evaluation and improvement without incurring the risks associated with live interactions.

---

Published as a conference paper at ICLR 2025
Another critical approach mitigating safety risks in RL is safe RL. Unlike traditional RL, where the primary objective is to maximize cumulative rewards, safe RL places a strong emphasis on producing actions that adhere to predefined safety constraints (Xu et al. 2022b] (022b]Chow et al., (018). This involves integrating safety considerations directly into the RL framework, ensuring that the agent's s behavior remains within acceptable safety bounds throughout the training and deployment phases. Safe RL. methods often formulate the problem as Constrained Markov Decision Process (CMDP), where the agent not only seeks optimize its performance but also to satisfy safety constraints (Altman 1998, 2021; Chow et al. 2018). This dual objective can be challenging to achieve, as it requires balancing the exploration needed for learning with the strict adherence to safety requirements.
In safe offline RL, the agent is tasked with learning a policy exclusively from pre-collected data while adhering to stringent safety constraints (Liu et al., 2023a Le et al. 2019). is common practice-and and often necessary-to constrain the policy such that it selects actions not only within the support of the dataset but also in compliance with the safety constraints (Xu et al. 2022a). However, solving this problem is inherently difficult due to the trade-off's involved in enforcing constraints. Overly restrictive constraints can limit the agent's ability to explore potentially rewarding actions, leading to suboptimal policies that fail to optimize for cumulative rewards. On the other hand, overly relaxed constraints may allow the selection of of violating safety constraints and leading hazardous outcomes.
(OOD) actions, increasing the risk
To address these challenges, we introduce Latent -rior--Priorttizee Constraints (LSPC), a framework that leverages Conditional Variational Autoencoders (CVAEs) model the distribution of safety constraints within a latent space. This allows the agent to operate within learned safety-prioritized boundary while maintaining sufficient flexibility for reward optimization. By incorporating implicit Q-learning, our approach formulates the policy learning problem as a Constrained Reward-Retur Maximization task, where the agent seeks to maximize cumulative rewards while adhering to the inferred safety constraints. This principled method ensures that the learned policy remains safe, even in offline settings where exploration is not possible. Our empirical results on standard safe offline RI. benchmarks demonstrate that LSPC achieves superior performance compared to existing approaches, balancing safety and reward return in a manner that is suitable for deployment in critical, high-stakes environments. The key contributions of this paper are follows:
We propose LSPC, a novel framework to model safety constraints derived from the static dataset in tractable and imposable form, thereby facilitating their integration into the policy learning process.
We formulate the problem as Constrained Reward- Maximization, ensuring safety compliance while maximizing cumulative rewards, supported by theoretical bounds on policy performance and sample complexity.
Through extensive empirical evaluations, we demonstrate that policies derived from LSPC significantly outperform existing methods in both safety and reward optimization, providing a robust solution for high-stakes environments.
2 PRELIMINARIES
2.1 SAFE OFFLINE RL
In Safe Reinforcement Learning (Safe RL) problems, the environment is defined as a Constrained Markov Decision Process (CMDP), represented by the tuple M AS, C, 7, pa). Here, S represents the state space, AA the action space, P S A x S - [0, 1] the state transition probability function, r S x AA- + R the reward function, c S x A - [0, C..] the cost function associated with constraint violations, ? the discount factor, and po the initial state distribution. Cm is the maximum value for each immediate cost. We also assume that the maximum value of the immediate reward is Rm, The cost function penalizes transitions that violate safety constraints, and the objective is to ensure safety by keeping the cumulative cost under predefined threshold,
trajectory T = (So: ag, ru, cu), (81, Q1, d1,71, (e.71.c)) cI...., (8I, ar, TT, cr) represents a sequence of states, actions, rewards, and costs over time. The discounted cumulative reward for a trajectory is defined as R(r) = Σt r"((s,, at), and the discounted cumulative cost is C(r) IT 7'((st, at). We then also define the stationary state-action distribution under the policy as (s, = (1
7) Et 7'p(st 8, a, = a) and the stationary state distribution under the policy as d= (s)
2

---

Published as a conference paper at ICLR 2025
State
Encoder
Latent Safety Constraint
Decuder
Safe Action
Action
Minimise V."
Safety constreint latent space
State
Encoder (4)
Maximize V,"
Action
Behavior policy distribution
(,(s,0)
(a) Illustration of the architecture and training objective of (b) Visual representation action selection proposed safe offline RL framework.
within the latent safety-constrainet space.
Figure Overview of the Proposed Method: A safe offline RL framework that balances safety and reward optimization. (a) The state-action pairs encoded into latent space by encoder a, where the latent safety constraints are inferred. Decoder 33 then uses these constraints to generate safe actions. Simultaneously, another encoder 5 maximizes the reward signal to find actions that optimize performance while adhering to safety constraints. The process balances between minimizing the violation of safety constraints V and maximizing reward returns V (b) Actions from the latent space are subject to additional restriction to ensure they meet safety requirements before being optimized for reward return. The Q-function (, a) represents the expected reward return for each safe action for that state.
E = s), where p signifies the probability. The goal in safe RL is to learn a policy that maximizes the expected return in the CMDP while keeping the expected cost return below allowable threshold. Thus, the safe RL problem is mathematically defined as follows:
max R [R())), s.t. [xxx(((TT] S K
Offline Reinforcement Learning is a variant of RL in which the learning process is confined to static dataset, eliminating any further interaction with the environment during training. The agent learns the policy from a pre-collected dataset D == (s, a, s', c), with both safe and unsafe trajectories. The value functions for both reward and cost can be defined in unified way V, (sp) wx E ? he/s: = so]. h E {r, cr. Denote by Tr the unknown behavior policy induced by the dataset D. We can now formulate safe offline RL as optimization problem within the framework of a CMDP:
max V (s), s.t., V2) S DKL(T||TL) E1.
where Ta governs the action distribution in the given dataset, and ει is a divergence tolerance parameter. The function DxI(=I[Ta) measures the divergence between the learned policy æ and the behavior policy ne. typically using metrics such as Kullback-Leibler (KL) divergence other statistical distance measures. These constraints ensure that the learned policy т remains within the safety limits defined by the cost threshold and does not deviate significantly from the behavior policy ne thereby mitigating the risks associated with out-o--distribution actions.
2.2
CONDITIONAL VARIATIONAL AUTOENCODER
Conditional Variational Autoencoder (CVAE) generative model that extends the standard Variational Autoencoder ( (VAE) by conditioning the generation process on additional information (Doersch 2016). Formally, the objective of the CVAE is maximize the conditional likelihood po(xl) of the data given the condition To achieve this, CVAE introduces a latent variable and optimizes the variational evidence lower bound (ELBO) on the conditional log-likell likelihood:
log p(zly) 2 Eg(=+rz.) [log 2. y)] K (g(2|2, 3) ())((())))
where q(2(x, 3) the variational posterior, p(x]2, ,, y) is the likelihood, and p(zly) is the prior. The KL divergence term Dx Dxı(g(z|z, v) (p((bl)) regularizes the learned posterior to be close the prior. During training, the CVAE jointly optimizes the encoder and decoder networks by minimizing the negative ELBO, which balances the reconstruction error and the regularization term. This approach allows the CVAE to generate new data samples (x) from the distribution of the dataset conditioned on the variable y.
3

---

Published as a conference paper at ICLR 2025
2.3 IMPLICIT Q LEARNING
Offline Reinforcement Learning (RL) algorithms often address covariate distribution shift by em- ploying various strategies, such as using regularization methods to constrain the policy and/or the critic. In some cases, they avoid training the model outside the support of the distribution of the dataset altogether. Implicit Q-learning (IQL) incorporates an additional state-value network to prevent querying out-o--distribution actions during the training of the Q-network (Kostrikov et al. 2021b). Notably, training IQL does not require sampling actions from a policy but can be performed solely using actions from the dataset. The typical losses for IQL are follows:
(0) ( (r + -
(4) The IQL framework involves training value network, denoted as Vo. in addition to the Q-network, denoted as Qo- The loss functions associated with training these critic networks are detailed in Eqs. 3 and 4 State value network (V₂) is trained with expectile regression objective and uses an asymmetric squared error loss function defined L (u) = 15 1(u < 0) 1²² where & E (0.5, 1.0). With both Q and value networks in place, the policy can be trained or extracted using the advantage- weighted regression (AWR) method (Kostrikov et al. 2021 Peng al. 2019). This method uses the learned advantage function, which is derived from the Q and value networks, and is defined as A(s, a) = Q0 a) (+), to guide the policy training process effectively within the distributional constraints of the dataset.
(d) ((E,))<D Lg (Qo VV()))]
IQL in the context of offline RL is focused on learning reward-value from the data and using reward-advantage weighted regression to extract policy that predicts actions with higher expected reward-retum. similar method can be used learn the volt- from the offline data. We use the superscript to denote the cost and define the cost-ad cost-advantage of an action taken at state s as (s, a) = Q a) Vş(()) A similar asymmetric loss can be used to learn the cast-v of a state to discourage the underestimation of Qā(s, a):
(V) = (s.. 2ND (e + (V,('') (+()))))
(5)
(n) = (o,a)/D [L (v)(s) (+))a))
METHODOLOGY
In this section, we detail the formulation of the proposed Latent Safety Prioritized Constraints (LSPC), which we employ to formulate the constrained reward optimization problem. As depicted in figure I the framework consists of two key components: (1) the derivation of conservatively safe policy with Conditional Variational Autoencoder (CVAE), and (2) the maximization of reward returns subject to the safety constraints inferred within the latent space.
As will see, the CVAB-base architecture encodes state-action pairs into a latent representation that prioritizes adherence to safety constraints while being trained to reconstruct safe actions. Moreover, it also serves as parametric generative model to implicitly enforce the policy output actions within the support of the dataset. By selecting the action most likely in the dataset D, the generative model helps minimize errors in value (both cost and reward) approximation during policy evaluation, (OOD) state-action pairs (Fujimoto et al.) 2019; Zhou et al. 202T). This obviates the need of explicit constraint on the divergence between the learned policy (T) and behavior policy (TT))
which often arise in offline RL due to out-of-distribun
Thus, the CVAE plays an important role in enforcing the two constraints outlined in Equation 2 ensuring that the policy operates within the boundaries of safe behavior as defined within the dataset. The latent space inferred through this training encapsulates these constraints in a more tractable form derived solely from a static, cost-labeled dataset. For these reasons, the constraints imposed via the latent space in our method are referred to as Latent Safett-Prioritizer Constraints (LSPC). Below, we derive two specific policies: LSPC-S, conservative policy trained solely to predict actions within these safety constraints, and LSPC-O, which is optimized to predict high reward- value actions while still adhering these constraints.
4

---

Published as a conference paper at ICLR 2025
3.1 LEARNING CONSERVATIVERS SAFE POLICY
Behavior Policy Modeling. To derive a safe policy n. (a]s) (the subscript in n, is short for safe), we start by employing a conditional variational autoencoder (CVAE) to model the underlying behavior policy (a s) in the dataset. The objective of the CVAE is to maximize the likelihood log To s) by maximizing a variational lower bound formulated as:
max log (a s) 2 max Ezzg, [log (a s, 2)] DxL [9a(2 | s,a) || p(z s. s.a)]
(7)
a,
o.,
where is the latent variable, and 3 are the parameters of the encoder and the decoder, respectively. trained encoder (x 8, a) encodes the state-action pair to a probability distribution in the latent space. A trained decoder pa 8, provides a mapping from the latent space to the action space, conditioned on the state. When the latent variable takes on values that are likely under the prior p(z 8, a), which is modeled expected to generate actions that align closely to the behavior policy distribution Tz(a 8)..
standard normal distribution (0, 1), the decoder pa 8, 8,22 is
Safe Policy Derivation. We define safe policy as one that minimizes the expected cumulative discounted cost return over the course of interactions with the environment:
T
T arg min E, Σ 7c((s) at). a ~ x(. |s)]]
(8)
In off-policy actor-critic methods, safe policy extraction can be achieved using advantage-weighted regression (AWR). This approach reformulates the problem to maximize the expected log likelihood of actions weighted by their advantage over the cost. The safe policy म. can be derived as follows:
ns arg max ())DD) [exp (> (V; Q (s, a))) log ((a s)]
where E [0, 00) is a hyperparameter in AWR called inverse temperature. This formulation is equivalent to maximizing the log likelihood (To(a I s)) of an action given state, with weights assigned according to the advantage, thereby promoting actions that are expected to minimize the cumulative cost. In the CVAE framework, parameters of the safe policy can be derived as a", 3 = arg mina,9 (a, 3), where
L 3) = B(E,0)~D [- exp (A (V,((s) (+(,, a))) (Ez=a. [logpa(a [,,)]]
Drz [y. (z a,a) p[= | s,))]] (10)
The CVAE is trained to reconstruct safe actions conditioned on given states and actions from the static dataset, effectively capturing the distribution of safe policy.
(a s) f
ps(a z)p(= s, s, dz (p(s s,
(I))
For the values of that have a high probability under the prior p(z s, a), the decoder pa(a s, 2) is expected to generate actions with high likelihood under the policy distribution T. (a s). Thus, in our framework we define mean and unit variance, ensuring that the sampled actions align with the safe policy distribution in expectation. The latent safety constraint serves probabilistic embedding that make the safety constraints present in the static dataset more tractable. When sampling z from the full prior A (( 1). we refer to the resulting policy as the CVAE policy, which uses the decoder to generate 'safe' actions conditioned on the state. In practice, we restrict the sampling of to a high- high-probability region of the prior, leading to a safer policy due to this restriction. We refer to the resulting policy as LSPC-S, as prioritizes safety during execution.
LSPC constraint, whose distribution is modeled as Gaussian with zero
3.2 CONSTRAINED REWARD- RETURN MAXIMIZATION
The objective of this constrained optimization problem is to solve for a policy a that selects action with the maximum Qr value while also respecting the latent safety constraint.
(s)) arg max (s, a), a ps s, 2), ~ 1))
(12)
To achieve this, we train an additional encoder pa(z s), which we refer to latent safety encoder policy conditioned on the state and it learns the distribution of safety embedding such that the expected reward-return is maximized.
A (z s) arg max Qr ~ pa(- |s 2), (a(1
(13)
y Advantage Weighted Regression in Latent Space. To learn the reward-maximizing safety em- beddings, we train the encoder [=[= 8) using reward advantage-weightted regression (AWR). The
5

---

Published as a conference paper at ICLR 2025
optimization objective is given by
( TE (s.o)>/ [- exp (5 (Q)() -s)( log Here (is the inverse temperature hyperparameter for the above AWR. While this objective focuses on maximizing the reward-return, it is equally essential enforce the latent safety constraint to certify policy safety. To ensure that the embeddings produced by the encoder fá lie within the probabbibty regions of the latent safety space, we employ a tanh squashing function, which restricts the outputs to the range (-e,e), >> O. The resulting policy, which we refer to as LSPC-O, optimizes reward-return within the restricted latent safery constraints space. The LSPC-O policy is parameterized by latent safety encoder policy parameter ò and CVAE decoder parameter 3. During the gradient step to train the latent safety encoder policy (16)) the CVAE decoder parameter is kept frozen, but the gradients are allowed to pass through.
8, 8, , ps(( 1s)
(14)
An overview of the training process showing each gradient step is provided in Algorithm is important to note that the policy extraction steps for both the CVAE policy and the latent safety encoder policy do not affect the value function training defined in section 2.3] and therefore the policy training can be done concur- rently with the critics in Actor- Critic framework or after cemporal-differnnee (TD) training of the critics. Detailed implementation specifics can be found in Appendix A.3
Algorithm 1: LSPC Training
[
Initialize: o, 8, n, b a, 3, for each gradient step do
TD learning with IQL:
é f- á 1/ VaLv
0 0 14 VaLo
1J + 1 - 1"',, V f 1o Vy V (16)
Policy extraction with AWR:
{a, B} {a. Vos V Co.
8 8 - 1s VoC
THEORETICAL ANALYSIS
We provide theoretical analysis for LSPC-O as the safe optimal policy is our target, while similar techniques can be used for LSPC-S. In what follows, we begin by imposing assumptions regarding the distances between various policies. These assumptions will allow us to derive performance bounds and sample complexity guarantees for learned policies, ensuring both safety and reward maximization. The metrics for evaluation are based on Eq. 2 such that we adopt V.- (Po) V," (po) and (Po) - K. In light of the definition for the stationary state- action distribution and the fact that T = argmax, , E.-r ===
E/=1n' rrr(st, 0t)], we have πt = argmax, Segmaa,E(s.aa-odd [r(s, a).. This will assist
in the analysis presented later. All necessary proof is deferred to the Appendix A.
Assumption Suppose that the policy n is optimized by Eq. 2 and that the safe policy n, induced by CVAE. We have DKL(m [TT]] S E'-
Assumption 2. Suppose that the policy x² is the optimal policy and that the safe policy T. is induced by CVAE. We have DEL (T, [T][[T") S E
In Yao et al. (2024). the authors resorted to similar assumptions involving Ty instead iTs (see Appendix A for detail) to ensure behavior regularization and establish constraint violation bound. Immediately, we can obtain that the upper bounds for D (πl|m)) and D D (TD |TT)) are looser, since intuitively, n, should be closer to and n² than mg. One may argue that when resort to n, in the above assumptions, whether the behavior regularization can still be satisfied. This is affirmatively true as n, is safe reconstruction of Te with the generative property of the CVAE. The designed loss also enables the appropriate latent safety embedding construction for training т. Alternatively, some pre-processing can be done to D such that only the safe data samples are utilized learn the policy, as done in BC-Safe Liu et al. (2023a). This may be more conservative to guarantee the safety constraint, but with the compromise of optimality, as an unsafe sample could lead to the higher returns. Therefore, we select the first approach for training in our proposed pipeline. We next present
a lemma show the error propagation.
Lemma Suppose that the stationary state distributions for n and n² are defined as d= d=((s) and d= (s). The following relationship holds
Drv(d" (s)]ld" (s)) S T E. Banar' (
7 -
(15)
where DrV is the total variation distance.

---

Published as a conference paper at ICLR 2025
The proof of Lemma follows similarly from Lemma 2 in Xu et al. (2020). This lemma quantifies the relationship between the stationary state distribution discrepancy w.r.t. the policy distribution discrepancy, which is critical to tighten the performance bound presented in the next.
Lemma 2. Define V," (po) == Bzoupl [V,(ss) given policy x. Then, we have the following:
28m
[V7 - Vē" (po)) - 71- E Bendr (o) (+ss)]
(16)
Different from existing works Cen et al. (2024); Yao et al. (2024) where the authors bounded Drv [s)]ld" (s)) by Drv (d (s) ddd (s)). thanks to Lemma Lemma Sttates that the perfor- mance gap now stems only from the policy distribution discrepancy between и and '² and eventually decays if the number of samples in D is infinite in our sample complexity analysis. Thus, we next
states
upper bound E (6) (r) ( (-1s) (=r (-1s))]
Lemma 3. Let Assumptions 1 and hold. We have the following relationship:
Ea (a) (rv (+1s)) S V V /
(17)
Hence, the first main result to reveal the performance gap between any policy a optimized by Eq. 2 and the optimal one x¹ is as follows.
Theorem 1. Let A hold. For the policy π optimized by Eq. 2 with the dataset D, the performance gup between and nt' can be bounded as
2Pm
V (po) - V S (1 (1- (on22(ee/ (2/2/2+ /2)
(18)
Theorem I Jsugessts that the performance gap is attributed to the policy distribution discrepancies. This bound is the worst-case bound irrespective of the number of samples in D. However, based on traditional supervised learning, more data (which requires a weak assumption) should reduce the error such that π is closer to π²" We will investigate this later and show it is indeed the case. In this context, we analyze the constraint violation bound. Due to 2 the safety constraint dictated by the threshold value k. Nevertheless, as the policy learning proceeds, such constraint may or may not be violated, particularly at the early phase of learning. We aim at deriving the similar worst-case bound.
Theorem 2. Let Assumptions/ D and holld For the policy optimized by Eq. 2 with the datusel DD the constraint violation of x. can be bounded as
20m (1-
(1-72(/et Ve//2+ Ves/2).
Vo
S
(19)
We notice that both Theorem 7 and Theorem imply that the error bounds for the reward and the cost incurred during learning are W.E. (1-2}² V2,/2+ + VEg/2). differing in constant. This intuitively makes sense
Remark 1.
have adopted the same IQL to train both critic networks. The dependence on (1 (1-2)a is necessarily inevitable due to the error propagation from the policy distribution discrepancy to the state distribution discrepancy, which has theoretically been justified in Xu et al. (2020) and Schulman (2015). So far, thanks Assumption 7 and Assumption 2 the performance gap and the constraint violation are upper bounded. However, would increasing the size of 'D reduce them? ? Particularly, when we have a sufficiently large amount of pre-collected data, can this enforce V (s) S k. to hold strictly? Inspired by the theory of empirical process Shorack & Wellner (eelner(200p) (2009) we provide the answers to the above questions and the resulting sample complexity in Appendix|A.2
Remark 2. Another remark is also provided in this contexl for the connection between the proposed method and theoretical results. In LSPC, we leverage CVAE to reconstruct safe policies residing in the offtine dataset D, strategically ensuring that the further optimal solutions are safe. This justifies Assumption
Assumption 2 that provide new upper bounds for the distribution distances between π and Ws. and n. and π². Previously, instead of n.. it was the unknown behavior policy To Yao et al. (2024), which has less assurance of safety than n. and results in relatively looser bounds. Moreover, in LSPC, we adopt the IQL and AWR for critic model updates and policy extraction, which motivates us to use the value function as the metric to evaluate the proposed algorithm. As the policy is learned via static offline dataset, the performance gap and constraint violation are assessed based on the stationary distributions, instead of a dynamic metric such as regret Zhong & Zhang 2024). Theorem I and Theorem 2 suggest the performance gap and constraint violation during
7

---

Published as a conference paper at ICLR 2025
learning are upper bounded by constants that are correlated with the new distribution distance bounds induced by the safery policy Rg. Additionally, as the policy evaluation and extraction steps are all parametric with deep learning models, the number of data samples plays central role in controlling the performance, which has been a well-established fact in modern deep learning community. In offline RI,, this motivates the need to assess performance gap and constraint violations with respect to the size of D. Thereby, Theorem and Theorem { (In Appendix (.22) imply the decay rate with respect to the size of D. Though the analysis can apply related algorithms with some adaptation, the constants in these results, particularty E' and E22 are uniquely defined and thus the theoretical conclusions are tailored the proposed algorithm.
RESULTS AND DISCUSSIONS
Datasets and Metrics. This section presents a thorough evaluation of our method within the context of safe offiin reinforcement learning. Our evaluation uses the DSRL benchmark (Liu al. 2023a focusing on normalized return and normalized cost to measure performance. The normalized reward return is given by R (R, Rmsnn) accumulated in an episode, and Rmaz and Rmin are constant for a task and represent the maximum and minimum empirical reward return. Similarly, the normalized cost return is given by C C* ///,, where > 0 is the targeted cost threshold. The evaluation spans actoss tasks from Metadrive (Li et al. 2022). Safety Gymnasium (Ji et al., 2023) and Bullet Safety Gym (Gronauer, 2022). More about these tasks and benchmark can be found in appendix (..ddixAA.4aaaa and et al. (2023a). Following the DSRL constraint variation evaluation, each method is evaluated on each dataset with three distinct target cost thresholds and across three random seeds. Although our method is agnostic to cost threshold variation, we adhere to the same evaluation criteria for consistency. A normalized cost return below 1 indicates adherence safety requirements, with safety being the primary performance criterion. The results in Table I use boldface to represent safe agents with normalized costs smaller than I and blue color to highlight safe agent(s) with the highest reward return in the dataset. These results demonstrate how our methods not only adhere to safety constraints but also optimize rewards.
R, where R, is the undiscounted total reward
Table 1: Comparison of our methods with baselines across benchmark tasks. Bold indicates safety, and blue denotes both safety and high performance.
Method
BC-Safe
CDT
CPQ
FISOR
LSPCS
I.PC--
Task
reward
cost
roward
cost
roward
008t
roward 1
cost
reward 7
cost
reward 7 cost
Metadrive:
EasySpurse
0.17
0.23
0.877
0.38
0.15
0.62
0.71
0.46
EasyMean
0.04
0.29
0.45
0.54
-0.07
0.07
0.38
0.08
0.62
0.04
0.69
0.26
Dasseense
0.11
0.14
0.32
0.62
-0.06
0.03
0.36
0.08
0.55
0.06
0.68
0.37
MediummSprrs
0.33
0.30
0.87
1.10
-0.08
0.07
0.42
0.07
0.96
0.32
0.94
0.12
MediumMean
0.31
0.21
0.45
0.75
-0.08
0.05
0.39
0.02
0.85
0.43
0.94
0.11
MediumDense
HardSparse
0.17
3,25
0.25
0.41
-0.05
0.06
0.30
0,00
0.50
0.24
0.54
0.47
HardMean
0.13
0.40
0.33
0.97
-0.05
0.06
0.26
0.09
0.51
0.21
0.53
0.57
HardDense
0.15
0.22
0.08
0.21
-0.04
0.08
0.30
0.10
0.47
0.08
0.50
0.23
Average
0.18
0.58
0.42
0.80
-0.06
0.06
0.36
0,08
0.67
0.17
0.72
0.29
Safety Gym:
Carbuston!
0.07
0.85
0.21
1.6
0.42
9.86
-0.02
0.04
-1.02
0.14
-0.01
0.11
CarButton2
0.39
CarGoall
0.24
0.28
0.66
1.21
0.79
1.42
0.49
0.12
0.22
0.23
0.31
0.40
CarGoal2
0.14
0.51
0.48
1.25
0.65
3.75
0.06
0.05
0.13
0.44
0.19
0.42
CarPushl
0.95
0.04
0.18
0.32
0.18
0.33
CarPush2
0.05
0.45
0.19
1.3
0.24
4,25
0.14
0.13
0.012
0.34
0.055
0.62
SwimmerNal
HopperVel
0.36
0.67
0.63
0.61
0.14
2.1
0.17
0.32
0.26
0.39
0.69
0.00
HallCheetahNel
0.88
0.54
1.0
0.01
0.29
0.74
0.89
0.00
0.79
0.01
0.97
0.10
Walker2dVel
0.21
0.36
0.56
1.28
0.76
0.02
AutVel
0.98
0.29
0.98
0.39
-1.01
0.0
0.89
0.00
0.95
0.07
0.98
0..45
Average
0.27
Bullet Safety Gym:
HallRun
0.27
1.46
0.39
T.16
0.22
1.27
0.18
0.00
0,08
0.001
0.14
(.00
CarRun
0.22
0.99
0.65
0.95
1.79
0.73
0.04
0.72
0.00
0.97
0.13
DioneRun
0.28
0.74
0.63
0.79
0.33
3.52
0.30
0.16
0.54
0.00
0.57
0.00
AntRun
0.45
0.00
0.29
0.04
0.44
0.45
HallCirrae
0.52
0.65
0.77
1.07
0.64
0.76
0.34
0.00
0.27
0.28
0.47
0.01
CarCircle
0.84
0.75
0.95
0.71
0.33
0.40
0,03
0.35
0.00
0.72
0.04
DroneCirele
0.60
0.98
-0.22
1,28
0.48
0.00
0.16
0.00
0.58
0.60
AntCircle
0.40
0.96
0.54
1.78
0.00
0.00
0.20
0.00
0.13
0.00
0.44
0.40
Average
Discussion Empirical Results. In the comparative analysis presented in Table proposed method, LSPC-O, demonstrates a significant performance improvement over existing baselines. Among these, FISOR demonstrates consistency in upholding safety even under strict cost thresholds,
8

---

Published as a conference paper at ICLR 2025
as it imposes hard constraints in the RL problem formulation. However, this approach leads to excessively conservative policies, significantly limiting reward-return performance. Another method CDT conditions the policy on target reward and cost returns. In practice, this method is often unreliable because it is challenging to satisfy both conditions simultaneously. Although the authors suggest improvements, our empirical results show frequent safety violations, especially under smaller cost thresholds. Additionally, interpreting results across different prompt conditions is difficult, making prompt selection a non-trivial task in CDT implementation. BC-Safe applies behavior cloning to safe subset of the dataset, but it lacks consistent safety and offers no empirical guarantees. Its primary limitations are the need for sufficient quantity of 'safe" data and retraining with the filtered data when the cost threshold is changed. Despite being agnostic to the reward labels in the dataset, this method often outperforms FISOR and CDT in performance, as evident in the [
Our method effectively addresses the limitations observed in these baselines. Unlike SafeBC, we employ weighted regression to derive safe policy, which theoretically only relies on a much weaker prerequisite the explicit presence of the safe data points (Assumption 6). while also ensuring policy improvement using the reward-labels. Unlike CDT, our method disentangles safety and reward objectives, providing reliable safety guarantees across different tasks and cost thresholds. Additionally, we derive a safe policy in our methodology that can be utilized in safety-criticn operations, where constraint satisfaction is prioritized over reward maximization. Finally, compared to FISOR, the conservativeness in our methods can be controlled by adjusting the degree of restriction in the latent safety constraint space, yiclding better reward performance while maintaining safety.
Reward Return
1000
Return
200
Reward Return
400
Cost Return
100
500
100
200
50
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Training Step
1e6
Training Stop 1e6
Training Step 1e6
Training Step 1e6
(a) Pybullet Car Run
(b) Metadrive Easy Sparse
Figure 2: Training progress illustrating how the reward return and cost return evolve over the course of training, highlighting the agent's accumulated reward and adherence to safety constraints.
The graphs in Figure illustrate the evolution of both reward return and cost return training progresses, comparing how various algorithms perform in adhering to safety constraints while optimizing for reward across two tasks: Pybullet Car Run and Metadrive Easy Sparse. Our method, LSPC-S, demonstrates a conservative approach with a strong emphasis on safety, while LSPC-O achieves favorable balance between reward maximization and cost minimization, consistently outperforming baseline methods. This is particularly evident in its lower average episode costs as training progresses. In contrast, FISOR fails to outperform even the conservative LSPC-S in both tasks. Behavior Cloning with safe trajectories (BC-Safe) performs reasonably well in the Car Run task, but shows poor performance in Metadrive Easy Sparse, struggling with both reward accumulation and safety adherence. CPQ achieves higher reward in Car Run, though at the cost of increased episode costs near the end training. However, in the Metadrive task, CPQ's performance is trivially safe, with near-zero reward accumulation.
Figure 3 illustrates how the learned policies operate in the action space for the Pybullet Car Run and Metadrive Easy Sparse tasks. The plots on the left show kernel density estimates for three policies: (I) the CVAE policy, (2) the safe policy after applying Latent Safety Prioritized Constraints (LSPC), and (3) the optimal policy within the restricted action space defined by LSPC. The right plots display scatter plots of actions sampled from the CVAE policy, convex hull estimates of actions sampled from the safe policy, and the optimal action predicted by LSPC-O, the reward-optimized policy. The color map represents the reward values (r a) of the sampled actions for the given state. This representation underscores the ability of our method to prioritize safety while maximizing reward, as the policy concentrates on regions of high Q within the restricted latent space.
6 RELATED WORKS
Offline RL has demonstrated success in domains such as robotics, healthcare, and recommendation systems (Tarasov et al., 2024 Prudencio et al.) [023 Lange et al. 2012). However, the absence of online interactions can lead to outtof-disttrbution (OOD) actions that overestimate the value function

---

Published as a conference paper at ICLR 2025
1.0
1.00
224
Steering Angle
0.5
0.75
222
Throttle
0.0
220
0.25
-0.5
216
0.00
LSPC-S
LSPC-0
-1.0
Qr
0.25
CVNE
-1.0
-0.5
-1.0
-0.5
0.0
0.01
0.01
Velocity
Steering Angle
(a) Pybullet Car Run
(b) Metadrive Easy Sparse
Figure 3: Action-space distributions for (a) Pybullet Car Run and (b) Metadrive Easy Sparse. Kernel density estimates (left) and convex hulls within the scatter plot (right) visualize the unconstrained CVAE policy, the constrained safe policy (LSPC-S), and the optimal policy (LSPPC-) identified by the latent safety encoder.
due to unseen data (Levine et al. 2020). Solutions include value function regularization (Kumar et al.
2020 2019
Kostrikov et al. [021a), policy constraints to stay near the behavior policy (Fujimoto &
Gu 2021:
Wu | (ujj Fujimoto et al. (019)). and avoidance of OOD actions' evaluations (Chen
ctt al. 2021 Kostrikov et al. 2021b; Janner et al. 2021). Generative models like CVAE (Fujimoto
et al.
2019 Zhou et al. 2021) y and diffusion-based methods (Janner et al. 2022; Wang et al. 2022) have also been applied to sample actions in the latent space, reducing OOD occurrences.
Safe Offline RL aims to ensure safety within offline RL by incorporating techniques that balance policy improvement and safety constraints (Liu et al., 2023a)). Methods such as COptiDICE (Lee) al. 2022) and OASIS (Yao et al. 2024) extend distribution shaping (Lee et al. 202T) to constrained RL settings. Lagrangian-baan based approaches like CPQ (Xu et al.) 2022a), BCQ- Lag, and BEAR-Lag Lag (Liu et al. 2023a) integrate safe policy learning within offline RL methods like CQL (Kumar et al.) 2020). BCQ (Fujimoto et al.) 2019) and BEAR (Kumar et al., 2019). Other approaches, such as VOCE (Guan et al. 2024). use probabilistic inference with non- con-paramttric variational distributions (similar Liu et al. (2022)) ). while models like CDT (Luu et al. 2023b) and Saformer (Zhang et al., 2023) leverage decision- transformer (Chen al., (021) frameworks for cost-prompted sequence modeling. Other generative model-based baselines in safe offline RI. include CPQ, which uses CVAE for behavior policy modeling, and FISOR (Zheng et al. 2024). which employs a diffusion model (Ho et al. 2020) as an actor, trained to select actions based on feasible region identification.
CONCLUSIONS
In this work, we introduced the Latent Lafety-Prioritiiee Constraints (LSPC) framework for safe offline reinforcement learning, addressing the critical challenge of balancing reward maximization with stringent safety constraints. By leveraging Conditional Variational Autoencoders to model the latent safety constraints, enable principled approach to enforce safety while optimizing cumulative rewards within the constraint space. Our method offers a robust solution that avoids the pitfalls of overly restrictive or loose constraints, leading to policies that are both safe and high- performing. Theoretical analysis of policy performance illustrates that the reconstructed safety policy from the unknown behavior policy shrinks the performance gap induced by the value function. The additional sample complexity analysis also reveals the decay rate of the gap w.r.t. the number of samples. Extensive empirical evaluations demonstrate that LSPC-O consistently outperforms existing/recent methods on challenging benchmarks.
Although our proposed methods demonstrate strong theoretical backing and empirical performance, a key limitation may lie in determining how restrictive the latent space for the optimal policy search should be (controlled by e). Future work could focus on deriving theoretical bounds that establish a relationship between this restrictiveness and the cost value, providing both theoretical insights and practical guidelines. From an application perspective, transferring learned policies to real-world robotic tasks via sim2real or sho- fine-tuning is promising avenue. Additionally, exploring safety prioritization dynamics in multi-agent cooperative or adversarial settings offers another exciting direction for both theory and practice.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by the National Science Foundation, USA, under grants NSF CNS- 2313104, CAREER CNS-1845969 and CPS Frontier CNSS-154555
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48:387-417, 1998.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao. Henry Lam, and Ding Zhao. Learning from sparse offline datasets via conservative density estimation. arXiv preprint arXiv:2401.088/9 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Miklós Csörgö, Barbara Szyszkowicz, and Qiying Wu, Donsker's theorem for self-normalized partial sums processes. The Annals of Probability, (1):1122 1240, 2003.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprins arXiv: rrXiv:/606.05900 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pp. 2052-200200000 2062. PMLR, 2019.
Javier Gareia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Sara A Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Sven Gronauer. Bullet-safety- gym:: A framework for constrained reinforcement learning. 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang. Zhijun Li, al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Aria HasanzadeZonuzz, Archana Bura, Dileep Kalathil, and Srinivas Shakkottai. Learning with safety constraints: Sample complexity of reinforcement learning for constrained mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7667-7674, 2021.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1 1266, 2021.

---

Published as a conference paper at ICLR 2025
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999, 2022.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information P Processing Systems Dalusets and Benchmarks Truck, 2023.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini 10.v:22100001169 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-1191, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-arl, pp. 45-73. Springer, 2012.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMLR, 2021.
Jongmin Lee, Cosmin Paduraru, Daniel Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arxiv preprint arXiv:2204.00 089557, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0/66 0064443,, 2020.
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Setting the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52(1):233-260. 2024.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp. 13644-13668. PMLR, 2022.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao. Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu. Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.099 093003 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Con- strained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pp. 2161 21611-. PMLR, 2023b.
Shuangge Ma and Michael R Kosorok. Robust semiparametric m-estimation and the weighted bootstrap. Journal of Multivariate Analysis, 96(1):190-217, 2005.
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. arXiv preprint arxiv: / 420..4810, 2012.
12

---

Published as a conference paper at ICLR 2025
Thanh Nguyen- Tang, Sunil Gupta, Hung Tran-The and Svetha Venkatesh. Sample complexity of offline reinforcement learning with deep relu networks. arXiv preprint arXiv:2103.06677) 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weiighted regrests regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 190.000/71 2019..
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learming Systems, 2023.
John Schulman. Trust region policy optimization. arXiv preprint arXiv: 2015.
Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural compulation, 26(7):1298-1322 2014.
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research. 25(200):1-91, 2024.
Galen R Shorack and Jon A Wellner. Empirical processes with applications statistics. SIAM, 2009.
Adam Stooke, Joshua Achiam, and Picter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 36, 2024.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou, Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061 06199, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu, Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7753-8760, 2022a.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.0800 08025, 2022b.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 13337-1115,,4 2020.
Tsung-Yen Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprini arXiv:2010.03152: 2020.
Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. arXiv preprint arXiv:2407. 200:22407.4453, 2024.
Qin Zhang, Linrui Zhang, Haoran Xu, Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301. 2023.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li. Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with sesiibbilit-guiidee diffusion model. arXiv preprint arXiv:2401. 10700, 2024.
Han Zhong and Tong Zhang A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. Advances in Neural Information Processing Systems, 36, 2024.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, pp. 7719-1735. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
ADDITIONAL ANALYSIS
In this section, provide additional analysis and numerical results support validate proposed algorithm. In what follows, we provide the missing proof for the main theoretical results.
A.1.1 AUXILIARY ASSUMPTIONS
We present the following assumptions for completeness and illustration, though will not use them for characterizing the main results.
Assumption 3. There exists a constant &1 > O such that for the policy optimized by Eq. 2 and the behavior policy Th. they satisfy the following relationship
(KL(T][TT) S E1.
(20)
Assumption 4. There exists constant E2 0 such that the optimal policy 't" and the behavior policy Th, they satisfy the following relationship
D S
(21)
Assumption retains the behavior regularization such that the distribution shift will be significant. While Assumption 4 secures region where и is valid policy. Due to the non-asymptotic manner, x² may not be practically achievable. However, such an optimal policy should not be far away from m Otherwise, the distribution drift will degrade the model performance. These two assumptions have been used in a recent work Yao et al. (2024) to dictate the performance gap. Due to the intermediate safe policy n, we introduce in this work, the performance gap should be narrowed. Since intuitively, T, should be closer to и and π than T6.
A.1.2 MISSING PROOFS IN SECTION
For completeness, restate all lemmas and theorems in this context.
Lemma 2 Define (Po) = "eoopplV"" [V,""ss)) given policy æ. Then, we have the following
relationship:
28m
(P)) - V (po)) S
(1 - 7)2 Eande (T)
(22)
Proof. In light of the stationary state-action distribution, the following relationship is obtained
1
1V7 (Po) V," (pv)l
1 (e (s.a))rss ((a)))a=(
Rm
Σ
(s d" ds
1
(5.0)
2Rm
Drv (s, (s.))
(s,a))
1-7
2Rm
(Drv (d (s, a)lld' (s) ((-ss) Drv (s) (-(s))ld (s, (1)))))""))))
1-7
2mm
Drv (d (s) ldd (s)) + 2R... E Eand PTV (+ss)]
1-7
2Rm
EE
(rV (T (1) (+)ss=
71-7)3
(23)
The first inequality is due to the maximum immediate reward. The second inequality follows from the Triangle inequality. The last inequality is based on Lemma T This completes the proof.
Lemma Let Assumptions and 2 hold. We have the following relationship: b
E Ex-at" (w) (DrV (+|s)) S V
(24)
14

---

Published as a conference paper at ICLR 2025
Proof. Based on the Triangle inequality, the following is attained
Drv (+(())))) (-1s) x (r((((())] + Drv (1
(25)
Thus.
E an (s) Drr (p]])
< Ex (o
3 B (0) DR
Pr ( + Band (0) D
(26)
< v R (x) (1)1) + B Ex-drt
The second inequality follows from the Pinsker's inequality and the third is based on the Jensen's inequality. This completes the proof.
Theorem Let Assumptions ! and hold. For the policy π optimized by Eq, 2 with the dataset D, the performance gap between т and π can be bounded as
2Rm
Vi (po) V, (po)
(27)
(1
Proof. The conclusion is immediately obtained by combining the conclusions from Lemma 2 and Lemma 3
To derive the constraint violation bound, we first need to establish similar conclusion as in Lemma based on the cost value functions V (po) and V (Po)-
Lemma 4. Define Ve (Po) == Eso~po (so)l given a policy n. Then, we have the following relationship:
20m
IV. (po) Ve¹ (po)1 S
Bendd (s) (DTv((((s))) ((1))
(28)
(1 7)2
Proof. The proof follows similarly from that in Lemma
Theorem Let Assumptions D and hold. For the policy π optimized by Eq. with the dataset D, the constraint violation of π can be bounded as
20m S (1 7)2
E
Vi.
V
+
(29)
2
2
Proof. According to Assumption and Assumption 2 have
combining the conclusion from Lemma 4 we
20m
c
[V: VT (po)) S
(V
VV
(30)
(1 - 7)2
Combining the above inequality with the fact that the optimal policy is constraint satisfactory, i.c., V= (po) S K. yields the desirable result.
A.2 ANALYSIS FOR DECAY RATE AND SAMPLE COMPLEXITY
Recalling the following relationship and applying the importance sampling to it, have:
2Rm
|V (po) V. (po)) <
E Eada (o) (TV
(1 7)²
2Rm
d= (8)
E Banntt (s))
Drv (T
(31)
(1 7)2
(
2Rm
<
Eynnrhh (1)) DrV (1)
(1 7)22
15

---

Published as a conference paper at ICLR 2025
The last inequality follows from Ex-d"* (s) [ = 1. The reason why we use To instead of T, has two folds. First, the action of i is not sampled from actions induced by TTs as the extra encoder block in the pipeline also takes input directly from D. Second, directly using d= (s) will yield a relationship about how the performance gap varies along with the size of a dataset only comprising safe samples. However, this is not practically feasible since D can contain samples generated by both safe and unsafe policies. Next, we will derive how (x)dd [Drv (-|ss)][(x (-1s)) evolves with the size of D. With the Pinsker's inequality, D-v(-ll-) can be converted to D-I(-ll-). Hence, to achieve the best performance V. (po), it is equivalent to minimizing (o) [Dxr. (TT())s) [| n² (+1s)] by (s)
using D. Suppose that the size of 'D is N and that the learned policy is parameterized by w E R",
such that the following minimization problem can be obtained:
min Σ (-w (1s)) (-1s:).
(32)
Next, we start with the an assumption for the function of KL-divergence.
Assumption 5. Suppose that the learned policy optimized by Eq. 2 is parameterized by w E RR" and that its parameter space is denoted by W C R Let (s) D (w (+|s). The function class F := {=<val>)>> (-) S + R/w E W) is a Donsker class such that it satisfies Donsker's 's theorem Csörgö et al. (2003), Additionally, its variance (()l(s) ~ d= is assumed to be bounded above for all E W.
This assumption is key to obtain the rate of how policy distribution discrepancy between Tu and T decays when the size of DD increases. It is common assumption when considering training with finite data samples in an empirical process Ma & Kosorok (2005); Geer (2000).
In general, the offline dataset DD should include samples produced by the safe and unsafe policies. However, unfortunately, in many existing works, there is no discussion on how good the data quality is, as it will influence the model learning. Denote by D, and D,, the safe and unsafe data subsets in DD and N₂ and N,, the corresponding sizes. We now conduct a thought experiment in this context to motivate the following assumption. In scenario, if 0 N./N << 0.01, which means the number of safe samples are quite small, one may argue how the agent would learn the optimal policy from such dataset D. while ensuring the safety constraint. Ideally, since we resort to the advantage weighted regression in our work, in an asymptotic manner with infinitely many iterations, i.e., T - oo (or a more practical sense, sufficiently large number of iterations), we should attain " even when N, 1. Certainly, if N₂ = 0, then the learning would fail since there is no safe policy to be induced from D, regardless of what T' is. Thus, the above discussion essentially results in a more difficult challenge between the time complexity and the sample complexity, which can be key future work of interest. In this work, we pay only attention to the sample complexity such that the following assumption should be imposed on the dataset D.
Assumption 6. Consider a pre-collected dataset D such that its safe data subset D. + 0. Denoting by N. the size of D.. the ratio N./N O.
Though in Assumption 6 we need D to include at least some safe data samples, we don't really require: a specific number due to the proposed method. Compared to BCSafe [iu et al. (2023a), which is only trained with safe trajectories that satisfy the constraints, Assumption 6lis is much weaker and more practically feasible. Thus, we arrive at the following main result.
Theorem 3. Let Assumptions 5 and 6 hold. Define V (po) == Esorupo IV.T (su)] given policy For the policy ww optimized by Eq. 32 with a dataset D consisting of N samples, the performance
gap satisfies the following relationship
V.5" (po) - V.. "po) 20 (N0OLst..)
(33)
for any > 0, when N + 0o.
Proof. By Pinsker's inequality and Jensen's inequality, we have the following relationship
Es (Drv (m)))>
S v (e) [Drr. (<.()1)))(=* (+1s))/2.
(34)
Based on the traditional supervised learning, it suffices to solve the empirical risk minimization
as shown in Eq. 32 so as to minimize Ex-..o [Dx ( [[ss)][T] ((1s) in the upper bound. By Assumption 5 the distribution function induced by the state distribution (s) is (s)
16

---

Published as a conference paper at ICLR 2025
Similarly, its empirical distribution function is N E Uw (s:). Hence, based Donsker's theorem, we can acquire
N (s)
Σ
vw (O, ((uw(s)|s d (s))
(35)
N
which tells us that the centered and scaled version of N >i (st) converges in distribution to a normal distribution with mean O and bounded variance (v (s)ls= d (s)). Immediately, we have
N
VN (s)
(s:)
N 0, in probability.
(36)
N ==1
With this in hand, multiplying Eq.
34 by N7..25 can be rewritten
NO.25 (4) Drv
(-1s))
S N
(1ss))/2
= 2.
(s)
(37)
= 2
(s)
1 (8e)
N
i=1
1 No.5 (s
1
Uno (s:)
N-00, 0, in probability.
2
N
Thus, the above equation implies that Bond (e) Dry (T((|))] (+|+)) should decay in a rate
O No.26oo for any > 0, when N approaches cc, which completes the proof.
In Cen et al. (2024), they also arrived at the similar conclusion, but the performance gap is still dictated by the state distribution discrepancy between dx)(s) and dd (s). This means their performance gap is reduced when the size of 'D expands, but cannot reach to O. Instead, Our work has further reduced it by converting the state distribution discrepancy to policy distribution discrepancy, given sufficient time. When applying the similar techniques to the cost value function, have the following result.
Theorem 4. Let Assumptions holdd. Define V (po) Ex [V. (so) given policy For the policy Tw optimized by Eq. 32 with dataset 'D consisting of N samples, the constraint violation bound satisfies the following relationship
VC - K 50
(38)
NO.25.0
for any 0.
Proof. The proof can be obtained by combining the proof from Theorem 2 and Theorem
Surprisingly, when + oc, (To) K, which ensures the safety constraint. This delivers useful insight that when N is sufficiently large, the cost constraint violation will be significantly small, even neglected.
Sample Complexity. If we set an accuracy threshold for both performance gap and constraint violation bound as X for any arbitrarily small constant X >> then roughly the size of D satisfies N ( ), which suggests that the sample complexity is much larger than that in Hasanzade et al. 2021). This is attributed to the offline manner, while in their case, distribution drift is not an issue. Our result also resembles the claim in Theorem from Xu et al. (002a)). We remark on the sample complexity obtained in this context that it is the worst case sample complexity the positive constant V can be any value. Practically speaking. it falls into the range of (0, 0.25) as suggested by al. (2021): Shi & Chi (2024). Our analysis on the sample complexity facilitates the theoretical understanding of safe offline reinforcement learning and offers useful insights for more efficient future algorithm design.
existing works Nguyen- Tang al. (2021); Li et al. (2024); HasanzadeZonuzy
17

---

Published as a conference paper at ICLR 2025
A.3 ALGORITHM AND IMPLEMENTATION DETAILS
In this section, we outline the algorithmic framework and provide implementation details for our method, focusing on the practical aspects. Our methods build upon the CORL Tarasov al. 2024) implementation for Implicit Q-Learning (IQL), with the codebase inspired by the OSRL (Liu et al., 2023a) style. The training process involves sampling batch from the replay buffer containing offline data during each gradient step. The expectations described in the methodology are approximated either by computing means over this batch or by using a Monte Carlo approximation.
We employ two Q-value networks, denoted as Qo, (s, a) and Qe, (s, a)), to improve training stability and mitigate overestimation Q-values. The value function (s) is trained using the minimum of the two Q-networks, following Double Q-Learning approach.
The reward value function (s) trained using the following asymmetric L.2 loss function:
(v
Σ
LE
min Qo. (3,
Vo(s))
(39)
B
(K.O)
Both Q-networks are updated simultaneously using the following loss function:
(Q(01.02) = 1 Σ E
(r+>Vo(6') -o. (s.))²)
(40)
Similarly, we use two q-value networks for cost and use the maximum of this ensemble. The cost critic networks are updated with the following loss functions:
max 2g (s, (s.0)) 112
(n)
LE (g((s)
(41)
(s,a)B
(51
Σ Σ ((++ Q (s,
(42)
B
1=12
(s,
The loss funetion used for the CVAE policy approximated as:
(a, 3)
Σ
wo(s, a) (log s, - DxL (ga s, a) p(> | a,)))), (43)
B
(s,0)
where Ga is the latent variable and we(s, a) exp (v(vVs) - 02(s, ())(s.))) is the weight
given to the behavior cloning loss, common in policy extraction methods like advantage- regression. This method assigns greater weight the state-action pairs with cost value (Q)(s, a)) smaller than the cost value associated with that state (s)), allowing the policy to selectively imitate safer actions from the dataset. To further motivate the CVAE learn the distribution of safe state-action and reconstruct safe actions, particularly in metadrive, we assign w. (.,ss, a) = 0 for transitions with either cost values greater than threshold of 0.02. Other hyperparameters used in our methods are discussed in the section that follows.
Finally, we reward-advantage weighted regression loss train the latent safety encoder policy to predict
embeddings in the latent space:
(.)
-1 [
Σ
[exp (5 (QT(s, Vz()))) log 8,2)] , po(. (44)
(s,a)EB
18

---

Published as a conference paper at ICLR 2025
Algorithm 2: Learning LSPC Policies
Input: Dataset 'D and hyperparameters including learning rates v, soft update rate T and others Initialize: Network parameters 0,01, 02. 9, th1, 1/2, Ox, ,3,0
for Gradiens Steps do
Sample batch B of transitions (s, a, C, from D
Use equation 39 to update the parameters of the reward value function:
ó + Φ V
Use equation 40 to update the parameters of the reward q-value functions:
{01, 02) - 11 02) uy (01.02) Lo 02)
Use equation 41 to update the parameters of the cost value function:
!) 1
(1)
Use equation 42 to update the parameters of the cost q-value functions:
{th, 12) f 111 1 1/2 Ma V (v), (1 (/2)
Use equation 43 to update the parameters of the CVAE encoder and decoder:
{a, } < 33 Vas V ( (p.
(3)
10
Use equation 44 to update the parameters of the latent safety encoder:
8 + (z
"
Update the parameters of the target functions:
0 (1- Tá, B2 (1 T)02 T02
1 - (1 - L 1222 - (1-7)0'
12 Return: Trained network parameters ④, 01, 0z vi, 22, a,
A.3.1 HYPERPARAMETERS AND TUNING
To facilitate understanding and analysis of hyperparameter effects, we classify them into three broad categories: IQL critic learning hyperparameters, AWR policy extraction hyperparameters, and those specific to our method, LSPC. The IQL and AWR hyperparameters are kept consistent across all experiments in this study. These common hyperparameters are listed in Table 2
Table 2: Common Hyperparameters for IQL and AWR
Hyperparameter
Value
Batch size ([B)
1024
Discount factor (7)
0.99
Soft update rate for Q-networks (7)
0.005
Inverse temperature for reward
2.0
Inverse temperature for cost
2.0
Learning rates for all parameters
x 10-1
Asymmetric L2 loss coefficient (E)
0.7
Max exp advantage weight (both cost and reward)
200.0
For the LSPC-specific hyperparameters, the values are chosen to balance the performance and safety constraints while trying minimize the need for extensive tuning. The KL divergence coefficient in VAE loss functions regularizes the latent space by encouraging the learned distribution to approximate prior distribution, balaneing reconstruction accuracy with structured latent representations for effective data generation. We set this to 0.5 in all of our experiments. The dimension of the latent safety space fixed 32, the general idea being that the space should be of sufficient capacity to capture the safety-wise distribution of the state-action in the dataset. The degree of restriction that constrains the policy to operate within the high-likelihood region of the latent safety space is set to 0.25 for LSPC-S. Generally, the same restriction value of 0.25 works for LSPC-O as well. However, for BulletSafety environments (except AntRun), this restriction is relaxed to 0.6 to enhance performance against baselines, given the upperhand our method has in managing cost returns. While we strive for consistent hyperparameters across tasks, the diversity of the benchmark tasks, with each using different simulators, dynamics, and objectives, might require adjustments.
19

---

Published as a conference paper at ICLR 2025
60
LSPC-O LSPC-S
LSPC-O
Normal normalzzed Reward Return
50
LSPC-S
CVAE Palicy
CVAE Policy
80
Cost Return
40
Cost Thresholds
70
30
20
60-
10-
0.6
0.2
0.8 Restriction Hyperparameter
Restriction Hyperparameter
Figure 4: Effect of variation of the latent space restriction hyperparameter while evaluating in Metadrive Easy Sparse environment. The red dashed horizontal lines represent the cost thresholds suggested by Liu al. (2023a) for this environment.
Figure illustrates the impact of varying the restriction hyperparameter for both LSPC-S and LSPC-O. As the latent safety restriction is increased, the reward performance LSPC-O improves progressively relative to LSPC-S and CVAE Policy. This is because the latent safety encoder policy LSPC-O can explore a larger region within the latent space to find award-maximizing actions. However, this improved performance comes with trade-off, i.e., higher cost returns. The figure demonstrates how different settings of the restriction hyperparameter yield different LSPC-O policies, each safe under certain cost thresholds used in our experiments.
A.3.2 ABLATION
Ablation studies are critical for evaluating the influence of individual components of a model on its overall performance. While the main text primarily focuses on presenting the core contributions of our Latent Safeet--Constraig
Policy (LSPC) framework, this section provides detailed exploration of specific ablation studies to address the impact of hyperparameters, architectural design choices, and loss functions on performance and safety adherence.
Role Reversal of CVAE and Safety Encoder. In this experiment, we investigated the impact of reversing the roles of the CVAE (a. 3) and safety encoder (6) in the LSPC framework. Specifically, in this configuration (referred to as Converse LSPC-O), the CVAE is trained to maximize rewards while the encoder minimizes costs within the inferred latent space. The architecture is detailed in figure 5a and the training is conducted for 300k timesteps in the HalfCheetah-veloe -ellociy environment with varying levels of restriction (c) in the latent space. The respective training curves are in figure 5bb)
1000
Reward Return
Eacoder
Latene Space
Decoder (9)
Action
Cost Return
Action
2000
0.25
0.05
500
05
-.0
Maximize
2.0
State
Encoder
Minimize V
Training Step 1e5
Training Step le5
(a) Converse LSPC Architecture
(b) Converse LSPC-O HallCheetah-Velune
Figure 5: Ablation study on role reversal CVAE and Safety Encoder
With low (high restriction), the encoder (6) struggled to learn an effective cost-reducing policy. As increased, some decrease in costs was achieved; however, this came at the expense of lower reward returns. While we can further loosen the restriction to get safer policies, very loose restrictions may cause nut-of-distributie
(OOD) action generation issues. A general concern in CVAE frameworks that utilizing latent space samples too far from the mean might cause the decoder to generate outtr--distribution (OOD) samples, the density of training samples decreases further from the
20

---

Published as a conference paper at ICLR 2025
mean. This highlights the necessity of designing latent space representation that satisfies both the in-distribution requirement and safety constraints, as addressed in the original LSPC framework. By carefully structuring the roles of the CVAE and safety encoder, the original framework ensures that the inferred latent space boundaries contain safe and in-distribution samples. Moreover, the original framework always maintains conservative policy (LSPPCS) learned during the training of LSPC-O as backup. This ablation study underscores the importance of these architectural design choices in balancing performance and safety.
Inverse Temperature Hyperparameter in LSPC-S. In AWR, the inverse temperature hyperpa- rameter À dictates the tradc-off between behavior regularization and value function optimization. For LSPC-S, lower indicates higher degree of behavior cloning, while larger À places sharper weight on samples with better cost advantages. To evaluate the impact of this parameter on both reward performance and safety adherence, we trained LSPC-S with values of 1, 2, and 4 in the CarRun and BallCircle environments.
Reward Return
500
Cost Cost Retum
40
Rewart Return
400
Cost Return
40
My
250
20
20
0:
0.0
0.5
1.0
0.0
0.0
1.0
0.0
0.5
1.0
Training Step
Training Step 1e6
Training Step
Training Step
(a) Pybullet Car Run
(b) Pybullet Ball Circle
Figure Ablation study on the effect of inverse temperature hyperparameter (A) in LSPC-S
In the training logs presented in figure 6 it can be observed that safety adherence is consistently maintained by LSPC-S across all tested X values. However, higher À values typically results in restricted reward-wise performance. This effect is particularly pronounced in the BallCircle environment, where lower À values, such as À = 1, enable LSPC-S to achieve significantly higher reward returns compared
and 4.
Reward Return
40
Reward Return
40
Cost neturn
500
Cost Return
500
Mapp
My
20
250
20
8-05
0.7
0.0
1.0
0.0
0.0
1.0
0.0
0.5
1.0
Training Step
Training Step 1eB
Training Step
Training Step
(a) Ball Cirele (Variation in A)
(b) Ball Circle (Variation in
Figure 7: Ablation study on how the effect of variation in and & translate to LSPC-O
These findings raise an important question about how the varying reward returns and consistent safety adherence in LSPC-S translate to LSPC-O. Our results indicate that when the CVAE (or LSPC-S) is trained with smaller À values, the reward performance of LSPC-O improves. However, this improvement comes with an increased cost returns, potentially leading unsafe policies as depicted in figure 7a While LSPC-S with lower À effectively balances reward and cost returns, the encoder LSPC-O, trained further maximize rewards, can exacerbate safety risks. This highlights the critical trade-off between performance and safety optimization when selecting À in the LSPC framework.
Asymmetric Loss in IQL. this experiment, we analyze the dependence of our method on the asymmetric loss function of IQL, particularly for cost management and safety adherence. As listed in A.3.1 we use a coefficient of & = 0.7 in the asymmetric loss function used for expectile regression in Implicit Q-Learnng (IQL) our LSPC framework. In LSPC-S, this is intended to discourage the underestimation of cost Q-values. Here, we compare the results against & = 0.5, which leads to a symmetric Mean Squared Error (MSE) loss, equivalent SARSA-style policy evaluation.
21

---

Published as a conference paper at ICLR 2025
Reward Retur
500
Cost Return
Reward Return
400
Cost Returr
40
250
20
20
0.5.5
8-0.5
0.7
8 0.7
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Training Step
1e6
Training Step
1e6
Training Step
1e6
Training Step le6
(a) Pybullet Car Run
(b) Pybullet Ball Circle
Figure 8: Ablation study the effect of IQL expectile loss coefficient ( in LSPC-S
As can be observed in figure g using the symmetric loss in LSPC-S led to improved reward-wise performance compared to the asymmetric loss (conservative cost estimation), with both & values resulting in policies that adhere to the safety requirements. However, for LSPC-O, where an additional encoder is trained to optimize rewards further, the symmetric loss resulted in higher cost returns, indicating chances of safety compromises. These findings depicted in figure @ggure7bJian highiigh a critical trade-off, similar to the ablation with while less conservative policies with symmetric loss can achieve better rewards, they may translate to increased safety risks in LSPC-O.
A.3.3 SAFETY AND PERFORMANCE TRADE-OFFS IN LSPC-S AND LSPC-0
The CVAE in the LSPC framework is trained using a cost-advantage weighted ELBO loss, aiming to minimize the cost value function V.". By sampling actions from probbbbbbbbiyt regions in the CVAE latent space, LSPC-S rerpesents conservative policy that effectively mitigates the risk of out-of-distribution (OOD) actions and generates safe actions. While L.PC-O similarly samples from this restricted latent space and is theoretically expected to maintain equivalent safety, it may introduce some trade-offss in safety by optimizing for wwward-maximizing latent variables rather than random latent samples. These trade-offs in safety often be attributed the latent safety encoder in LSPC-O exploiting inaccuracies in the action mappings (from the CVAE decoder) to maximize rewards, potentially leading OOD unsafe actions. For example, as seen in the ablation study, while LSPC-S ensures safety even under less restrictive hyperparameters, LSPC-O, due to its reward-seeking objective, can exhibit higher cost-return, often leading to disproportionately higher costs.
Despite these trade-offs, the flexibility of our framework allows for the adjustment of safety require- ments via tunable restriction hyperparameter. A higher restriction (lower e) ensures that samples are drawn from regions closer to the mean of the latent distribution prior (which means stricter LSPC constraint). In addition, when c is smaller, the encoder in LSPC-O is also forced to operate within a smaller region, resulting in a more restrictive policy. This flexibility in tuning e according to the safety requirements allows the framework to adapt to different safety conditions for a given task, while still optimizing for rewards within those constraints, whether loose or stringent.
A.3.4 TRAINING TIME OF THE EXPERIMENTS
The device used for reporting the training times in this section is Dell Alienware Aurora R12 system with an 1 1th gen Intel Core i7 processor, 32 GB DDR4, and an NVIDIA GeForce RTX 3070 8GB GPU. All experiments were run on a CUDA device.
LSPC training requires approximately 3.5 hours for million training steps, achieving rate of around 90 training iterations per second. It is important to note that this reported training time incorporates the training of both LSPC-S and LSPC-O components of our framework. This training time performance comparable to CPQ under similar conditions and training steps. Behavior Cloning (BC), being more straightforward, is faster, requiring about 35 minutes for I million timesteps 490 training steps per second. CDT training for 100k timesteps takes approximately 3.5 hours, running at about iterations per second.
22

---

Published as a conference paper at ICLR 2025
A.3.5 COMPARISON WITH CPQ
CPQ (Xu et 2022a) also employs CVAE and is referenced multiple times in this paper. Here, highlight the key differences between our proposed method and CPQ across the following dimensions.
Value Learning. CPQ utilizes cost-sensitive version of CQL (Kumar al., 2020 which max- imizes the cost value for out-of-distribution (OOD) actions while using only constraint-saf and in-distribution-saff samples to learn the reward Q-function. In contrast, constraint- -penalized Bellman operator like CPQ. Instead, we adopt IQL for ciiporaa-diffeeence (TD) learning of both reward and cost values, employing standard Bellman updates to ensure simplicity and stability.
method does not the
Policy Extraction. CPQ applies the off off-policy deterministic policy gradient method for policy extraction. Our approach, however, employs Advantage Weighted Regression (AWR), where the actor updated via weighted behavior cloning. This ensures alignment of the policy with the behavior distribution, promoting stability in offline RL settings, Specifically, we train cost-AWR to optimize the CVAE policy and -eWRrd-AWR to optimize an additional encoder.
CVAE Usage. While both CPQ and our framework incorporate a CVAE, their purposes diverge significantly. CPQ uses the CVAE detect and penalize OOD actions high-cost during Q-learning. However, this approach can distort the value function, thereby hindering generalizability and safety adherence, as observed in our experiments and noted in prior works such as DSRL (Liu et al.) 2023a) and FISOR (Zheng et al.] (024). By contrast, our method employs the CVAE as policy model that expressively represents safety constraints in the latent space, rather than as an OOD detection mechanism.
A.4 BENCHMARK DETAILS AND ADDITIONAL BASELINES
For empirical evaluation of our proposed methods, we use the standard Datasets for Safe Reinforce- ment Learning (DSRL) benchmarking suite introouucdd by Liu al. (023a). The authors present comprehensive set of datasets designed to facilitate the development and evaluation of offline safe reinforcement learning algorithms across standard safe RL tasks. This suite includes D4RL-styled (Fu eet al., 2020) datasets and robust baseline implementations for offline safe RI RI,_aiding aiding in both training the official implementation and hyperparameters
and deployment phases. For baselines, for FISOR, as provided by the authors, along with their published results (Zheng et al. 2024). For all other baselines, we rely on the OSRL's official implementation, hyperparameters, and results
as reported in the corresponding whitepaper| Liu et al. (2033a). OSRL implementations as well as hyperparameter configuration can be found here"
(a) Metadrive
(b) Bullet Safety
(c) Safety Gym
Figure 9: Example images from the simulation environments used in the experiments
For evaluation and comparison against the baselines, we use the metrics suggested by the DSRL whitepaper. use the normalized reward return Rnormalized, calculated as:
R2 Tmin (M)
Rnormalized
Tima (M) T Tmin (M)
http::///gtthhbb.mm////egggg// AIR/P ISOR
23

---

Published as a conference paper at ICLR 2025
where I'mxx (M) and (MM are the task-specific maximum and minimum empirical returns, respec- tively. Additionally, we use the normalized cost return normalized. defined as:
Cx +
noorallizdd
K: K
where i is the target cost threshold and ç is a small constant to avoid division by zero. These metrics ensure consistent comparisons across safe RL tasks. Following the DSRL constraint variation evaluation, each method is evaluated on each dataset with three distinct target cost thresholds and across three random seeds. DSRL evaluation uses the average of the normalized reward and cost to better characterize performance under varying conditions; this is particularly relevant for the Lagrange-based algorithms that depend on the cost threshold. The cost threshold values used in DSRL evaluation are set at [10, 20, 40] for MetaDrive and Bullet Safety Gym environments, and [20, 40, 80) for Safety Gymnasium environments. The same criteria are applied in our evaluation tables the normalized cost threshold is set to 1. Bold letters in the evaluation table represent safe agents whose normalized cost is smaller than I, while blue and bold indicate the safe agent with the highest reward. Additionally, if safer agent is present within a -0.05 normalized return, that agent is also highlighted in blue and bold.
A.4.1 METADRIVE
MetaDrive is a self-driving simulator developed using the Panda3D engine, designed to simulate realistic driving environments with complex road structures and dynamic traffic conditions. The autonomous driving tasks in MetaDrive evaluate safety through three primary criteria: (i) collisions, (ii) off-road driving, and (iii) speeding violations. There are nine distinct offline learning tasks, each named in the format (Road) } [ehicll). The Road category features three difficulty levels for self- driving cars: easy, medium, and hard. The Vehicle category includes four levels of surrounding traffic: sparse, medium, and dense. The stochastic nature of the environment is introduced through the random initialization of traffic patterns and map layouts, providing a challenging and varied benchmark for assessing the performance and safety of reinforcement learning algorithms in autonomous driving scenarios (Li et al. 2022 Liu et al. (023a). Table [ summarizes the different configuration and map types used in the different evaluation environments that we use within the MetaDrive self-driving simulator.
Table 3: MetaDrive Performance Metries
ESFCCD
Teil
sevad"
ex0
wwari
exe.
reauet
[ewerl
ceel
(1wes)
0.37
1.54
8.11
8.31
1.77
8.33
0.76
3.00
0.11
0.55
-000
14.7
5356
5.44
8.3
1..5
8.62
1080
8.71
E:46
woyemee
0.37
1.6
8:11
8.14
0.33
8.33
0.36
0.27
0.62
0.41
-836
0.5
2.30
8:36
1080
8.55
8.18
6:21
nadkoupana
0.33
1.38
0.37
6.11
3.41
6.42
1077
0.54
0.52
0.15
2.53
0876
6.00
-616
relersied
01.53
141
0.34
8.11
11:55
2.4
0.:35
6,11
0.25
4077
0.87
2.30
6.09
0.12
8.59
607
0.53
Laddess
0.30
157
0.13
8.46
0.33
8.9
0.4
355
6080
0.21
4005
0.85
0.32
3.77
6.36
1699
8.53
0.21
8.53
6.57
Averu:
0.42
236
0.55
0/2
406
2.7
008
0.00
0.17
A.4.2 SAFETY GYMNASIUM
Safety Gymnasium 5 is a collection of environments based on the MuloCo physics simulator, designed to offer variety of tasks with adjustable safety constraints and challenges, enabling different levels of difficulty (Ji et al. 2023). We assess the car agent on Button, Goal, and Push tasks, where the numbered suffixes (I and 2) indicate the difficulty levels. Additionally, Safety Gymnasium includes subset of Safe Velocity tasks that impose velocity constraints on agents, extending the standard GymMuJoCo locomotion tasks to incorporate safety considerations. We evaluate the methods on Swimmer, Hopper, HalfCheetah, Walker2D, and Ant for the velocity tasks.
A.4.3 BULLET SAFETY GYM
suite of environments built on the PyBullet physics simulator, designed similarly to Safety Gymnasium but with shorter time horizons and more agents. Unlike Safety
Bullet Safety Gym 0
24

---

Published as a conference paper at ICLR 2025
Table 4: Safety Gym Performance Metrics
rak
60:1
L001
550
ES.-1
revu21
1951
167-05
FEVES3
0033
1.15
0.85
08:1
1.5
1016
1.50
8:19
2:13
01:3
1.55
2018
1.95
-.82
8.14
0183
8.64
18.0
11.1
0.:01
8:53
0.34
0.20
0.80
123
0,41
8.30
0.6
1333
0.:9
1.4
0.20
1350
0.47
0.33
08.2
8.23
8:33
1uar
Damuaa
0.32
8.76
0011
0.33
8:33
0.8
0.33
8.33
8:33
0.53
0883
8.55
8.33
0:38
8:33
8:15
0635
03:99
8.09
806
2.35
600
0,50
8.85
0.2
1.9
0.29
8.34
600
0.79
0.44
6.18
616
6.79
0.84
0.33
6.6
133
073
100
Gymnasium, which features longer time horizons with shorter simulation steps, Bullet Safety Gym
employs shorter time horizons, which may facilitate faster training (Gronauer) 2022). The suite includes four distinct agent types: Ball, Car, Drone, and Ant, as well as two task types: Circle and Run. The tasks are named in the format [Agent) (Tass)).
Table 5: Bullet Safety Gym Performance Metrics
Taxt
roward 1 can:
Lonadd
coa
Invudd
ooe
sepurt
car.
soaad
cod
rrvadd
sceuri
con.
codl
rnvued
onr:)
[asa
0.5
314
11.1
1-db
1:15
11/00
3.71
-1.8
2715
1.51
11.77
3-3
8.15
11:11
670
0.11
$1.04
1000
1036
3.13
0.38
8:58
8:43
8.79
3:54
0183
3,87
337
1197
8:15
8.18
1..65
6:54
0.88
0.57
800
SuECack
0.75
4.53
0.64
0.56
0.50
0.34
01.30
1.44
0.6
0.66
01:4
2.16
8:11
0.33
0
7.14
0.40
0183
6.30
0.88
0.72
104
0338
4.50
0.40
6.66
0.34
035
8.00
0.80
0.17
306
9.20
0.9)
6.13
0.83
0.46
1.40
A.5 COMPOSITION SAFETY PROFILES OF DATASETS
020
1250
1000
Iropuaccy
250
200
20
20
43
50
50
10
20
60
20
60
00
(a) Ball Circle
(b) Ball Run
300
500
500
20
Cort Benum
Cost Ratur
(c) Car Circle
(d) Car
Figure 10: Distribution of the trajectories in the dataset based on cost return and reward return
Almost all of the datasets used in our evaluations include both safe and unsafe trajectories. The safety compliance of policies during evaluation assessed based on their adherence to the undiscounted cost-return threshold established in our experiments, consistent with standard benchmarks. The red dashed lines figure 10 indicate these thresholds.
The figures in 10a trajectories across four representative datasets, including those used the ablation study. Each point corresponds to a single trajectory/episode.. Additionally, the frequency distributions of cost returns for these datasets are provided in the right, highlighting that many trajectories exceed the safety threshold, often with unsafe trajectories outnumbering safe ones. These visualizations illustrate the datasets' composition and demonstrate the robustness of LSPC in learning policies that maintain safety compliance despite mixed safety profiles.
10d include scatter plots showing the reward return versus cost return for
25

---

Published as a conference paper at ICLR 2025
SCS
XST
TRO
Figure 11: Block sequence and corresponding map samples in the Metadrive Simulator
Table 6: Metadrive Environment Configurations
Environment
Traffic Density
Block Length
Block Sequence
Horizon
Lane Width
Lane Count
EasySparse
0.1
SCS
1000
3.5
3
EasyMean
0.15
SCS
1000
3.5
3
EasyDense
0.2
SCS
1000
3.5
3
MediumSpars.
0.1
XST
1000
3.5
3
MediumMean
0.15
XST
1000
3.5
3
MediumDense
0.2
XST
1000
3.5
HardSparse
0.1
TRO
1000
3.5
3
HardMean
0.15
TRO
1000
3.5
HardD
0.2
TRO
1000
3.5
3
A.6 METADRIVE TRANSFER EXPERIMENT
For rigorous evaluation of the proposed methods in varying scenarios, a set of nine distinct environ- ments with different configurations is employed in this study. These environments categorized into three difficulty levels: Easy, Medium, and Hard. Each difficulty level includes three variations based traffic density: Sparse, Mean, and Dense. Each environment is characterized by specific parameters, including the start seed, traffic density, block length, block sequence, simulation horizon, lane width, and lane count. The block sequences selected correspond the difficulty of the driving for for hard levels. The configuration is summarized in table 6 and the example maps corresponding to cach block sequence type illustrated figure|
task: SCS Brrright-Curv--Straiee medium levels, and TRO "TInterection-Quteee
for easy levels, XST SiireectiionSSrarg-t-TTnie
In the transfer experiment, aim to test the generalization capability of offline RL algorithms by training agents in one environment configuration and then transferring them to different configu- ration without retraining. Specifically, focus agents trained in two extreme scenarios: Easy Sparse and Hard Dense. We then evaluate their zero-shot transfer performance across a range of other configurations. This setup enables assess how effectively the agents adapt to
26

---

Published as a conference paper at ICLR 2025
80
80
60-
20
EasySparse
EasyDunse
EasyMtan
LSPC-0 Rewards
80
50
Costs
LSPC-
40
60
MeckumSparee
Medkmmteean
MediumDDevve
MedkumSpanne
MedumtMea
MediumDenee
40
30-
30
20
20
10
HardSparse
HardMenn
HarsDende
HardSporse
HardMean
HardDence
(a) Agent Trained in Easy Sparse
(b) Agent Trained in Hard Dense
Figure 12: Zero-shot transfer performance of agents trained the Easy Sparse and Hard Dense scenarios when evaluated in various other settings. The unhatched bars represent the average normalized reward return, while the hatched bars indicate the average cost return, evaluated across 20 episodes. The dashed lines represent the different cost threshold used in this paper and in prior works.
environments without additional training, providing insights into the robustness and flexibility of the learned policies.
The results of this experiment are illustrated in figure 12 In the figure, unhatched bars represent average normalized reward return, while hatched bars indicate the average cost return, both evaluated over 20 episodes. The dashed lines correspond to the different cost thresholds used in this study as well as the DSRL (Liu et al.] 2023a). Notably, the agent trained the Hard Dense scenario exhibited improved reward performance when transferred easier and sparser tasks, even surpassing its own performance in the Hard Dense environment. This outcome likely due to the agent's exposure more challenging data during training, which enhanced its adaptability to simpler settings-a result that is not always commonly observed in transfer learning scenarios. In contrast, the agent trained in the Easy Sparse environment experienced performance drop when transferred to denser and more complex tasks, with the decline being more pronounced in medium difficulty scenarios than in the hardest ones. Despite these variations in reward performance, both agents consistently maintained safety across all cost threshold levels, demonstrating robust safety across all difficulty and density configurations. This consistency aligns with the core principle of our method, Latent Safety- Priort-ii Constraints (LSPC), which ensures that the policy maximizes rewards while adhering to safety constraints within the latent space.
27


Reviews:
[{"review_id": "QZUWM5qxta", "reviewer": "Reviewer_He6z", "summary": "This paper aims to solve the offline safe RL. To balance safety constraints and distribution shifts in offline safe RL, they use the CVAE to model the policy. Since the CVAE is trained on the offline dataset, this paper expects the CVAE architecture to achieve a balance between safety constraints and distribution shifts. To ensure the CVAE achieves such balance, they propose two methods to use the CVAE. The first method, LSPC-S, penalizes the high-cost state based on the cost function learned by IQL. The second method,  LSPC-O, focuses on reward-maximizing under the CAVE-based policy. This paper demonstrates the effectiveness of their methods through empirical evaluation and theoretical analysis.", "questions": "1. Why choose CVAE? Many other generative models, such as the flow and diffusion models, have been shown to outperform VAEs in many fields. Could other generative models achieve better results? \n\n2. Regarding Lemma 2. Theorem 2 in Cen 2024 [1] is bounded by $ \\frac{2R_{max}}{(1-\\gamma)}D_{TV}[d^D(s)||d^*(s)]+e_N $. I am unable to understand why Lemma 2 is tighter than Cen 2024 since the relationship between $ D_{TV}[d^D(s)||d^*(s)]+e_N $and $ E_{s\\sim d^{\\pi^*}}[D_{TV}(\\pi|\\pi^*)] $is unclear. And it's surprising that the upper bound for safe offline RL is tighter than that for offline RL, as offline Safe RL is a more challenging task. \n\n3. I noticed that this paper is similar to [2], as both use CVAE to model policies. What is the difference between the two?\n\nReferences:\n\n[1] Cen, Zhepeng, et al. Learning from Sparse Offline Datasets via Conservative Density Estimation. \n\n[2] Xu, Haoran, Xianyuan Zhan, and Xiangyu Zhu. \"Constraints penalized q-learning for safe offline reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The authors provide a theoretical analysis of their method.\n- The proposed method outperforms baselines in safe RL tasks.", "weaknesses": "- The method proposed by this paper cannot strictly keep safe constraints. Since a state-action pair is safe iff $ V_c^\\pi(s)\\leq \\kappa $. Equation 10 only serves as a regularizer to penalize the high-cost actions and cannot guarantee $ V_c^\\pi(s)\\leq \\kappa $.\n- Theory insights are one of the contributions of this work. However, the authors do not derive an exact sample complexity; they only claim that the sample complexity for obtaining an $ \\epsilon $-optimal policy is about $ \\mathcal{O}(1/\\epsilon^4) $and put the sample complexity results in the appendix.\n- Lack of ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zl1wfeR30x", "reviewer": "Reviewer_NRFb", "summary": "This paper presents a novel Latent Safety-Prioritized Constraints (LSPC) framework aimed at addressing the balance between reward maximization and safety constraints in safe offline reinforcement learning. The authors model potential safety constraints using Conditional Variational Autoencoders (CVAE) and demonstrate experimental results across multiple benchmark tasks, indicating that the proposed method effectively enhances cumulative rewards while ensuring safety.", "questions": "- Although the experiments primarily focus on specific tasks, there is a lack of in-depth discussion on the application and effectiveness of the LSPC framework in multi-task learning. How will you address the variations in safety constraints and reward structures across different tasks?\n\n- When comparing with several benchmark methods, the analysis does not adequately demonstrate the specific advantages of LSPC across different tasks. Is it necessary to provide a more detailed discussion of the comparison methods, especially regarding their performance under different safety constraints?\n\n- When using Conditional Variational Autoencoders (CVAE) to construct a latent safe space, how do you ensure that the generated samples comprehensively cover all possible safety constraints in the environment? Is it necessary to validate the effectiveness and sufficiency of this coverage?\n\n- Decoupling reward maximization and safety constraints may lead to failures in safety assurance. How does the decoupled strategy effectively guarantee safety under different safety constraint conditions?\n\n- The thoretical analysisi is dense, it's better to provide more context and examples to help readers understand the impact and theoretical basis of this assumption.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\n\nThe LSPC framework offers a fresh perspective on managing the trade-off between safety and rewards, particularly within the context of offline reinforcement learning, demonstrating significant practical and theoretical value.\n\nSignificance:\n\nThe paper provides a detailed and persuasive theoretical analysis of policy performance, ensuring the scientific foundation of the proposed method.\n\nQuality:\n\nThrough systematic experimentation across various benchmark tasks, the effectiveness of LSPC is showcased, with significant advantages over existing methods.", "weaknesses": "Please refer to questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8sO5yGkIhN", "reviewer": "Reviewer_Qh7M", "summary": "Traditional RL learns optimal policies through interactions with the environment, but this approach can pose safety risks during the learning process. In industrial settings such as autonomous driving and robotics, even minor errors can lead to significant costs or safety hazards. To address these challenges, offline reinforcement learning has been introduced. However, offline RL, which relies solely on fixed datasets, risks generating unsafe policies when encountering new situations. This study proposes a new framework called Latent Safety-Prioritized Constraints (LSPC), designed to learn offline policies that optimize rewards while adhering to safety constraints, demonstrating its effectiveness across various high-risk tasks.", "questions": "## Theoretical Concerns\n\n1.  Setting safety constraints in the latent space\n    \n    LSPC uses a conditional variational autoencoder (CVAE) to learn safety constraints in the latent space. However, it is unclear whether the latent space can fully capture real-world safety constraints. If the safety constraints are not properly learned, the model may fail to ensure safety in unforeseen situations.\n    \n2.  Reliability of constrained optimization\n    \n    LSPC performs constrained reward optimization to maintain a balance between rewards and safety. However, it raises concerns about whether the optimization in the latent space will remain valid for actions not encountered during training. The paper lacks guarantees that the policy will behave appropriately for out-of-distribution state-action pairs.\n    \n3.  KL divergence-based safety assurance\n    \n    While safety is ensured by minimizing the Kullback-Leibler (KL) divergence between the learned policy and behavior policy, subtle behavioral differences, even with small divergences, could lead to significant risks in complex environments. Theoretical minimization of KL divergence may not fully capture all possible real-world scenarios.\n    \n\n----------\n\n## Experimental Concerns\n\n1.  Gap between simulation and real-world environments\n    \n    The paper evaluates performance primarily in simulated environments. However, simulations may not fully account for real-world uncertainties, such as noise, sensor errors, or dynamic changes. This gap poses a risk that LSPC’s safety and performance may fall short in real-world applications.\n    \n2.  Dataset limitations and bias\n    \n    Since LSPC relies on static datasets, there is a risk that the policy will not perform well in situations not included in the dataset. Furthermore, if the training data is biased toward specific patterns, the policy may make incorrect decisions.\n    \n3.  Interpretation of safety-performance trade-offs\n    \n    While the paper claims to maintain a balance between safety and rewards, it may not have thoroughly analyzed cases where this balance breaks down. For example, certain tasks may allow minor safety violations to achieve higher rewards, but the paper lacks a clear discussion of these scenarios.\n    \n4.  Stability of optimization\n    \n    The complex optimization process involving CVAE and implicit Q-learning can lead to convergence issues, resulting in unstable learning or local optima. The paper does not provide adequate solutions to address these potential issues.\n    \n\n----------\n\n## Comprehensive Concerns\n\n-   Are the safety constraints in the latent space sufficient? It remains unclear whether CVAE can effectively capture all necessary safety constraints.\n-   Lack of generalization to real-world scenarios: Reliance on static datasets raises concerns about performance degradation in new situations.\n-   Trade-offs between safety and performance: While the framework claims to balance safety and rewards, it may fail to guarantee absolute safety in specific scenarios.\n-   Complexity of the optimization process: Additional analysis is required to address potential convergence issues and instability during learning.\n\nThese concerns highlight important considerations for evaluating the applicability and reliability of LSPC. Future research should focus on validating its generalization performance in real-world environments and ensuring that the balance between safety and rewards is consistently maintained across various scenarios.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  Balance of safety and performance: Designed to optimize rewards while satisfying safety constraints.\n2.  Efficiency of offline learning: Learns policies without further interactions with the environment by using fixed datasets.\n3.  Theoretical stability guarantee: Offers theoretical bounds on policy performance and sample complexity.\n4.  Applicability in various environments: Demonstrates strong performance in complex tasks, including autonomous driving and robotic manipulation.", "weaknesses": "1.  Difficulty in latent space configuration: Finding the optimal latent space configuration requires empirical tuning.\n2.  Dependency on dataset quality: Safe policy learning heavily relies on the quality of the dataset.\n3.  Limitations in real-time application: Offline RL struggles to respond immediately to dynamic changes in real-time environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UEFRaRagI", "reviewer": "Reviewer_MBPj", "summary": "This paper develops a novel approach to safe offline reinforcement learning; a task that aims to maximize rewards and minimise safety constraint violations during training on an offline dataset. The proposed methodology leverages conditional variational autoencoders (CVAEs) to model latent safety constraints, effectively learning safety bounds. The proposed methodology can then be encompassed into two further methodologies that the authors evaluate upon: LSCP-S which prioritizes selecting the safest actions, and LSPC-O, which aims to maximize the reward within these latent safety bounds. In the evaluation, it is seen that LSCP-S effectively reduces cost and LSCP-O manages a nice balance between reducing cost and maximising reward.", "questions": "Can the authors comment on the following:\n- The disproportionate increase in rewards to the large increase in costs from the -S to -O models. A point that is then observed again in Figure 2.\n- Are the oscillations observed in Figure 2 a minor weakness with the method or simple hyperparam tuning that is needed?\n- Can you add experiment insights (computation/time taken/machine experimented on) into the appendix to improve transparency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- A promising approach to capture complex, non-linear safety constraints. Results clearly back this up and I was impressed with the methodology and theoretical analysis.\n- The literature review is comprehensive and up-to-date.\n- The broad range of benchmarks that the methodology is evaluated upon is robust and impressive.\n- Authors are transparent about the methodology's limitations and future work, showing openness and foresight.\n- Figure 3, in particular, is visually engaging and provocative for the reader. It enables further transparency into the proposed method and I really appreciated it.\n- Code is available and open.", "weaknesses": "- There is little to no discussion, apart from minor comments in the related work section, regarding comparative methods and hyperparameters which strongly hinders the assessment of Table 1. For example, in the metadrive experiments, CPQ doesn't seem to learn anything. More detail into this method or the hyperparams used at the start of the the evaluation section could have provided me with further insight. Currently, I am left thinking that the implementation wasn't tuned correctly. Also, there is no standard RL baseline against which to assess all methods.\n- An analysis point, I would like/liked to have seen a discussion on surrounds the comparison between LSPC-S and LSPC-O. In Table 1, there seems to be a disproportionate increase in cost between the -S and -O models. I would have expected, within reasonable deviation, to have seen a reasonably proportionate increase in reward to increase in cost between the -S and -O models. But, in the hard metadrive experiments, for example, the reward increased +0.03 on average whilst the cost increased +0.24 on average. I am not concerned at all about this, I would just like to see some discussion surrounding this interesting trade-off that the -O model has to try to balance.\n- Another analysis point, which slightly draws parallels from my previous point, is in Figure 2 it seems that LSCP-O observes large oscillations in terms of cost in comparison to other methods. The small increase in reward is seen from the -S to the -O model but a larger cost is also observed alongside some stronger oscillation. Could this oscillation be due to the model trying to find a trade-off balance or is this some hyperparam tuning that is needed?\n- Computational/Time Taken/Machine experimented on details are needed in the appendix. Without these, I personally would draw the conclusion that the proposed methods are computationally more intensive than comparative methods. Personally, I do not computation as a major weakness, but I would appreciate details about it to be transparent.\n\nMinor weaknesses\n======================\n- Figures 2 and 3 have very small text and need to be increased. The legends are also in a poor place, personally, and could be better placed above/outside the axis.\n- Tables could also benefit from rows having alternating highlights (white to light gray) to make them easier to read.\n- Please put boldface and blue colour reasoning in the table caption for readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)