OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Authors: Yihang Yao
Zhepeng Cen
Wenhao Ding
Haohong Lin
Shiqi Liu
Tingnan Zhang
Wenhao Yu
Ding Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe Reinforcement Learning
Offline Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Offline safe reinforcement learning (RL) aims to train a policy that satisfies con- straints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired safe and rewarding performance. In this paper, we mitigate this issue from a data-centric perspective and introduce OASIS (cOnditionAl diStributIon Shaping), a new paradigm in offline safe RL designed to overcome these critical limitations. OASIS utilizes a conditional diffusion model to synthesize offline datasets, thus shaping the data dis- tribution toward a beneficial target domain. Our approach makes compliance with safety constraints through effective data utilization and regularization techniques to benefit offline safe RL training. Comprehensive evaluations on public benchmarks and varying datasets showcase OASIS’s superiority in benefiting offline safe RL agents to achieve high-reward behavior while satisfying the safety constraints, out- performing established baselines. Furthermore, OASIS exhibits high data efficiency and robustness, making it suitable for real-world applications, particularly in tasks where safety is imperative and high-quality demonstrations are scarce. More details are available at the website https://sites.google.com/view/saferl-oasis/home.

OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Yihang Yao"l 1 Zhepeng Cen"1, Wenhao Ding's Haohong Lin', Shiqi Liu',
Tingnan Zhang", Wenhao Yu², Ding Zhao?
1 Carnegie Mellon University, 2 Google DeepMind Equal contribution, {yihangya, zcen)&andrew. cmu. edu
Abstract
Offline safe reinforcement learning (RL) aims train a policy that satisfies con- straints using pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired safe and rewarding performance. In this paper, we mitigate this issue from a data-centria perspective and introduce OASIS (cOnditionAl diStributIon Shaping), new paradigm offline safe RL designed to overcome these critical limitations. OASIS utilizes a conditional diffusion model to synthesize offline datasets, thus shaping the data dis- tribution toward a beneficial target domain. Our approach makes compliance with safety constraints through effective data utilization and regularization techniques to benefit offline safe RL training. Comprehensive evaluations public benchmarks and varying datasets showcase OASIS's ' superiority in benefiting offline safe RL agents to achieve high-rewar behavior while satisfying the safety constraints, out- performing established baselines. Furthermore, OASIS exhibits high data efficiency and robustness, making it suitable for real-world applications, particularly in tasks where safety imperative and high-quality demonstrations are scarce. More details are available at the website
Introduction
Offline Reinforcement Learning (RL), which aims learn high-reward behaviors from pre-collected dataset [1] has emerged as a powerful paradigm for handling sequential decision-making tasks
TO I.. Although standard offline RL has achieved remarkable success in some environments, many real-world tasks cannot be adequately addressed by simply maximizing a scalar reward function due the existence of various safety constraints that limit feasible solutions. The requirement for safety, or constraint satisfaction, is particularly crucial in RL algorithms when deployed real-world tasks [3 15 [7 [8 19].
such autonomous driving [3 4 5 6, and robotics 7
&
9
To develop an optimal policy within a constrained manifold 20 21] offline safe RL has been actively studied in recent years, offering novel ways integrate safety requirements into offline RL (22). Existing research in this area incorporates techniques from both offline RL. and safe RL, including the of stationary distribution correction 23 24 regularization (25 26], and constrained optimization formulations [27]. Researchers have also proposed the use of sequential modeling methods, such as the decision transformer (28] 29 and the decision diffuser 30
achieve advantageous policies
and meet safety requirements.
Although these methods show promise, it is difficult to handle state-action pairs that absent from the dataset, which is known notably as out-o--distriiuttin (OOD) extrapolation issues [31] 32 33
34 35 36 37]. To solve this, many works utilize regularization methods to push the policy toward behavior policy to achieve pessimism [35 36) However, this approach worsens the situation when the dataset is imbalanced and biased: regularization by imperfect demonstrations such as datasets
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

composed predominantly of low-reward data or containing few safe trajectories collected using unsafe behavior policies. This regularization also leads to another challenge: striking the optimal balance between learning objectives such as task utility efficiency and safety requirements, leading to reward
degradation or aggressive behavior [28 38 39].
To address these challenges, we introduce data- centric learning paradigm in offline safe RL, OASIS (cOnditionAl diStributlon Shaping), which focuses on improving the training dataset quality by steering the offline data distribution to a beneficial target domain as shown in Fig. ! OASIS distills knowledge from the imperfect dataset, and generates rewarding and safe data using conditional diffusion model accord- ing to the safety preference to benefit offline safe RI. training. This data-centric approach is parallel to and compatible with general model-centric offline safe RL algorithms which emphasize improvements to the learning algorithm and model architecture. The key contributions summarized as follows.
rewaan
cost limit
00st Boni
cost
cost
Figure 1: An example of distribution shaping in offline safe RL. We generate low-cost and high-reward dataset from the original dataset for subsequent RL training.
Identification of the safe dataset mismatch (SDM) problem in offline safe RL. We identify the mismatch between the behavior policy and the target policy and investigate the underlying reasons for performance degradation with this condition.
2. Introduction of the OASIS method to address the SDM problem. To the best of our knowledge, this is the first successful application of distribution shaping paradigm within offline safe RL. Our theoretical analysis further provides insights on performance improvement and safety guarantees.
3. A comprehensive evaluation of our method across various offline safe RL tasks. The experiment results demonstrate that OASIS outperforms baseline methods in terms of both safety and task efficiency for varying tasks and datasets.
2 Related Work
Offline RL. Offline RL addresses the limitations of traditional RL, which requires interaction with the environment. The key literature on offline RL includes BCQ [35]. which mitigates the extrapolation error using generative model, and CQL [40], which penalizes the overestimation of Q-values for unseen actions. BEAR [1 further addresses the extrapolation error by constraining the learned policy to stay close to the behavior policy. OptiDICE [24] directly estimates the stationary distribution
corrections of the optimal policy, and COptiDICE 23 extends the method to the constrained RL setting. Recent advances have increasingly focused on the use of data generation to improve policy learning. S4RL. [42 shows that surprisingly simple augmentations can dramatically improve policy performance. 43 explores leveraging unlabeled data to improve policy robustness, while [44 proposes survival instincts to enhance agent performance in challenging environments. Counterfactual data augmentation is another promising direction in offline RL 45 46 47 48
highlighting the potential of data generation to significantly improve efficiency and effectiveness.
Safe RL. Safe RL is formulated as constrained optimization to maximize reward performance while satisfying the pre-nefi safety constraints (20 49 50
51
53
54
55
Primal-dual framework
is one common approach to solve safe RL problem 56
57
58
59
60
6
63]. Another line of
work for safe RL is to extend to offline settings, which learn from a fixed dataset to achieve both high
reward and constraint satisfaction [64] (2]. Among them, [26] 65 66 67 tailor online prime-dual-
style algorithms to reduce the -fittr--iistiittioon issue in the offline setting. [28 68) use decision transformer [69] to avoid value estimation and exhibit consistent performances across various tasks.
In addition to these Model-centric approaches, cantricentic approaches, which emphasize improving
or optimizing the quality of the dataset used for model training [70] 71 72 73), have gained more attention in recent studies. While some previous work proposed methods for learning from safe demonstration (28 [24] or relabeling data to achieve conservativeness 28), how systematically
curate datasets for offline safe learning remains a largely unexplored area.
Diffusion Models for RL. Diffusion models have recently gained attention in RL for their capabilities in planning and data generation 775 76 77 78 Specifically, Diffuser (79] uses a diffusion process to plan the entire trajectory in complex environments. [80] extends this to the Decision Diffuser,

---

which conditions the diffusion process specific goals and rewards to improve decision-making SafeDiffuser [8]] and FISOR [I] enhance safety by ensuring the planned trajectories satisfying constraints. Combined with the data augmentation capability of diffusion models, AdaptDiffuser [82
achieves state-of-the-an results on offline RI. benchmarks. 83 proposes Synthetic Experience Replay, leveraging diffusion models to create synthetic experiences for more efficient learning. [84] demonstrates that diffusion models are effective planners and data synthesizers for multi-task RL, showcasing their versatility and efficiency. In this work, we investigate the power of diffusion models for safe RL, where the balance between reward and cost presents further complexities.
3 Problem Formulation
3.1 Safe RL with Constrained Markov Decision Process
We formulate Safe RL problems under the Constrained Markov Decision Process (CMDP) frame- work 885] CMDP M defined by the tuple (S, P, r, (o)., where S E Ihth is the state space, A E R is the action space, P S x A x S + [o, is the transition function, r : S x A x S - IR x + Rgo is the cost function, 7 is the discount factor, and Mo S + [0, 1] is the initial state distribution. Note that this work can also be applied to multiple- constraint tasks, but we use a single-constraint setting for easy demonstration. A safe RL problem is specified by CMDP and a constraint threshold E [0, +o0). Denote E II: S x A + [0, 1] as the policy and r = {(s1 a1 11. c1), ($2, 02 r2,, C2),. as the trajectory. The stationary state-action distribution under the policy и is defined d' (s.a) (1 7) E, Pr(ss,
the reward function, c
8, ag a). The reward and cost returns are defined as R(T) = Σ,r, and C(()) = =E, c. The value function is VF (Ho) = E /B-=n, 1 fff.], E {r,c), c). which the expectation of discounted return under the policy π and the initial state distribution μoo The goal of safe RL is to find the optimal policy x² that maximizes the expectation of reward return while constraining the expectation of cost return to the threshold
arg max [[()). s.t. E,,, [C(T)] S K.
π
3.2 Regularized offline safe RL
For an offline safe RI problem, the agent can only access pre-collected dataset D U,Dr, where D; ~ x is collected by the behavior policy nH E 17 To solve the problem in Eq. we convert the constraint optimization problem into an unconstrained form:
(x",, X*) arg max min [(T, A), J(T,)) = E.--EE-RRTT
A(E,, R(T) ks).
The primal-dual-basee al algorithm solves the optimal policy x² and the dual variable X by updating (x, X) iteratively [86] 23 [66]. offline safe RL tasks, regularization term is usually introduced to prevent the action OOD issue [87), that is, the objective is converted to:
(I A^) = arg max min Jon(TT (T, A), Jan( A) = J (T, A) + (LT,,, x.)
(3)
X
where w >> is constant weight, L(T, π T²) is regularization term and π is the empirical behavior policy and can be viewed as a mixture of { Practically, regularization is formulated as the MSE regularization [88] or the evidence lower bound regularization [35] 41]. In offline safe RL, there are two main challenges: (1) Distribution shift [25] The agent has poor generalizability when facing OOD state-action pairs during online evaluation; and (2) Efficiency-safety performance balancing (28). The agent tends to be over-conservative or aggressive when overestimating or underestimating the safety requirements.
4 Method
In this section, we first identify the safe dataset mismatch (SDM) problem, which leads to performance degradation when solving the regularized offline safe RL objective in Eq. (3) Then we present the proposed OASIS (cOnditionAl diStributIon Shaping) method to solve this problem. In contrast to the model-centric safe RL approaches that focus on optimizing policy update process and network architecture, OASIS is data-centric learning method that aims to improve the quality of the dataset, thus benefiting offline safe RL training. The OASIS method utilizes the diffusion model to realize conditional distribution shaping, solving the challenges mentioned above, thus benefiting offline safe RI training. Following the proposed algorithm, we provide a theoretical guarantee of the safety performance of the policy learned in this paradigm.
3

---

4.1 Safe Dataset Mismatch Problem
The regularized offline safe RL objective in Eq. 3 pushes policy to behavior policy to prevent action OOD is- sues [36]. When given an imperfect dataset, the state- action distribution deviates from the optimal distribution, and the SDM problem arises: if the behavior policy is too conservative with low costs and low rewards, it leads to task efficiency degradation; if the behavior policy is too
500-
400-
reward return
D
aggressive with high costs and high rewards, it leads to safety violations. To further investigate the SDM problem and the effect of dataset distribution on offline safe RL, we define the tempting dataset and conservative dataset, which are based tempting and conservative policies:
Ey (C.
200-
Ep, (C,
Eg (C,R)
100
Van, (C,
Cost limit
Definition 1 (Tempting policy [5T] and conservative pol- icy). The tempting policy class is defined as the set of policies that have higher reward return expectation than the optimal policy, and the conservative policy class is detined as the set of policies that have lower reward and cost return expectations than the optimal policy:
Cost return
Figure 2: D2 is a conservative dataset, and D2 is a tempting dataset. Each point represents (C(T), R(T)) of a trajectory T in the dataset.
11" =-=="= V (NO) > V." (KO)), 11c = { : V (po) < Vi" (Ho), (po) < V. (mo)}. (4)
Intuitively, a tempting policy more rewarding but less safe policy than the optimal one, and conservative policy is with lower cost but less rewarding. According to these policies, we define two types of datasets:
Definition (Tempting and conservative dataset). For an offline dataset D. ~ n,, if n; E B IIIT, then the dataset is tempting: if T E 11 IIC, then the dataset is conservative.
Staying within the tempting dataset distribution results in tempting (unsafe) behavior, while staying within the conservative dataset distribution causes reward degradation. A theoretical analysis of
performance degradation due to the SDM problem is presented in Sec. 4.4 Fig. Lillustrates examples of both conservative and tempting datasets.
is important to note that tempting and conservative datasets are prevalent in offline safe RL since optimal policies are rarely available for data collection. The SDM problem is distinct feature of offline safe RL, indicating that training the policy on either tempting or conservative datasets will violate safety constraints or result in sub-optimality, both of which are unde- sirable. Therefore, addressing the SDM problem is essential for the development of regularized offline safe RL algorithms.
cast Init
cool line
Gosl retum
Coul neturn
(a) Full dataset
(b) Tempting datasee
Tempting-Coo
gho
Tempting
Howar
Cosl
410
4.2 Mitigating the Safe Dataset Mismatch
400
315
Inspired by recent research works on data-centric learning [70), which emphasizes the quality of data used for training the agent, we focus on enhancing the data pipeline to ensure the agent learns effectively instead of modifying the RL algorithm or model ar- chitecture. We propose to use the distribution shaping of the dataset to mitigate the SDM problem, that is, generating a new dataset D, by reshaping the orig- inal data distribution. As shown in Fig. D the key idea is to adjust the dataset distribution towards the optimal distribution under n², reducing the distribution discrepancy and mitigating the SDM problem, thus both mitigating the action OOD issue and balancing efficiency and safety in offline safe RL.
315
30
215
23
60 50
42 36
(c) Reweighting experiment results
Figure 3: (a) Reweighting in the dataset with
comprehensive coverage. (b) Reweighting in the tempting dataset. (c) Performance evalua- tion with different weights and datasets.

---

Dataset
Inverse dynamics
Reward / cost models
Offline Safe RL Agent
(s,a, s' .)))) Initial state
New Dataset (s.a, s' .,,)
Preference
Reward
Cost limit
Condition Condition-guided denoising
Figure 4: OASIS: data-centric approach for offline safe RL. Conditioned on the human preference, OASIS first curates an offline dataset with a conditioned diffusion data generator and learned labeling models, then trains safe RL agents with this generated dataset.
Among the data-centric algorithms, Dataset reweighting (weighted sampling) which assigns different sampling weights data points, is a straightforward way shape the data distribution [39 [9]. In the offline RL domain, researchers proposed methods to assign high weights to data that achieve high
rewards and superior performance many RL tasks 39 89
To validate this idea, we deploy a Boltzmann energy function considering both the reward and the cost for the reweighing strategy solve the problem (see Appendix [B] for details). The experimental results, shown in Fig. 3] validate the effectiveness of this distribution shaping method when the coverage of the dataset complete.
However, for a more general case where we can only access the low-quality dataset (e.g., tempting datasets in Fig. B) simply performing data reweighting does not work well due to the absence of necessary data. Thus, we propose to use a conditional generative model for more ficxible distribution shaping. which generates new data by stitching sub-optimal trajectories for offline training.
4.3
Diffusion Model as Data Generator
To overcome the limitation of reweighing methods, we propose using diffusion models generate the dataset that fits the target cost limit to achieve distribution shaping. In the following, we introduce details of the generator training and dataset generation phases.
Training. In previous works 82
can be viewed as the [S0, St. where is a subsequence of trajectory with
[9). the trajectory planning in offline can
sequential data generation:
length L. Denote (T) and y as the k-step denoising output of the diffusion model and the denoising conditions such as reward and cost returns, respectively. Then the forward diffusion process is to add noise to xy (T) and gradually convert it into Gaussian noise:
[x (T) 1)-1(7)) : NN::::-NN(000 (I)( /1
B, 3e1) = K
(5)
where 3& is pre-defined beta schedule, K is the total denoising timestep. Then the trainable denoising step aims at gradually converting the Gaussian noise back to a valid trajectory:
pe (Ix-1(7) (xx == A (œx-1(7) μe (Ix( (TT), Ex),
(6)
where 8 the trainable parameter. We use simplified surrogate loss 900 for optimization:
Lieenose B ||| eo (xx (T), k)l
In this work, we use the classifier-free guidance [91] for conditional data generation. The condition (T) in Eq. (6) and Eq. 7 is set to (r) [C(r), (C(T).R(() R())]. Thus, the denoising process depends on the target reward and cost returns of the planned subtrajectory. During training time, the diffusion model learns both an unconditional denoising core еs (T) Ø, k) and a conditional denoising core (Ix(T)), ((T), k). We adopt masking [0 for the training to zero out the condition of one training trajectory and categorize it as the Ø class with probability
p < Within the given raw dataset, we also train an inverse dynamics model f:S x S A, and reward and cost models f(s, s'), a(s, s') a,
S Ax S > RR for labeling.
5

---

Generation. After obtaining trained model, the next step is to generate new dataset following the conditions. For diffusion model inference, the denoising core eo (ax (T) ). y, k) is calculated by: Eg (Ix) k)- = ce (Ix(T), @, k) + wa (Eg (zx(T), (e()), ))) - cæ (.x(7), k)), (8)
where Wa > 0 is a constant guidance scale and ye := [č, ft] is the generation condition, we set C S k to align safety preference. For guided generation, we fix the initial state, which means that we replace the initial state of each k-step noised trajectory as an [0] mo(0
After generating one subtrajectory Tg = Fo. we can get the state and action sequence 5g = Tyl:
Algorithm 1: OASIS (dataset generation)
s/,. = T ag = f(gg. s'y), then label the data rg f(5g, ag. sy), č(Sg. Og. s()) Finally, we get a generated dataset Dg {sg. ag: "g.Tg.C. [g] with 1و91 1 transition pairs. With this new dataset, we can further train offline safe RL agents.
Input: Raw dataset D. safety threshold
Output: Generated sub-trajectory Ty
1: Sample a initial state: s ~ D
Get initial noisy sub-trajectory:
F6 = [s, $1. $2-1l.&, 8; N0 I)
Determine the condition ye:
4: for k = K..., 1
86 as the base offiine safe RI algorithm. The process of generating one subtrajectory Ty is summarized in Algorithm ! More details of the implementation are available in Appendix C
In this work, we consider BCQ-Lag 35
5: Calculate (#x, yo k) (
6: Inverse sampling sequence T4-1
7: end for
8: Get actions, rewards, and costs from Zoi
9: Return: trajectory Tg
4.4 Theoretical analysis
We first investigate how the distribution mismatch degrades the policy performance on constraint satisfaction. Suppose that the maximum one-step cost is Cmax = max,, c(s, a). Based on Lemma in [92] and Lemma 2 in (93), the performance gap between the policy Icarned with the dataset D and the optimal policy is bounded by
IV (po) - V. (1o)) - I-7 Drv (d (s)) + 20mox 1-7 Ex (s) (P)r
20max
(9) where d' d)), denote the stationary state distribution of the dataset and optimal policy. The proof is given in Appendix [A.] Therefore, a significant mismatch between the dataset and the optimal policy results in both a substantial state distribution TV distance and large policy shift from the optimal one, which can cause notable performance degradation, especially when the offline RL algorithm enforces the learned policy to closely resemble the behavior policy of the offline data.
Then we provide a theoretical analysis of how our method mitigates this mismatch issue by shrinking the distribution gap, which provides a guarantee of the safety performance of the regularized offline safe RL policy. Let dy (x]y) denote the state marginal of the generated data with condition We first make the following assumptions.
Assumption (Score estimation error of the state marginal). There exists a condition y such that the score function error of the state marginal bounded by
Ex V. log dg (s yy) - V. log d" (s)] S Excure
(10)
where d' (s) is the stationary state distribution induced by the optimal policy n².
This assumption is also adopted in previous work 94 95
For simplicity, we omit the condition y" in the following analysis and use d, (s), dz a) to denote the generated state the state-action distribution with condition y. As we use inverse dynamics fa to calculate actions based on the generated state sequence, the quality of the dataset is also determined by the inverse dynamics. Therefore, we further make the following assumption.
Assumption 2 (Error of inverse policy). The error of action distribution generated by the inverse dynamics is bounded by
Ex)(s) [DRL (-Is) (-1s)) S Sinvs
'This training/generationn formula fixing initial state in the sequence is adopted from Decision Diffuser 80 However, we also found that fixing the initial state is unnecessary for our dataset curation and conditional distribution shaping task. Keeping this condition or not does not significantly impact the generation quality.

---

where Nan (a|s) Bs [xrlf(alsss)) denotes the empirical inverse policy, which is marginal of inverse dynamics over s'.
Then the distance of generated data distribution to the optimal one is bounded as:
Theorem 1 (Distribution shaping error bound). Suppose that the optimal stationary state distribution satisfies that I its score function V. log d' (s) ()) is L-Lipschitz and 2) its second momentum is bounded.
Under Assumption 7 [and? 2 the gap of generated state-action distribution to the optimal stationary
state-action distribution is bounded by
DTv (dg(s, a)lld (s, a)) S (Euoore
Eim/24 (() K),
(12)
where C(d' (s), L, K) represents constant determined by d' (s), L and K.
The proof is given Appendix A.2 Theorem Iindicates that using the proposed OASIS method, we can shape the dataset distribution towards bounded neighborhood of the optimal distribution.
Given the generated data, will then train a regularized offline safe RL policy by Eq. (3). Notice that the regularization term in the objective function in Eq. (3) is equivalent to an explicit policy constraint, and the coefficient assumption on the distance between the learned policy Te and the behavior policy.
the corresponding dual variable. Therefore, we make the following
Assumption 3. Denote the generated dataset as Dg and the corresponding behavior policy as "g. given a fixed coefficient for the policy अ optimized by Eq. (3), there exists Ereg such that
Ed, (s) [DKL
(+1)) S Ereg.
(13)
Based on the above assumptions, we can derive the bound of constraint violation of the policy learned on the offline data generated by OASIS. The proof is given in Appendix (vrendixAA.)
Theorem (Constraint violation bound). For policy n optimized by regularized-uwn based offline safe RL generated dataset Dg. under Assumption 7. policy is bounded as:
and 3
the constraint violation of the trained
Ve (μa) 20max
(0 VK) + (c),
(Ō
Eime/2 2 + y /Eee/2)
(14)
where ((d' (s), L, K) represents a constant determined by d' (s), L and K.
The theoretical analysis sheds insights by answering two questions: (1) Why do we use the diffusion model for conditional distribution shaping, and (2) How does conditional distribution shaping benefit offline safe RL training? Theorem shows that by using the conditional diffusion model data generator, the TV distance between optimal and generated state-action distribution is bounded; Theorem 2 shows that the safety performance of agent trained the generated dataset is guaranteed with OASIS.
Experiment
In the experiments, answer these questions: (1) How does the distribution of the dataset influence the performance of regularized offline safe RL? (2) How does our proposed distribution shaping method perform in offline safe RL tasks? (3) How well does the conditional data generator shape the dataset distribution? To address these questions, we set up the following experiment tasks.
We adopt the continuous robot locomotion control tasks in the public benchmark Bullet Mullee-Safety--e -Safety-Gym [%6] for evaluation, which commonly used in previous works [68] 28 60). We consider two tasks, Run and Circle, and three types of robots, Ball, Car, and Drone. We name the environment as Agent-Task. A detailed description is available in the Appendix
Environments.
Datasets. Our experiment tasks are mainly built upon the offline safe RL dataset OSRL 74
To better
evaluate the tested algorithms with the challenging SDM problem, we create four different training dataset types, full, tempting, conservative, and hybrid. The tempting ing dataset contains sparse safe demonstrations, the conservative dataset lacks rewarding data points, and the hybrid dataset has scarcity in the medium-reward, medium-coss trajectories. We set different cost thresholds for different datasets. A detailed description and visualization of the datasets are available in Appendix B For the main experiments presented in Table I we train on tempting dataset, with threshold 20.
7

---

Seward-llybrd
Reward fdi
Reward-ilybii
060-15
BEARLeg
BEASLS5
Copibice
copub c2
CPg
420
600
150
600
600
250
400
CoobFull
Cool-Conee Coll-Coneevabvvee
Cook-itybiid
Cosk/Nul
Cost: Consgrvaties
Cost-fyond
800
Coyjbkt
copubic
0P0
00
75
75
52
75
(a) Ball-Cirele
(h) Car-Circle
Figure 5: Performance with different datasets and varying constraint thresholds. The visualization of these datasets is available in Appendix B
Baselines and OASIS. For baselines, compared our method with both model-centric and data-
centric baseline methods. Model-cenrtic: (1) 2-Jearning-based algorithms: BCQ-Lag [35 86].
BEAR-Lag [1 [6]. and CPQ [26] (2) Imitation learning: Behavior Cloning (BC) [6] (3) Distribution
correction estimation: COptiDICE [33]. and (4) Sequential modeling algorithms: CDT
(28 and
FISOR [31] Data-cenrtic: (5) Data augmentation: CVAE-BCQL: we train BCQ-Lag agents on the
datasets generated by Conditional Variational Autoencoder (CVAE) (97). The CVAE training set and the dataset generation conditions set as the same with our OASIS, For OASIS implementation, we set three cost conditions Ć' align safety preference and guarantee data coverage. Code is available on our Github repository checkpoints and curated datasets are available on our HuggingFace repository
Metrics. We use the normalized cost return and the normalized reward return as the evaluation metric for comparison in Table 2 The normalized cost is defined Coormalized Cx/Ks where C₂ is the cost return and is the cost threshold. The agent is safe if Coomalizedd S 1. The normalized reward / (M), where r mmax(MM (M) is the maximum empirical reward return for task M within the given dataset. We report the averaged results and standard deviations over 3 seeds for all the quantity evaluations.
computed by Reommaiiedd
Table 1: Evaluation results of the normalized reward and cost. The cost threshold is 1. 1: the higher the reward, the better. J: the lower the cost (up to threshold 1), the better. Bold: Safe agents whose normalized cost is smaller than 1. Gray: Unsafe agents. Blue: Safe agent with the highest reward.
Tasks
Algorithm
Stats
BallRun
CarRun
DroneRun
BallCircle
CarCirele
DroneCirele
reward
0.55 0.23
0.94 + 0.02
0.11
0.73 + 0.05
0.59 + 0.11
0.82 1
BC
cost
2.04 1.32
1.50
0.68
2.53 4
0.85
3.19
reward
0.25 =
0.53
0.33
0.39
0.64
0.00
CPQ
cost
1.34 1.32
1.43 1.82
2.29
0.73
0.12
3.86
COpüDICE
reward
0.00
0.00
0.01
0.73
0.01
0.35 0.02
0.17
0.24
3.87
0.23
0.16
0.12 0.10
reward 0.65+ 0.08
0.66
0.33
0.89 0.02
0.80
0.89 0.04
BEAR-Lag
cost
0.228
8.44
3.22
2.84
0.89
4.03 003=0.5.
reward
0.51
0.06
0.00
0.76
0.79
0.88 + 0.04
BCQ-Lag
cost L.
1.96 +
2.31
5.19 1.08
2.62 0.29
3.25
3.90
reward
0.96
0.11
0.73
0.71
0.008
CDT
cost J.
1.56 1.10
0.67 0.03
0.33
1.03
1.08 0.19
reward
0.03
0.85
0.14
0.28
0.24
0.49 0.05
FISOR
cost L
0.04 0.06
0.25
0.66
0.00
0.25
0.02 0.03
CVAE-BCQL
reward
0.25 0.002
0.00
02.077
0.49
0.60
cost L
1.40
0.00
2.60
1.39
1.77
3.31 1.66
reward
0
0.04
0.08
0.70
0.76 0.03
0.60 0.01
OASIS (ours)
L
0.79 0.37
0.03
0.79 + 0.54
0.45
0.89
0.25 0.10

---

5.1 How can conditional data generation benefit offline safe RL?
Performance degradation with SDM problems. The comparison results on the tempting dataset are presented in Table with the cost threshold ε = 20 before normalization. Results of BC show the mismatch between the behavior policy and the safe policy, as the cost returns significantly violate the safety constraints. The results of BCQ-Lag and BEAR-Lag show this mismatch further influences the regularized-bue based algorithms. leading to constraint violations. This is because the regularization term pushes the policy towards the unsafe behavior policy. The conservative Q function estimation method CPQ, exhibits significant reward degradation in all tasks, which arises from the drawback of the pessimistic estimation methods that learn over-conservative behaviors. COptiDICE also fails to learn safe and rewarding policies, showing that even using distribution correction estimation is not enough to solve the SDM problem. For sequential modeling algorithms, CDT shows poor safety performance and FISOR tends to be over-conservative with poor reward performance. This is because both methods require a large amount of high-quality data while the trajectories with low cost and high reward are sparse in this task. The unsatisfactory performance of these model-centric algorithms further motivates the effective data-centric learning algorithm for offline safe RL.
Performance improvement using OASIS. From Table []] we find that only our method OASIS can learn safe and rewarding policies by mitigating the SDM problem. addition to the results on the tempting dataset, we also pro- vide evaluation results within different types of datasets and constraint thresholds in Fig. Sajand Fig. [b We can observe that most baselines still fail to learn safe policy within different task conditions due to the SDM issue. In contrast, our proposed OASIS method achieves the high- est reward among all safe agents, which shows strength in more general cases.
OASIS
COptiDICE CDT
cost limit
BCQ-Lag
Reward
Cost
60
700
50
40
600
20
10
400
(%)
a (%)
Figure Data efficiency on the Ball- -Cirle
task with a tempting dataset.
High data efficiency of OASIS. In this experiment, we vary the amount of data for offline RL agent training to evaluate data efficiency. The evaluation results are shown in Fig. [ The x-axis a represents the size of the RL agent training dataset. For OASIS, it denotes the size of the generated data. A subsequent BCQ-Lag agent is trained on this generated dataset to obtain the safe RL policy. For the baseline methods, we randomly sample trajectories from the original dataset to construct the training dataset.
The comparison results indicate that we can still learn a good policy using a small amount of high-quality data (a < 2%) generated by OASIS. In contrast, baseline methods show significant performance degradation when the data are sparse as the noisy data is of low quality.
This observation demonstrates that the agent can learn a good policy with high data effi- ciency given high-quality data with minimal safe dataset mismatch (SDM) issues. As OASIS of- fers a solution to shape the dataset distribution, it also reduces the required training dataset size while maintaining good performance, further reaffirming the effectiveness of data-centric approaches in offline safe learning and highlighting that prioritizing quality is essential (70
Table 2: Ablation study on denoising step K
K
10
20
40
Ball-
reward
0.71 = 0.02
0.70
0.71# 0.01
Circle
cost
0.72 + 0.10
0.45
0.14
0.99 0.13
Ball-
reward
0.29
0.29 0.28
Run
cost
0.14
0.00 0.79
5.2 How can OASIS shape the dataset distribution?
Successful distribution shaping. To show the distribution shaping capability of the proposed OASIS and baseline CVAE, we generate the dataset under different conditions and visualize them in Fig. 7 When using different conditions, the expectations of reward and cost of the generated dataset change accordingly. This shows the strong capability of our method in distribution shaping. We also visualize the density of the generated data. In the Car-circle task, the robot receives high rewards when moving along the circle boundary and receives costs when it exceeds the boundaries on both sides, as shown in Fig. 7cc.. The original dataset contains trajectories with various safcty performances.

---

roward conditiion
reward oondition
0.1
0,2b
0.75
9.9
0.1
0:25
0,5
0,75
0.9
Task dataset
OASIS
CVAE
0.9
0.09
0.15
0.41
0.75
0.76
1:26
1:09
1.20
0.89
0.81
comditco
0.75
0.09
0.18
0.45
0.73
0.75
66
0.75
0.80
0:80
68
0.08
0.74
64
0.25
3.74
0.25
0.12
0.1
0.09
0.22
0.50
0.63
0.74
D:1
0.05
0.00
0.09
0.17
0.47
(c) density visualization
Figure 7: (a)(b) Reward and cost performance of the generated data: E [r(s, a)] E [c(s, a)] (s. a) ~ dg. The x-axis and y-axis mean the reward and cost conditions and the values of both conditions and expectations are normalized with the same scale. (c) Visualization of the data density. 1: Car-circle task; 2: The density of the (ar, y) position of the raw dataset; 4: The density of the position (x; y) of the data generated by OASIS under conditions [0.1, 0.5] and [0.75, 0.75]: 5, 6: the density of the position (z, y) of the data generated by CVAE under conditions [0.1, 0.5) and [0.75, 0.75]
When using a low-cost condition, the generated data are clustered within the safety boundary to satisfy the constraints. When using high-reward condition, the generated data points are closer to the circle boundary and receive higher rewards. In contrast, the baseline CVAE cannot successfully incorporate the conditions in data generation, resulting in almost similar datasets with different conditions as shown in Fig. C.. More experiment results including the visualization of generated dataset comparison and corresponding analysis are available in Appendix B.2
5.3 Robust performance against denoising steps
We conduct an ablation study on the key hyperparameter of the proposed OASIS method. The experiment related to the denoising steps K is presented in Table Performance does not change much with different values, which shows the robustness the proposed OASIS method.
Conclusion
In this paper, we study the challenging problem in offline safe RL: the safe data mismatch between the imperfect demonstration and the target performance requirements. To address this issue, we proposed the OASIS method to employ conditional diffusion model to shape the dataset distribution and benefit offline safe RL training. we also conduct extensive experiments to show the superior performance of OASIS in learning a safe and rewarding policy on many challenging offline safe RL tasks. More importantly, our method shows good data efficiency and robustness to hyperparameters, which makes it preferable for applications in many real-world tasks.
addition to the theoretical guarantee of performance improvement,
There are two limitations of OASIS: (1) Offline training takes longer: our method involves prepro- cessing the offline dataset enhance quality, which requires more time and computing resources; (2) Achieving zero-constraint violations remains challenging with imperfect demonstrations. One potential negative social impact is that misuse of our method may cause harmful consequences and safety issues. Nevertheless, we believe that our proposed method can inspire further ceta-centric research in the safe learning community and help to adapt offline RL algorithms real-world tasks with safety requirements.
Acknowledgment
The work is partially supported by Google Deepmind with unrestricted grant. The authors also want acknowledge the support from the National Science Foundation under grants NNS-2047454.
10

---

References
[1] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[2] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[3] Kyle Stachowiez and Sergey Levine. Racer: Epistemic risk-sensitive rl enables fast driving with fewer crashes. arXiv preprint arXiv:2405.04714 2024.
[4] Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters, 2024.
[5] Zhili Zhang, Songyang Han, Jiangwei Wang, and Fei Miao. peatialltemporllawwaare safe multi-agent reinforcement learning of connected autonomous vehicles in challenging scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5574-5580. IEEE, 2023.
[6] Xing Fang, Qichao Zhang, Yinfeng Gao, and Dongbin Zhao. Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference Intelligent Transportation Systems (ITSC), pages 3417-3422. IEEE, 2022.
[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint AXXi::2303.0413,, 2023.
[8] Wenhao Ding, Laixi Shi, Yuejic Chi, and Ding Zhao. Sceing is believing: Robust rein- forcement learning against spurious correlation. Advances in Neural Information Processing Systems, 36, 2024.
[9] Weiye Zhao, Rui Chen, Yifan Sun, Ruixuan Liu, Tianhao Wei, and Changliu Liu. Guard: A safe reinforcement learning benchmark. arXiv preprint arXiv:2305. 33..22335.3.8,1 13681, 2023.
[10] Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, and Wei Zhan. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. arXiv preprint arXiv:2309. 200.02000099 2023.
[1]] Wenhao Ding, Tong Che, Ding Zhao, and Marco Pavone. Bayesian reparameterization of reward-conditioned reinforcement learning with energy-based models. In International Conference on Machine Learning, pages 8053-8066. PMLR, 2023.
[12] Dohyeong Kim, Yunho Kim, Kyungjae Lee, and Songhwai Oh. Safety guided policy optimiza- tion. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2462-2467. IEBE, 2022.
[13] Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, and Guanya Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint 2XXv:2401111583. 2024.
[14] Wenli Xiao, Tairan He, John Dolan, and Guanya Shi. Safe deep policy adaptation. arXiv preprint arXiv:2310.08602. 2023.
[15] Weidong Huang, Jiaming Borong Zhang, Chunhe Xia, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations, 2023.
[16] Zhaorun Chen, Binhao Chen, Tairan He, Liang Gong, and Chengliang Liu. Progressive adaptive chance-constrained safeguards for reinforcement learning. arXiv preprint 23 10. 10.03377 2023.
11

---

[17] Wenjun Zou, Yao Lv. Jie Li, Yujie Yang, Shengbo Eben Li, Jingliang Duan, Xianyuan Zhan, Jingjing Liu, Yaqin Zhang, and Keqiang Li. Policy bifurcation in safe reinforcement learning. arXiv preprint arXiv:2403. 12847, 2024.
[18] Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 833):1295-1300 2023.
[19] Qian Lin, Zongkai Liu, Danying Mo, and Chao Yu. An offline adaptation framework for constrained multi-objective reinforcement learning. arXiv preprint vrXiv:240.009558 2024.
[20] Javier Garcia and Fernando Fernnndez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[21] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-basee control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.
[22] Akifumi Wachi, Xun Shen, and Yanan Sui. A survey of constraint formulations in safe reinforcement learning. arXiv preprint arXiv:2402.0202:. 2024.
[23] Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204. 08957, 2022.
[24] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130 PMLR, 2021.
[25] llya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774 5783. PMLR, 2021.
[26] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 8753-8760, 2022.
[27] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
[28] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.0735.1 2023.
[29] Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang. Bo Yuan, and Dacheng Tao. Saformer: conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv: /2.00,,. 2023.
[30] Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. arXiv preprint arXiv:23066. 2300:2306.00003,, 2023.
[31] Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with reasibility-guidee diffusion model. arXiv preprint 24X1v:2401. 1 10700, 2024.
[32] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1260-11273, 2021.
[33] Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepther- mal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4680-4688 2022.
12

---

[34] Yao Liu, Pratik Chaudhari, and Rasool Fakoor. Budgeting counterfactual for offline rl. Advances in Neural Information Processing Systems, 36, 2024.
[35] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[36] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pages 19967-20025. PMLR, 2022.
[37] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off off-policy reinforcement learning without great exploration. Advances in neural information processing systems. 33:1264-a0 1274, 2020.
[38] Yihang Yao, Zuxin Liu, Zhepeng Cen, Peide Huang, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Gradient shaping for multi-constraint safe reinforcement learning. arXiv preprint arXiv:2312155127, 2023.
[39] Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023.
[40] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 2020.
[41] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[42] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning, pages 907- 917. PMLR, 2022.
[43] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pages 2561 I 25635. PMLR, 2022.
[44] Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 2024.
[45] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics, Advances Neural Information Processing Systems, 01:3976-39900 2020.
[46] Ahmed Aloui, Juncheng Dong, Cat P Le, and Vahid Tarokh. Counterfactual data augmentation with contrastive learning, arXiv preprint arXiv22311.03631 2023.
[47] Chaochao Lu, Biwei Huang. Ke Wang, José Miguel Hernández-Lobule Kun Zhang, and Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-bseed data augmentation. arXiv preprint arXiv:2012.00099. 2020.
[48] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems, 35:18143-18156, 2022.
[49] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
[50] Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. 2022.
arXiv preprint
13

---

[51] Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205. 2022.
[52] Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based safe distributional reinforcement learning for multiple constraints. Advances in neural information processing systems, 36, 2024.
[53] Sheng Xu and Guiliang Liu. Uncertainty-awar constraint inference in inverse constrained reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.
[54] Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. arXiv preprint arXiv:2201.12427 2022.
[55] Dohyeong Kim, Mineui Hong. Jeongho Park, and Songhwai Oh. Scale-invariant gradi- ent aggregation for constrained multi-objective reinforcement learning. arXiv preprint arXiv:2403.000 00282, 2024.
[56] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1-51 2018
[57] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint 180iv: 1074, 2018.
[58] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 19X1::1110001008, 7, 2019.
[59] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances Neural Information Processing Systems, 33:8378-8388 2020.
[60] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 2020.
[61] Zhepeng Cen, Yihang Yao. Zuxin Liu, and Ding Zhao. Feasibility consistent representation learning for safe reinforcement learning. arXiv preprint arXiv:2405.11718. 2024.
[62] Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Off-policy primal-dual safe reinforcement learning. arXiv preprint arXiv:2401.14756 2024.
[63] Dongsheng Ding. Zhengyan Huan, and Alejandro Ribeiro. Resilient constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3412-3420. PMLR, 2024.
[64] Jiayi Guan, Li Shen, Ao Zhou, Lusong Li, Han Hu, Xiaodong He, Guang Chen, and Changjun Jiang. Poce: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26243-26253, 2024.
[65] Nicholas Polosky, Bruno C Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning, pages 1 17801- 17810. PMLR, 2022.
[66] Kihyuk Hong. Yuhang Li, and Ambuj Tewari. A constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 280-288. PMLR, 2024.
algorithm for offline
[67] Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, et al. Voce: Variational optimiza- tion with conservative estimation for offline safe reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
14

---

[68] Zijian Guo, Weichao Zhou, and Wenchao Li. Temporal logic speciication-conditioned decision transformer for offline safe reinforeement learning. arXiv preprint arXiv:2402.172117 2024.
[69] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01334 2021.
[70] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[71] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-callicn datasets. arXiv preprint wxxv:2406.18818 2024.
[72] Suncel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Information Processing Systems, 36, 2024.
[73] Yiming Huang. Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-po dey-point-drrven data synthesis with its enhancement on mathematical reasoning. arXiv preprint orXiv:2403.02333 2024.
[74] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09300 2023.
[75] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223, 2023.
[76] Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. Gta: Generative trajectory aug- mentation with guidance for offline reinforcement learning. arXiv preprint arXiv:2405. 16907, 2024.
[77] Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Dit-Yan Yeung, and Chuang Gan. Adaptive online replanning with diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[78] Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward- directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36, 2024.
[79] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 2XXv:2205.00991. 2022.
[80] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. conditional generative modeling all you need for decision- making? arXiv preprint w.Xiv:2211.156573 2022.
[81] Wei Xiao, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. arXiv preprint arXiv:2306.00148 2023.
[82] Zhixuan Liang, Yao Mu, Mingyu Ding. Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
[83] Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.
[84] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforeement learning. Advances in neural information processing systems, 36, 2024.
15

---

[85] Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48(3):387-
417, 1998.
[86] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133- 9143. PMLR, 2020.
[87] Qian Lin, Chao Yu, Zongkai Liu, and Zifan Wu. Policy-regularized offline multi-objectivv reinforcement learning. arXiv preprint arXiv:200.002244, 2024.
[88] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[89] Zhang- Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harness- ing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306. 13085, 2023.
[90] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[91] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. vrXiv:2207.12598 2022.
arXiv preprint
[92] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments, Advances in Neural Information Processing Systems, 33:15737-15749, 2020.
[93] Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, and Ding Zhao. Learn- ing from sparse offline datasets via conservative density estimation. arXiv preprint arXiv:2401.0881. 2024.
[94] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870- 22882, 2022.
[95] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.1121: 2022.
[96] Sven Gronauer. Bullet-safety- gym: Aframework for constrained reinforcement learning. 2022.
[97] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
[98] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[99] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[100] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171 PMLR, 2021.
16

---

Appendix
Table of Contents
A Proofs
17
A. Proof of 19.
17
Proof Theorem
A.3 Proof Theorem
B Supplementary experiments
19
Trajectory reweighting for distribution shaping
19
B.2
Supplementary data generation comparison results
C Implementation details
19
Environment details
19
Dataset details
21
Algorithm details
22
A Proofs
A.1 Proof of Eq.
By definition of the stationary state-action distribution,
IV_ (mo) - Vo (po))
(15)
1
Bssaaar [c(s,a)] 1E [[(s
(16)
Cmux
ld (d)
d'' (s)
(17)
8,0
20 20max
DTv(d (s.)dd
(18)
20 max
(DTv (s) Drv (a))()))))
(19)
1-7 (dv (s)lld + 20max
2C 2Cmax
1-7 E. DDr (r(ass)) (alss)
(20)
The second inequality holds by triangle inequality for total variation distance.
In general, the stationary distribution of learned policy is in between of the empirical distribution of offline data dº and optimal d' Therefore, we can obtain
IV (po) Vo (mo)) < 2Cmax Dry (d'(d") (d)
(s)) + 20mm
(Prrt (21)
A.2 Proof of Theorem
Proof. By triangle inequality, we first decompose the TV distance between state-action distributions into a state distribution distance and a policy distance,
Drv (dg(s, a)lld" (s.
(22)
= =Drv (s))
(23)
<Drv
Drv (19))
(24)
-Drv ldd
(25)
17

---

We then consider two parts separately.
For the stationary state distribution distance, we suppose that the optimal distribution (s) has T.-Lipschitz smooth score function and a bounded second momentum. Meanwhile, note that the score function in Assumption T is closely related to the denoising model cg
98 99
1 V log d (sly) vi at
co y):
(26)
where co yy is the state marginal the practical denoising model Eq. 8 Therefore, by theorem 2 in 1951, under Assumption have
(s) & V VDKL (d)) 1()))
mg) K V K (27)
where K is the number of denoising timestep, |S| is the dimension of the state space, and my is the second momentum of d' (s). Therefore, aggregating the first two terms in RHS, we have
Drv (dg(s)lld" (s)) S Ô (socore VK) C(d (s), L,),
(28)
where .)...) is a constant w.r.t d", L,, K.
Regarding the policy distance. By Pinsker's inequality,
S / (Tg((s))))
(29)
Meanwhile, since the action is generated by the inverse policy, i.e., Tg Tine. by Assumption have
2 we
(s)(s) [DVv ( s (s)
V (aal(
(30)
S V (o) (D)ul
(31)
Sine /
(32)
where the second inequality holds by Jensen's inequality.
Combining the Eq.
28 and
32
finish the proof of Theorem !
A.3 Proof of Theorem
We start from the Eq. 9 The policy distance be further decomposed into
Drv ( [a]s (a)" (als)) Σ Dry( (=))s( P(]ss) + Drv( (als)
(33)
By Assumption
and and Jensen's 's inequality, we have E. (x) Drv (rv(
(34)
(ar(()
(Dv(((((as)) (a (Pvr (Prv( (a|s)) + Bar V
(35)
<Ea (e) V DKL(
(36)
Ed)(e) (T(l(((()))]
(ar(o) [DRI (aba)
(37)
SV Erog/ / Eim/
(38)
Plug-in Eq. 28 and 38) into Eq. (1 we have
20max
IV. (Po) V. (HO)1
Excore VK) (s), Vemn + v /Eree/2
(39)
1-2
Meanwhile, notice that the optimal policy is constraint satisfactory, i.e.
Vi S k.
(40)
Therefore, we have
20max VCT(p) - K < 1-7
(Ō(EcOom Escorc VK) + (s), + / /inw/2 2 + /rrgg2
(41)
which finishes the proof of Theorem 2
18

---

B Supplementary experiments
B.1 Trajectory reweighting for distribution shaping
In this section, we provide details about the trajectory reweighting experiment presented in Sec. 4.1 Following previous work [9) (9) in offlin RI., adopted datapoint reweighting in policy optimiza- tion, which can be formulated via importance sampling as:
Joff (T, X) P.c.a))D.., (Tar( x, A)] (((ea))D((u( (u((,, To, A)];
(42)
where Foir iT, A) is the objective function after reweighting. In this experiment, we utilize a Boltzmann energy function adopted in [39] for offline RL tasks:
w ((T x exp (())(T)² a2(C(r) - -x)²
(43)
Here w(T) means that all the state-action pairs in one trajectory share the same weight, which is related to the cost and reward returns C(r), R(T). We adopt 0 = a2 in the experiments shown
in Fig.
B.2 Supplementary data generation comparison results
Due to the page limit, omit the visualization of reward and cost distribution using the CVAE method for data generation. Here we provide the results in Fig. g From the reward performance and the cost performance, reconstruction, leading to similar results when setting different conditions. From the trajectory reconstruction results shown Fig. c(). can observe that the generated trajectories are almost the same as the original one. This feature is not desirable for our distribution shaping purpose. In contrast, our method OASIS can successfully shape the distribution shown in Fig. 7 with the strong capability of the diffusion model in the condition coadition-guidedd guided denoising process.
observe that CVAR hardly encode conditions into the data
0.1
0.25
0.5
75
0,9
0,1
0.25
0.5
0.75
0.9
1.5
1.0
Orignnal Data Recoostr etion
0.9-
0.9
0.42
0.40
0.39
0.75
0:52
0.55
0.75
0.43
0.41
0.39
0.39
0.6
36
0.55
0.55
0.5
0.39
0.38
0.37
0.25
0:53
0.55
0.55
0.55
0.25
0.40
0.40
0.38
0.37
0.37
0.2
12
0.1
0.55
0.55
0.55
0.1
0.39
0.38
0.38
0.37
0.36
(8) Reward distribution
(b) Cost distribution
(c) Trajectory visualization
Figure 8: CVAE reconstruction. (a) Reward performance the generated data: E [r(s, a)] (s, a) dg. (b) Cost performance of the generated data: E [c(8, a)], (s, a) dg. In (a) and (b), the x-axis and y-axis mean the reward and cost conditions, and the value both conditions and expectations are normalized to the same scale: [0, 1]: (c) The data reconstruction results using the condition [0.1, 0.5] of 10 sampled trajectories in the dataset.
The comparison results of generated trajectories using OASIS and CVAE in the Brone-Ciren task presented in Fig. This figure illustrates the generation results of OASIS and CVAE under two conditions: low-cost-mediun-ewart and medium--oost-hih--eeaaa reconstruct the trajectories, it fails integrate conditions into the generation process. In contrast, OASIS successfully controls the generated trajectories, avoiding the restricted area when conditioned on low-cost-mediun-rwaan.
Although CVAE can
C Implementation details
C.1 Environment details
Due to the page limit, we omit some descriptions of experiments in the main context. Here we give more details about our experiment tasks. Both the Circle task and the Run task are from a publicly available benchmark (96).
19

---

Original datapoint
OASIS generaled
CVA generared
OASIS
CVAE
OASIS
CVAE
Low-cost, medium reward: = 0.2, = 0.75)
Medium-cost, high rewurd: (c = 0.5, 0.85)
Figure 9: Visualization generated trajectories in the Drone-Circle task.
Circle tasks. The agents are rewarded for running along circle boundary. The reward function is defined as:
yuu + ITy
a 1+1/22 x2 + y² radius) (ro
(44)
where z, y are the positions of the agent with state s', Uz and By are velocities of the agent with state s'. radius is the radius the circle area, and (8t) the specific reward for different robot.
The agent gets cost when exceeding the boundaries. The cost function defined as:
Boundary: c(st) 1[[x] 2lim)
(45)
where Fism is the boundary position.
Run tasks. Agents are rewarded for rumning fast along one fixed direction and are given costs if they across the boundaries exceed a velocity limit. The reward function is defined as:
r(s, a, s') || glla || gll2 + Traboe
(46)
The cost function is defined as:
c(s, a, s') max(1, 1(1y) 9tom) + 1(1122l12 > vtim))
(47)
where Vtim the speed limit, and ytum the y position of the boundary, d, (Ua, Uy is the velocity of the agent with state s', g = [92, Syl is the position a virtual target, of the agent at timestamp t. xt-1 is the Cartesian coordinates of the agent with state s, æt is the Cartesian coordinates of the agent with state s', and Trobor ( ) is the specific reward for the robot.
[Tt Vr. is the position
Agents. We three different robot agents in experiments: Ball, Car, and Drone. The action space dimension, observation space dimension, and the timesteps for these six tasks are shown in Table
Table 3: Environment description
Max timestep
Action space dimension
Observation space dimension
BallRun
100
2
CarRun
200
2
7
DroneRun
200
4
17
BallCircle
200
2
8
CarCirele
300
2
DroneCircle
300
4
18
20

---

C.2 Dataset details
We provide details about the dataset types we presented in the experiment part. The Full, Tempting Conservative, and Hybrid datasets for respectively. All the Tempting datasets associated with results in Table are shown in Fig.
and Car Carclir tasks shown in Fig, 10
Full dataset
Tempting
Conservative
Hybrid
reaard retura
roward raturn
rewerd
lime
Cost return
Cost return
Cost return
Cost return
Figure 10: BallCirele Dataset types. Each point represents (C(T), of a trajectory T in the dataset.
Full dataset
Tempting
Conservative
Hybrid
copcar return
releard neturr
roward
cost limit
cöst limit
cast limit
édiet limit
return
Figure 11: CarCircle Dataset types. Each point represents (CTT),(TT) of trajectory in the dataset.
Ball-Run
Car-Run
Drone-Run
-200
rewoadd
COGT limit
cost limit
cost Imit
Cost return
Cost return
Coor return
Ball-Circle
Car-Circle
Drone-Circle
rewoad retunn
CoGt lime
Cost return
Cost return
Cosl retum
Figure 12: All tempting datasets. Each point represents (C(T), R(T)) of a trajectory 7 in the dataset.
21

---

C.3
Algorithm details
OASIS algorithm training diagram In this work, we use the cosine 3 schedule [0O] to calculate
3.. L.... K. Then let a 3r, and a; = I a, and denote the state dimension as m. With these notations, we show the training process of the OASIS data generator for one epoch in Algorithm
Algorithm 2: OASIS (training)
1: Input: Original Dataset D, predefined Be and og, diffusion core 6g (Zt, y k), learning rate Ir,
loss function +.
2: for each tobbtrajeettryy т; E "D do
3: Extract the states from 7ı: {(50, :15..,11..,,59989 ST)} m]:
4: Get the return conditions y [C, R2 associated with these sub-trajectories:
5: With probability p = 0.25 mask the condition information as: y Ø
6: Get Gaussian Noise noise = ~O # [T.
7:
Randomly sample time E [0, ..., 1};
8:
Calculate the forward sampling state Unoise = ā. + (1 ax)) noise;
9:
Apply initial state condition Pnoise [0] f- 80:
10: Reconstruct noisy sub trajectory Trecon
(Enoise: k);
Minimize the reconstruction loss 0
V (Xnossee Frecon)
12: end for
13: Output: Updated diffusion core eg (#t, y, k))
Network and hyperparameter details. For the dynamics model p. utilize MLP. For the
denoising core, we utilize U-net, which has also been used in previous works [8 80 The U-net
visualized in Fig. [3] The hyperparameters for our method are summarized Table are available in the code.
More details
time MLP
Eemb
Down layers
condition
Yemb
Mid layers
Up layers
Conv layers
y(r)
MLP
X1 (T)
x++1(t)
Figure 13: U-Net structure.
Baseline details. For the baseline methods BC, BCQ -ag, BEAR-Lag, COptiDICE, CPQ, and CDT, we adopt the code base provided in the benchmark (74). For the FISOR method, we use the code provided by the authors BT.
Computing resources. The experiments Processor CPU, 2XNVIDIA RTX A6000 graphics, and 252 GB memory. For one single experiment, OASIS takes about 4 hours with 200, 000 steps train the data generator. It takes about 1.5 hours to train a BCQ-Lag agent on this generated dataset for 200, 000 steps.
with AMD EPYC 7542 32-Core
Table 4: Hyperparameters
Hyperparameters
Value
L (length of subsequence)
32
K (denoising timestep)
20
Batch size
256
Learning rate
3.0e-5
Wa
2.0
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are provided in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
They are provided in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

The assumptions are provided in Section 4.4 and the proof is provide in Section[A]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
They are provided in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[Yes]
Justification:
The code is provided the anonymous link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guuii wwwtii/gguie///eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are provided in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: They are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources are provided in Appendix More details about exper- iment results such reweighting for distribution shaping also provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp://neurips http://ppp...p..oo...cceeeeeaaaa
Answer: [Yes]
Justification: Provided in the openreview form.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
They are provided in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not contain pre- pre-trained language models, image generators, scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The codebase utilize for baseline comparison is under Apache License 2.0. Related materials are cited in our code (See the README file in the code).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "gaio0W5dgY", "reviewer": "Reviewer_CUpS", "summary": "This paper proposes OASIS, which uses a conditional diffusion model to reshape the dataset distribution and achieve effective offline safe RL learning. Theoretical analysis gives the error upper bound of distribution reshaping and constraint violation upper bound. A large number of experiments show that the proposed algorithm has significantly improved compared to offline safe RL baselines.", "questions": "***Q1:*** Compared with CVAE, OASIS shows that the conditional diffusion model has a better ability to generate according to the condition information, as shown in Figure 7(c), but this example is a bit simple. Can the author show more comparisons of the two similar to Figure 7(c)? For example, add the OASIS generation results to Figure 8(c)?\n\n***Q2:*** When showing the effectiveness of the newly generated dataset, in addition to showing the distribution of the generated state like Figure 7(c), it is also necessary to verify whether the annotations of the inverse dynamics model and reward & cost models are accurate. Can the author supplement the accuracy of these three models?\n\n***Q3:*** By comparing the results of OASIS and CDT, I came to a conclusion: in offline safe RL, both methods conditioned on cost and reward, generating data is more effective than generating policy. I wonder if the author agrees with this conclusion? This conclusion seems to be uncommon in the field of offline RL, or it may be because I am not familiar with distribution shaping methods.", "limitations": "See the weaknesses and questions sections.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The logic is clear and the paper is easy to understand.\n- The theoretical analysis is sufficient.\n- A large number of experiments prove the effectiveness of the proposed algorithm.", "weaknesses": "***W1:*** Some baselines are not introduced in related work, such as CVAE, FISOR, etc. This may cause difficulties in understanding.\n\n***W2:*** Typos, such as line 173 \"reweighing\" -> \"reweighting\"; wrong citations, such as line 290 \"COptiDICE [17]\" -> \"COptiDICE [16]\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for the valuable feedback. We are glad to know that the reviewer recognizes the clear logic, sufficient theoritical analysis, and experiments proving the effectiveness. We provide our response to the questions and concerns below.\n\n> W1: Some baselines are not introduced in related work, such as CVAE, FISOR, etc. This may cause difficulties in understanding.\n\nWe thank the reviewer for pointing this out. We have added the following sentences to the related work section. FISOR identifies the largest feasible region where agents can operate safely, and optimizes for high rewards within this region and minimizes risks outside it with a diffusion policy. CVAE is an extension of the standard VAE that incorporates additional conditional information, such as class labels or attributes, into data generation. This enables more controlled and specific output generation.\n\n> W2: Typos.\n\nThanks for your careful review. we fixed these typos in our revised version.\n\n> Q1: Compared with CVAE, OASIS shows that the conditional diffusion model has a better ability to generate according to the condition information, as shown in Figure 7 c, but this example is a bit simple. Can the author show more comparisons of the two similar to Figure 7 c? For example, add the OASIS generation results to Figure 8 c?\n\nWe present additional comparisons in Figure R-1 of the supplementary PDF file. This figure illustrates the generation results of OASIS and CVAE under two conditions: medium-cost-high-reward and low-cost-medium-reward. While CVAE can reconstruct the trajectories, it fails to integrate conditions into the generation process. In contrast, OASIS successfully controls the generated trajectories, avoiding the restricted area when conditioned on low-cost-medium-reward.\n\n> Q2: When showing the effectiveness of the newly generated dataset, in addition to showing the distribution of the generated state like Figure c, it is also necessary to verify whether the annotations of the inverse dynamics model and reward & cost models are accurate. Can the author supplement the accuracy of these three models?\n\nWe provided the scaled MSE loss values of inverse dynamics, reward and cost models in the following tables. We provide the inital value of the loss when the training begins for a comparison. The error of these models is small.\n\n**Inverse dynamics**\n||BallCircle| CarCircle | DroneCircle | BallRun | CarRun | DroneRun |\n|- | - | - | - | -| - | - |\n|init | 1.08 | 1.04 | 1.58 | 0.68 | 1.14 | 1.03 |\n|final | 0.037 | 0.18 | 0.041 | 0.056 | 0.26 | 0.015 |\n\n**reward model**\n|| BallCircle| CarCircle | DroneCircle | BallRun | CarRun | DroneRun |\n|- | - | - | - | - | - | - |\n|init | 1.86 | 7.89 | 1.54 | 3.96 | 1.96 | 1.92 |\n|final | 0.009 | 0.05 | 0.004  | 0.013 | 0.002 | 0.017 |\n\n**cost model**\n|| BallCircle| CarCircle | DroneCircle | BallRun | CarRun | DroneRun |\n|- | - | - | - | - | - | - |\n|init | 5.59| 5.29  | 5.41 | 4.09 | 4.42 | 5.80 |\n|final | 0.13 | 0.15| 0.15 | 0.32 | 0.38 | 0.27 |\n\n\n> Q3: By comparing the results of OASIS and CDT, I came to a conclusion: in offline safe RL, both methods conditioned on cost and reward, generating data is more effective than generating policy. I wonder if the author agrees with this conclusion? This conclusion seems to be uncommon in the field of offline RL.\n\nIn the offline RL setting, the distribution of datasets matters a lot. If the data quality is low, i.e. the dataset is imbalanced, and most contain unsafe or low-rewarding demonstrations, our OASIS, which shapes the dataset distribution towards the target distribution, is more effective than directly generating safe policy by conditional sequential modeling such as CDT. The importance of distribution shaping for imbalanced dataset in offline RL has also been discussed in some related works [1, 2], where the authors proposed sampling strategies to solve this issue.\n\nWe acknowledge that the performance comparison between general policy generation and data generation in offline safe RL remains an open question. This represents a fascinating area for future research.\n\n---\n[1] Hong, Zhang-Wei, et al. \"Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets.\" NeurIPS 2023.\n\n[2] Hong, Zhang-Wei, et al. \"Harnessing mixed offline reinforcement learning datasets via trajectory weighting.\" ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1HPlxzR9G", "reviewer": "Reviewer_fA23", "summary": "Offline Safe Reinforcement Learning (RL) is used to learn policies satisfying cost constraints from a given dataset. This proves to be a challenge when the dataset is biased in a certain way. This paper introduces a method to use the offline training dataset to capture the environment using a diffusion model that can generate data conditioned on our cost constraints and performance objectives. Experimental results show the approach generates data that better fit the constraints and outperform alternatives like dataset reweighting.", "questions": "1. How are the initial states decided to generate the data from the diffusion model? Are they the same as the initial states of the training dataset or randomly sampled from the training dataset trajectories?\n2. How long is inference time for the OASIS model i.e., data generation time? Is that included in the training time (L999, App C.3)?\n3. How do we decide the “target” cost given the cost threshold to get the best performance? Is the study done in Fig. 7 required for each setting or is there a good heuristic to set these targets? How expensive is this study (Fig. 7) over target values (i.e., multiple inference runs)? On a related note, how were these targets set for Table 1 on OASIS and the compared baselines?\n4. Is it right to say the primary reason for the success of the proposed method over the diffusion baselines (like FISOR), a more realistic conditional generative model that yields optimal generated data satisfying our cost constraints?\n5. Are the learned labeling models (inverse dynamics, reward, and cost) only used for labeling the data generated from the diffusion model?\n6. In Fig. 6, are all models using data generated by the same diffusion model? OASIS handles the diffusion model training and data generation with BCQ-Lag as the actual Offline RL policy learning. This makes me a little confused. How are the curves related in Fig. 6? \n7. Why is hybrid an interesting dataset setting vs. full?\n8. Typos:\n    - L224 (Sec 4.4) Theoretical\n    -  L973 (App C.1) min instead of max", "limitations": "Diffusion models being computationally heavy (see Weakness)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well organized and clearly shows the strengths of diffusion models to generate offline data with the given cost/performance objectives.\n- Theoretical results show that the policies learned from the generated data are cost constrained given some reasonable assumptions on the model.\n- Extensive comparisons are made to different SoTA baselines in offline RL with and without data generation.", "weaknesses": "- Data generation and training can be a slow process due to the use of diffusion models which are known to be computationally heavy.\n- Selecting hyperparameters such as number of denoising steps might vary results significantly. For example, Table 2 has greatly varying policy costs for different values of $K$ (albeit similar performances).\n- While results are mostly consistent, it is hard to say when the proposed method prefers to act more conservatively with lower cost (or more riskily i.e., higher reward). This is reflected in the reward and costs in Table 1 (e.g., CarRun). A study on a hyperparameter change (apart from cost/return targets) to control this balance would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the insightful suggestions and the praise of our theoretical results and experimental performance. We provide our response to the reviewer's comments and questions as follows:\n\n> W1: Data generation and training can be a slow process.\n\nWe agree that generating new data could cause extra computational cost to the training process. However, since we focus on the offline RL setting, this extra cost only occurs before online inference, which we believe is still acceptable.\n\n> W2: Selecting hyperparameters such as the number of denoising steps might vary results significantly. For example, Table 2 has greatly varying policy costs for different values of $K$.\n\nFor offline safe RL tasks, agents sometimes achieve similar rewards while obtaining different cost values when the cost limit is small and the cost signal is binary. The difference in cost performance in Table 2, where the threshold is set at $\\kappa=20$, may come from this. Nevertheless, our method consistently outperforms the baselines in terms of safety constraint satisfaction and reward maximization.\n\nWe also tested our method with moderate thresholds ($\\kappa=30, 40$) and show the results below. We observe that the cost performance of our method is more consistent under this setting. (All the values are normalized.)\n\n**BallCircle ($\\kappa=30$):**\n\n|$K$|10|20|40|\n|-|-|-|-|\n|Reward|$0.793\\pm0.014$|$0.777\\pm0.017$|$0.791\\pm0.010$|\n|Cost|$0.877\\pm0.182$|$0.857\\pm0.123$|$0.970\\pm0.099$|\n\n**BallCircle ($\\kappa=40$):**\n\n|$K$|10|20|40|\n|-|-|-|-|\n|Reward|$0.809\\pm0.007$|$0.783\\pm0.015$|$0.808\\pm0.014$|\n|Cost|$0.845\\pm0.121$|$0.740\\pm0.059$|$0.819\\pm0.041$|\n\n**BallRun ($\\kappa=30$):**\n\n|$K$|10|20|40|\n|-|-|-|-|\n|Reward|$0.317\\pm0.025$|$0.341\\pm0.031$|$0.318\\pm0.032$|\n|Cost|$0.890\\pm0.514$|$0.739\\pm0.385$|$0.797\\pm0.361$|\n\n**BallRun ($\\kappa=40$):**\n\n|$K$|10|20|40|\n|-|-|-|-|\n|Reward|$0.374\\pm0.019$|$0.392\\pm0.047$|$0.364\\pm0.011$|\n|Cost|$0.896\\pm0.137$|$0.825\\pm0.336$|$0.963\\pm0.176$|\n\n> W3: While results are mostly consistent, it is hard to say when the proposed method prefers to act more conservatively with lower cost. A study on a hyperparameter change (apart from cost/return targets) to control this balance would be helpful.\n\nIn our method, the balance between reward and cost of generated data for the same model is only controlled by the input condition, i.e., the target reward/cost.\n\nTo further study the influence of other hyperparameters on the learned generative model, we provide more ablation experiments on the sequence length $L$. We use the Ball-Circle task with a threshold of $\\epsilon=30$ and show results on the testing dataset in the following table. We find that our method is not sensitive to $L$ within the tested range.\n\n|$L$|32|48|64|\n|-|-|-|-|\n|Reward|$0.799\\pm0.014$|$0.803\\pm0.005$|$0.785\\pm0.014$|\n|Cost|$0.890\\pm0.132$|$0.926\\pm0.090$|$0.798\\pm0.070$|\n\n> Q1: How are the initial states decided to generate the data from the diffusion model?\n\nThey are randomly sampled from the training dataset trajectories.\n\n> Q2: How long is the inference time for the OASIS model? Is that included in the training time?\n\nThe generation time in all experiments is less than 1 minute with an A6000 GPU. Taking Ball-Circle as an example, we train the diffusion model of OASIS with a sequence length of $L=32$. During generation, we randomly sample 2000 states from the training dataset. After one-time generation, we obtain a dataset with $2000 \\times (32-1) = 62000$ transitions. This process only takes 11 seconds. The generation time is not included in the training time presented in L999, App C.3.\n\n> Q3: How do we decide the “target” cost given the cost threshold to get the best performance? Is the study done in Fig. 7 required for each setting or is there a good heuristic to set these targets? How expensive is this study (Fig. 7)? On a related note, how were these targets set for Table 1 on OASIS and the compared baselines?\n\nWe first set the “target” cost as the normalized cost threshold, then adjust the conditions according to the results in Figure 7 to find a good condition that has a high reward while adhering to the cost limit. It is a one-time inference for one set of conditions, which only takes about 15 seconds in total. Meanwhile, the performance of each condition is evaluated by the learned reward and cost and does not require online data. For CDT, which requires an additional reward condition, the reward conditions are adopted from the source code released by the authors.\n\n> Q4: Is it right to say the primary reason for the success of the proposed method over the diffusion baselines (like FISOR), is a more realistic conditional generative model that yields optimal generated data satisfying our cost constraints?\n\nIn the offline RL setting, the distribution of the dataset matters a lot. If the data quality is low, i.e., the dataset is imbalanced and mostly contains unsafe or low-rewarding demonstrations, our OASIS method, which shapes the dataset distribution towards the target distribution, is more effective than baselines (e.g., FISOR) directly modeling the policy by conditional generative model.\n\n> Q5: Are the learned labeling models (inverse dynamics, reward, and cost) only used for labeling the data generated from the diffusion model?\n\nYes, they are only used to label the data generated from the diffusion model.\n\n(continued)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFFMrbHpsJ", "reviewer": "Reviewer_XmVn", "summary": "This paper discusses the safe dataset mismatch (SDM) problem, highlighting how low-reward or unsafe samples in datasets can harm offline safe RL. Conditional distribution shaping (OASIS) is proposed to mitigate this problem by generating high-reward and safe samples via diffusion models and promoting general offline safe RL algorithms with the generated data. This paper evaluates OASIS through extensive experiments across various safe RL tasks and different types of datasets with varying data distributions.", "questions": "1. What is the size of the generated dataset $D_g$? I would like to confirm whether you directly train the policy on $D_g$ or a mixture of $D_g$ and the original datasets?\n2. How do you calculate the reward condition $\\hat R$ based on the given cost $\\hat C$?\n3. Why are the thresholds in Figure 5 different for different types of datasets? Did you use the same threshold when constructing different types of datasets? This is a concern because one tempting dataset constructed under the thresholds 20 may be considered tempting for one threshold 20 but not for another threshold 60.\n4. In Figure 6, how did you obtain the performance of baseline methods (e.g., CDT) under a specific $\\alpha$? Did you use the data of size $\\alpha$ generated by OASIS for these baselines? It would be helpful to provide more details about the data efficiency experiments.", "limitations": "The limitation are discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated. The influence of imbalance and biased data on offline safe RL is an important but underexplored problem. Solving the proposed problem through diffusion-based data generation is intuitive, reasonable and novel.\n- The empirical evaluation and ablation studies are comprehensive, carefully demonstrating the significance of the SDM problem and the effectiveness of the proposed approach.\n- Theoretical analysis provides certain guarantees for the proposed approach.\n- The paper is well written and well organized.", "weaknesses": "- Some technical and experimental details are a bit confusing. See Questions 1, 2, 3, 4.\n\n- Assumptions 1 and 2 seem kind of idealized to directly bound the distribution and policy discrepancy, since $\\epsilon_{score}$ and $\\epsilon_{inv}$ cannot be directly calculated or estimated. Admittedly, analyzing the distribution and policy discrepancy based on diffusion-generated data may be difficult and beyond the scope of this work. Maybe more discussion and explanation could help justify these assumptions.\n\n- Directly excluding mismatched data from training datasets could be an another intuitive approach for SDM problem. So it would be better to discuss the performance if tempting and conservative data were directly excluded from the full datasets, utilizing the remaining datasets (i.e., Full dataset - Hybrid dataset) for training. A comparison between these two approaches (i.e., adding matched data vs. reducing mismatched data) would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for recognizing the novelty, comprehensive experiment validation, and theoretical analysis contribution of our work. We provide our response to the comments below:\n\n> W2: Assumptions 1 and 2 seem kind of idealized to directly bound the distribution and policy discrepancy, since and cannot be directly calculated or estimated. Admittedly, analyzing the distribution and policy discrepancy based on diffusion-generated data may be difficult and beyond the scope of this work. Maybe more discussion and explanation could help justify these assumptions.\n\nAssumption 1 and 2 bound the function approximation error of diffusion model and inverse dynamics model, which depends on the implementation (e.g., network architecture, learning rate, etc.). As you mentioned, we also believe it may be out of the scope to directly analyze this type of error. Meanwhile, those assumptions bound the expectation of error instead of maximal error and they are also adopted by previous work [1,2]. Therefore, we argue that the assumptions are not overly idealized but necessary to derive theoretical analysis.\n\n> W3: Directly excluding mismatched data from training datasets could be an another intuitive approach for SDM problem. So it would be better to discuss the performance if tempting and conservative data were directly excluded from the full datasets, utilizing the remaining datasets (i.e., Full dataset - Hybrid dataset) for training. A comparison between these two approaches (i.e., adding matched data vs. reducing mismatched data) would be valuable.\n\nAccording to the reviewer's suggestion, we conduct the experiments with these datasets in the Ball-Circle task with threshold 40. The used hybrid, hybrid-R(rewarding only), hybrid-S(safe-only), and hybrid-Com(complementary) datasets are visualized in Figure R-4 presented in the supplementary PDF file. We can observe that when using the safe-only dataset, the learned policy is conservative with low cost and low reward. When using the rewarding-only dataset, the learned policy is tempting with high reward and high cost. When training with the complement of the hybrid dataset, we can get a better safe performance compared to using the hybrid dataset. However, since it still contains a large amount of imperfect demonstration with low reward, the reward performance is not satisfactory compared to our OASIS.\n\n|Stats|OASIS|BCQ-Lag (hybrid)|BCQ-Lag (hybrid-R)|BCQ-Lag (hybrid-S)|BCQ-Lag (hybrid-Com)|\n|-|-|-|-|-|-|\n|Reward|$684.87\\pm9.66$|$672.99\\pm40.54$|$780.91\\pm15.41$|$594.99\\pm27.75$|$646.98\\pm70.60$|\n|Cost ($\\kappa=40$)|$32.10\\pm3.27$|$52.11\\pm1.84$|$63.39\\pm2.00$|$24.17\\pm5.06$|$33.77\\pm9.23$|\n\n> Q1: What is the size of the generated dataset? I would like to confirm whether you directly train the policy on or a mixture of and the original datasets?\n\nThe number of transition pairs of the original tempting dataset and the generated datasets used in Table 1 is shown below. Each value presents the number of transition pairs.\n\n||BallCircle|CarCircle|DroneCircle|BallRun|CarRun|DroneRun|\n|-|-|-|-|-|-|-|\n|full dataset|177200|435000|576243|94000|130200|395099|\n|tempting dataset|126200|302100|426705|73900|29600|249432|\n|OASIS generated $D_g$|62000|62000|155000|62000|62000|155000|\n\nFor OASIS, we directly train RL agents on the generated datasets.\n\n> Q2: How do you calculate the reward condition based on the given cost?\n\nWe select the conditions with similar study in Figure 7 to find a good condition that has a high reward while sticking to the cost limit.\n\n> Q3: Why are the thresholds in Figure 5 different for different types of datasets? Did you use the same threshold when constructing different types of datasets? This is a concern because one tempting dataset constructed under the thresholds 20 may be considered tempting for one threshold 20 but not for another threshold 60.\n\nWe change the thresholds in Figure 5 to show that our method can perform well under varying threshold conditions. We used different thresholds to construct different types of datasets to make them tempting/conservative/hybrid. The visualization of the dataset and corresponding threshold is also available in Appendix C.2.\n\nAccording to the reviewer's suggestion, we also conducted experiments with the same thresholds on different datasets. The datasets and the experiment results are presented in Figures R-2 and R-3 in the provided PDF file. We can observe that when setting threshold=40, in all tested datasets including tempting, conservative, and hybrid, our method OASIS exhibits the best performance, achieving the highest reward among the safe agents.\n\n> Q4: In Figure 6, how did you obtain the performance of baseline methods (e.g., CDT) under a specific $\\alpha$? Did you use the data of size generated by OASIS for these baselines? It would be helpful to provide more details about the data efficiency experiments.\n\nWe apologize for any confusion. In Figure 6, $\\alpha$ represents the size of the RL agent training dataset. For OASIS, it denotes the size of the generated data. A subsequent BCQ-Lag agent is trained on this generated dataset to obtain the safe RL policy. For the baseline methods, we create the training dataset by randomly sampling $\\alpha\\%$ of trajectories from the original dataset for RL training. In this experiment, we aim to demonstrate that, for offline safe RL, the agent can learn a good policy in a data-efficient manner if the dataset has minimal safe dataset mismatch (SDM) issues. OASIS offers a solution to shape the dataset distribution, which can reduce the required dataset size for RL training while maintaining good performance. We have added a detailed explanation and analysis of this experiment in the revision.\n\n---\n\n[1] Holden Lee, et al. Convergence for score-based generative modeling with polynomial complexity.\n\n[2] Sitan Chen, et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We thank you all for your careful review and valuable feedback. In addition to addressing each reviewer’s comments, we would like to highlight the new examples and experiments during the rebuttal phase. The figures we refer to can be found in the attached PDF file.

1. **Additional Visualization of Comparison Between OASIS and CVAE**
In response to reviewer CUpS, we present additional comparison results of trajectories generated by OASIS and CVAE in Figure R-1.

2. **Accuracy of the Learned Inverse Dynamics, Reward, and Cost Models**
In response to reviewer CUpS, we present the accuracy of these models.

3. **Additional Ablation Experiments**
In response to reviewer fA23, we present more results and analysis of our ablation study.

4. **Additional Experiments for Distribution Shaping Methods**
In response to reviewer XmVn, we conduct experiments to show the performance of the distribution shaping method by adding matched data and reducing mismatched data.

5. **Additional Experiments with Different Types of Datasets**
In response to reviewer XmVn, we construct different types of datasets (tempting/conservative/hybrid) under the same threshold and provide supplementary experiments.

We sincerely appreciate your time, attention, and valuable feedback.

Best regards,

Authors of Submission 12456

Meta-review: One of the reviewers for this paper did not participate in the review process. The paper received relatively high scores: Reviewer CUpS (Score 6, Confidence 2), Reviewer fA23 (Score 7, Confidence 4), and Reviewer XmVn (Score 6, Confidence 4). As the AC, I took my responsibility seriously and carefully reviewed the paper. After the rebuttal, most of the reviewers' concerns were addressed. I thoroughly read through the communication between the authors and the reviewers, noting that the authors need to include some additional experiments in the revised version.

I also carefully reviewed the content of the paper. The main focus of the paper is the application of safe RL in the offline setting. However, I find the experimental setup to be overly simplistic, as it is limited to only two tasks: Run and Circle. Additionally, the introduction of the diffusion model leads to inefficiencies and hyperparameter issues, which the authors did not adequately resolve. Reviewers also pointed out that the theoretical assumptions are overly idealized.

Considering all the reviewers' opinions and my own thorough review as AC, this paper is accepted as a poster.
Decision: Accept (poster)