Title: OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Safe Reinforcement Learning

Offline Reinforcement Learning
Abstract: Offline safe reinforcement learning (RL) aims to train a policy that satisfies con- straints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired safe and rewarding performance. In this paper, we mitigate this issue from a data-centric perspective and introduce OASIS (cOnditionAl diStributIon Shaping), a new paradigm in offline safe RL designed to overcome these critical limitations. OASIS utilizes a conditional diffusion model to synthesize offline datasets, thus shaping the data dis- tribution toward a beneficial target domain. Our approach makes compliance with safety constraints through effective data utilization and regularization techniques to benefit offline safe RL training. Comprehensive evaluations on public benchmarks and varying datasets showcase OASIS’s superiority in benefiting offline safe RL agents to achieve high-reward behavior while satisfying the safety constraints, out- performing established baselines. Furthermore, OASIS exhibits high data efficiency and robustness, making it suitable for real-world applications, particularly in tasks where safety is imperative and high-quality demonstrations are scarce. More details are available at the website https://sites.google.com/view/saferl-oasis/home.

OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning (RL) aims to train a policy that satisfies constraints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired safe and rewarding performance. In this paper, we mitigate this issue from a data-centric perspective and introduce OASIS (c O ndition A l di S tribut I on S haping), a new paradigm in offline safe RL designed to overcome these critical limitations. OASIS utilizes a conditional diffusion model to synthesize offline datasets, thus shaping the data distribution toward a beneficial target domain. Our approach makes compliance with safety constraints through effective data utilization and regularization techniques to benefit offline safe RL training. Comprehensive evaluations on public benchmarks and varying datasets showcase OASIS's superiority in benefiting offline safe RL agents to achieve high-reward behavior while satisfying the safety constraints, outperforming established baselines. Furthermore, OASIS exhibits high data efficiency and robustness, making it suitable for real-world applications, particularly in tasks where safety is imperative and high-quality demonstrations are scarce. More details are available at the website https://sites.google.com/view/saferl-oasis/home.

1 Introduction

Offline Reinforcement Learning (RL), which aims to learn high-reward behaviors from a pre-collected dataset [1, 2], has emerged as a powerful paradigm for handling sequential decision-making tasks such as autonomous driving [3, 4, 5, 6], and robotics [7, 8, 9, 10, 11]. Although standard offline RL has achieved remarkable success in some environments, many real-world tasks cannot be adequately addressed by simply maximizing a scalar reward function due to the existence of various safety constraints that limit feasible solutions. The requirement for safety , or constraint satisfaction, is particularly crucial in RL algorithms when deployed in real-world tasks [12, 13, 14, 15, 16, 17, 18, 19].

To develop an optimal policy within a constrained manifold [20, 21], offline safe RL has been actively studied in recent years, offering novel ways to integrate safety requirements into offline RL [22]. Existing research in this area incorporates techniques from both offline RL and safe RL, including the use of stationary distribution correction [23, 24], regularization [25, 26], and constrained optimization formulations [27]. Researchers have also proposed the use of sequential modeling methods, such as the decision transformer [28, 29] and the decision diffuser [30, 31] to achieve advantageous policies and meet safety requirements.

Although these methods show promise, it is difficult to handle state-action pairs that are absent from the dataset, which is known notably as out-of-distribution (OOD) extrapolation issues [31, 32, 33, 34, 35, 36, 37]. To solve this, many works utilize regularization methods to push the policy toward behavior policy to achieve pessimism [35, 36]. However, this approach worsens the situation when the dataset is imbalanced and biased: regularization by imperfect demonstrations such as datasets

composed predominantly of low-reward data or containing few safe trajectories collected using unsafe behavior policies. This regularization also leads to another challenge: striking the optimal balance between learning objectives such as task utility efficiency and safety requirements, leading to reward degradation or aggressive behavior [28, 38, 39].

To address these challenges, we introduce a datacentric learning paradigm in offline safe RL, OASIS (c O ndition A l di S tribut I on S haping), which focuses on improving the training dataset quality by steering the offline data distribution to a beneficial target domain as shown in Fig. 1. OASIS distills knowledge from the imperfect dataset, and generates rewarding and safe data using a conditional diffusion model according to the safety preference to benefit offline safe RL training. This data-centric approach is parallel to and compatible with general model-centric offline safe RL

reward

cost

cost limit

reward

cost

cost limit

algorithms which emphasize improvements to the learning algorithm and model architecture. The key contributions are summarized as follows.

Identification of the safe dataset mismatch (SDM) problem in offline safe RL. We identify the mismatch between the behavior policy and the target policy and investigate the underlying reasons for performance degradation with this condition.

Introduction of the OASIS method to address the SDM problem. To the best of our knowledge, this is the first successful application of a distribution shaping paradigm within offline safe RL. Our theoretical analysis further provides insights on performance improvement and safety guarantees.

A comprehensive evaluation of our method across various offline safe RL tasks. The experiment results demonstrate that OASIS outperforms baseline methods in terms of both safety and task efficiency for varying tasks and datasets.

2 Related Work

Offline RL. Offline RL addresses the limitations of traditional RL, which requires interaction with the environment. The key literature on offline RL includes BCQ [35], which mitigates the extrapolation error using a generative model, and CQL [40], which penalizes the overestimation of Q-values for unseen actions. BEAR [41] further addresses the extrapolation error by constraining the learned policy to stay close to the behavior policy. OptiDICE [24] directly estimates the stationary distribution corrections of the optimal policy, and COptiDICE [23] extends the method to the constrained RL setting. Recent advances have increasingly focused on the use of data generation to improve policy learning. S4RL [42] shows that surprisingly simple augmentations can dramatically improve policy performance. [43] explores leveraging unlabeled data to improve policy robustness, while [44] proposes survival instincts to enhance agent performance in challenging environments. Counterfactual data augmentation is another promising direction in offline RL [45, 46, 47, 48], highlighting the potential of data generation to significantly improve efficiency and effectiveness.

Safe RL. Safe RL is formulated as a constrained optimization to maximize reward performance while satisfying the pre-defined safety constraints [20, 49, 50, 51, 52, 53, 54, 55]. Primal-dual framework is one common approach to solve safe RL problem [56, 57, 58, 59, 60, 61, 62, 63]. Another line of work for safe RL is to extend to offline settings, which learn from a fixed dataset to achieve both high reward and constraint satisfaction [64, 29]. Among them, [26, 65, 66, 67] tailor online prime-dualstyle algorithms to reduce the out-of-distribution issue in the offline setting. [28, 68] use decision transformer [69] to avoid value estimation and exhibit consistent performances across various tasks. In addition to these Model-centric approaches, Data-centric approaches, which emphasize improving or optimizing the quality of the dataset used for model training [70, 71, 72, 73], have gained more attention in recent studies. While some previous work proposed methods for learning from safe demonstration [28, 74] or relabeling data to achieve conservativeness [28], how to systematically curate datasets for offline safe learning remains a largely unexplored area.

Diffusion Models for RL. Diffusion models have recently gained attention in RL for their capabilities in planning and data generation [75, 76, 77, 78]. Specifically, Diffuser [79] uses a diffusion process to plan the entire trajectory in complex environments. [80] extends this to the Decision Diffuser,

which conditions the diffusion process on specific goals and rewards to improve decision-making. SafeDiffuser [81] and FISOR [31] enhance safety by ensuring the planned trajectories satisfying constraints. Combined with the data augmentation capability of diffusion models, AdaptDiffuser [82] achieves state-of-the-art results on offline RL benchmarks. [83] proposes Synthetic Experience Replay, leveraging diffusion models to create synthetic experiences for more efficient learning. [84] demonstrates that diffusion models are effective planners and data synthesizers for multi-task RL, showcasing their versatility and efficiency. In this work, we investigate the power of diffusion models for safe RL, where the balance between reward and cost presents further complexities.

3 Problem Formulation

3.1 Safe RL with Constrained Markov Decision Process

We formulate Safe RL problems under the Constrained Markov Decision Process (CMDP) framework [85]. A CMDP M is defined by the tuple ( S , A , P , r, c, γ, µ 0 ) , where S ∈ R m is the state space, A ∈ R n is the action space, P : S × A × S - → [0 , 1] is the transition function, r : S × A × S - → R is the reward function, c : S × A × S - → R ≥ 0 is the cost function, γ is the discount factor, and µ 0 : S - → [0 , 1] is the initial state distribution. Note that this work can also be applied to multipleconstraint tasks, but we use a single-constraint setting for easy demonstration. A safe RL problem is specified by a CMDP and a constraint threshold κ ∈ [0 , + ∞ ) . Denote π ∈ Π : S × A → [0 , 1] as the policy and τ = { ( s 1 , a 1 , r 1 , c 1 ) , ( s 2 , a 2 , r 2 , c 2 ) , . . . } as the trajectory. The stationary state-action distribution under the policy π is defined as d π ( s, a ) = (1 -γ ) ∑ t γ t Pr( s t = s, a t = a ) . The reward and cost returns are defined as R ( τ ) = ∑ τ r , and C ( τ ) = ∑ τ c . The value function is V π f ( µ 0 ) = E τ ∼ π,s 0 ∼ µ 0 [ ∑ ∞ t =0 γ t f t ] , f ∈ { r, c } , which is the expectation of discounted return under the policy π and the initial state distribution µ 0 . The goal of safe RL is to find the optimal policy π ∗ that maximizes the expectation of reward return while constraining the expectation of cost return to the threshold κ :

3.2 Regularized offline safe RL

For an offline safe RL problem, the agent can only access a pre-collected dataset D = ∪ i D i , where D i ∼ π B i is collected by the behavior policy π B i ∈ Π B . To solve the problem in Eq. (1), we convert the constraint optimization problem into an unconstrained form:

The primal-dual-based algorithm solves the optimal policy π ∗ and the dual variable λ ∗ by updating ( π, λ ) iteratively [86, 23, 66]. In offline safe RL tasks, a regularization term is usually introduced to prevent the action OOD issue [87], that is, the objective is converted to:

where w > 0 is a constant weight, L ( π, π B ) is a regularization term and π B is the empirical behavior policy and can be viewed as a mixture of { π B i } . Practically, regularization is formulated as the MSE regularization [88] or the evidence lower bound regularization [35, 41]. In offline safe RL, there are two main challenges: (1) Distribution shift [25]. The agent has poor generalizability when facing OOD state-action pairs during online evaluation; and (2) Efficiency-safety performance balancing [28]. The agent tends to be over-conservative or aggressive when overestimating or underestimating the safety requirements.

4 Method

In this section, we first identify the safe dataset mismatch (SDM) problem, which leads to performance degradation when solving the regularized offline safe RL objective in Eq. (3). Then we present the proposed OASIS (c O ndition A l di S tribut I on S haping) method to solve this problem. In contrast to the model-centric safe RL approaches that focus on optimizing policy update process and network architecture, OASIS is a data-centric learning method that aims to improve the quality of the dataset, thus benefiting offline safe RL training. The OASIS method utilizes the diffusion model to realize conditional distribution shaping, solving the challenges mentioned above, thus benefiting offline safe RL training. Following the proposed algorithm, we provide a theoretical guarantee of the safety performance of the policy learned in this paradigm.

4.1 Safe Dataset Mismatch Problem

The regularized offline safe RL objective in Eq. (3) pushes policy to behavior policy to prevent action OOD issues [36]. When given an imperfect dataset, the stateaction distribution deviates from the optimal distribution, and the SDM problem arises: if the behavior policy is too conservative with low costs and low rewards, it leads to task efficiency degradation; if the behavior policy is too aggressive with high costs and high rewards, it leads to safety violations. To further investigate the SDM problem and the effect of dataset distribution on offline safe RL, we define the tempting dataset and conservative dataset , which are based on tempting and conservative policies:

Definition 1 (Tempting policy [51] and conservative policy) . The tempting policy class is defined as the set of policies that have a higher reward return expectation than the optimal policy, and the conservative policy class is defined as the set of policies that have lower reward and cost return expectations than the optimal policy:

𝒟

!

𝒟

"

𝔼

#

∗

(𝐶, 𝑅)

𝔼

#

"

(𝐶, 𝑅)

𝔼

#

#

(𝐶, 𝑅)

∗

(𝐶, 𝑅)

Cost limit

500

400

rewardreturn

300

200

★

100

0

号

0

20

40

60

80

100

Cost return

Intuitively, a tempting policy is a more rewarding but less safe policy than the optimal one, and a conservative policy is with lower cost but less rewarding. According to these policies, we define two types of datasets:

Definition 2 (Tempting and conservative dataset) . For an offline dataset D i ∼ π i , if π i ∈ Π B ∩ Π T , then the dataset is tempting; if π i ∈ Π B ∩ Π C , then the dataset is conservative.

Staying within the tempting dataset distribution results in tempting (unsafe) behavior, while staying within the conservative dataset distribution causes reward degradation. A theoretical analysis of performance degradation due to the SDM problem is presented in Sec. 4.4. Fig. 2 illustrates examples of both conservative and tempting datasets.

It is important to note that tempting and conservative datasets are prevalent in offline safe RL since optimal policies are rarely available for data collection. The SDM problem is a distinct feature of offline safe RL, indicating that training the policy on either tempting or conservative datasets will violate safety constraints or result in sub-optimality, both of which are undesirable. Therefore, addressing the SDM problem is essential for the development of regularized offline safe RL algorithms.

4.2 Mitigating the Safe Dataset Mismatch

Inspired by recent research works on data-centric learning [70], which emphasizes the quality of data used for training the agent, we focus on enhancing the data pipeline to ensure the agent learns effectively instead of modifying the RL algorithm or model architecture. We propose to use the distribution shaping of the dataset to mitigate the SDM problem, that is, generating a new dataset D g by reshaping the original data distribution. As shown in Fig. 1, the key

(a) Full dataset

(c) Reweighting experiment results

(b) Tempting dataset

500

0.8

500

0.8

Rewardreturn

400

Rewardreturn

400

0.6

300

300

200-

0.4

200

0.4

100

0.2

100

0.2

cost limit

cost limit

20

Costreturn

40

60

80

100

20

Costreturn

40

60

80

100

Full-BCQL

Tempting-BCQL

cost limit

Full-BCQL(reweight)

一

Tempting-BCQL(reweight)

Reward

Cost

475

80

450

70

425

60

400

50

375

40

350

30

325

00E

10

275 L

20

30

40

50

60

0

20

30

40

50

60

constraint thresholdK

constraint threshold K

201

idea is to adjust the dataset distribution towards the optimal distribution under π ∗ , reducing the distribution discrepancy and mitigating the SDM problem, thus both mitigating the action OOD issue and balancing efficiency and safety in offline safe RL.

Dataset

Reward / cost

models

Inverse

dynamics

New Dataset

Offline Safe

RL Agent

Condition-guided denoising

Initial state

(𝑠, 𝑎,	𝑠

!

, 𝑟, 𝑐)

Reward

𝑟

Cost limit

𝜅

Preference

(𝑠, 𝑎,	𝑠

!

, 𝑟, 𝑐)

00

O

O

90

00

Among the data-centric algorithms, Dataset reweighting (weighted sampling) which assigns different sampling weights to data points, is a straightforward way to shape the data distribution [39, 89]. In the offline RL domain, researchers proposed methods to assign high weights to data that achieve high rewards and superior performance in many RL tasks [39, 89].

To validate this idea, we deploy a Boltzmann energy function considering both the reward and the cost for the reweighing strategy to solve the problem (see Appendix B for details). The experimental results, shown in Fig. 3, validate the effectiveness of this distribution shaping method when the coverage of the dataset is complete.

However, for a more general case where we can only access the low-quality dataset (e.g., tempting datasets in Fig. 3), simply performing data reweighting does not work well due to the absence of necessary data. Thus, we propose to use a conditional generative model for more flexible distribution shaping, which generates new data by stitching sub-optimal trajectories for offline training.

4.3 Constraint-Conditioned Diffusion Model as Data Generator

To overcome the limitation of reweighing methods, we propose using diffusion models to generate the dataset that fits the target cost limit to achieve distribution shaping. In the following, we introduce the details of the generator training and dataset generation phases.

Training. In previous works [82, 79], the trajectory planning in offline RL can be viewed as the sequential data generation: τ = [ s 0 , s 1 , . . . , s L -1 ] , where τ is a subsequence of trajectory with length L . Denote x k ( τ ) and y ( τ ) as the k -step denoising output of the diffusion model and the denoising conditions such as reward and cost returns, respectively. Then the forward diffusion process is to add noise to x k ( τ ) and gradually convert it into Gaussian noise:

where β k is a pre-defined beta schedule, K is the total denoising timestep. Then the trainable denoising step aims at gradually converting the Gaussian noise back to a valid trajectory:

where θ is the trainable parameter. We use a simplified surrogate loss [90] for optimization:

In this work, we use the classifier-free guidance [91] for conditional data generation. The condition y ( τ ) in Eq. (6) and Eq. (7) is set to y ( τ ) = [ C ( τ ) , R ( τ )] . Thus, the denoising process depends on the target reward and cost returns of the planned subtrajectory. During training time, the diffusion model learns both an unconditional denoising core ϵ θ ( x k ( τ ) , ∅ , k ) and a conditional denoising core ϵ θ ( x k ( τ ) , y ( τ ) , k ) . We adopt masking [80] for the training to zero out the condition of one training trajectory and categorize it as the ∅ class with probability 0 < p < 1 . Within the given raw dataset, we also train an inverse dynamics model ˆ f : S × S - →A , and reward and cost models ˆ r ( s, a, s ′ ) , ˆ c ( s, a, s ′ ) : S × A × S - → R for labeling.

Generation. After obtaining a trained model, the next step is to generate a new dataset following the conditions. For diffusion model inference, the denoising core ϵ θ ( x k ( τ ) , y , k ) is calculated by:

where w α > 0 is a constant guidance scale and y c := [ ˆ C, ˆ R ] is the generation condition, we set ˆ C ≤ κ to align safety preference. For guided generation, we fix the initial state, which means that we replace the initial state of each k -step noised trajectory as x k [0] = x 0 [0] 1 .

After generating one subtrajectory τ g = x 0 , we can get the state and action sequence s g = τ g [: -1] , s ′ g = τ g [1 :] , a g = ˆ f ( s g , s ′ g ) , then label the data r g = ˆ r ( s g , a g , s ′ g ) , c g = ˆ c ( s g , a g , s ′ g ) . Finally, we get a generated dataset D g = { s g , a g , s ′ g , r g , c g } with | τ g | -1 transition pairs. With this new dataset, we can further train offline safe RL agents.

In this work, we consider BCQ-Lag [35, 86] as the base offline safe RL algorithm. The process of generating one subtrajectory τ g is summarized in Algorithm 1. More details of the implementation are available in Appendix C.

4.4 Theoretical analysis

We first investigate how the distribution mismatch degrades the policy performance on constraint satisfaction. Suppose that the maximum one-step cost is C max = max s,a c ( s, a ) . Based on Lemma 6 in [92] and Lemma 2 in [93], the performance gap between the policy π learned with the dataset D and the optimal policy is bounded by

where d D ( s ) , d ∗ ( s ) denote the stationary state distribution of the dataset and optimal policy. The proof is given in Appendix A.1. Therefore, a significant mismatch between the dataset and the optimal policy results in both a substantial state distribution TV distance and a large policy shift from the optimal one, which can cause notable performance degradation, especially when the offline RL algorithm enforces the learned policy to closely resemble the behavior policy of the offline data.

Then we provide a theoretical analysis of how our method mitigates this mismatch issue by shrinking the distribution gap, which provides a guarantee of the safety performance of the regularized offline safe RL policy. Let d g ( s | y ) denote the state marginal of the generated data with condition y . We first make the following assumptions.

Assumption 1 (Score estimation error of the state marginal) . There exists a condition y ∗ such that the score function error of the state marginal is bounded by

where d ∗ ( s ) is the stationary state distribution induced by the optimal policy π ∗ .

This assumption is also adopted in previous work [94, 95]. For simplicity, we omit the condition y ∗ in the following analysis and use d g ( s ) , d g ( s, a ) to denote the generated state or the state-action distribution with condition y ∗ . As we use inverse dynamics f ( a | s, s ′ ) to calculate actions based on the generated state sequence, the quality of the dataset is also determined by the inverse dynamics. Therefore, we further make the following assumption.

Assumption 2 (Error of inverse policy) . The error of action distribution generated by the inverse dynamics is bounded by

Algorithm 1: OASIS (dataset generation)

Input: Raw dataset D , safety threshold κ Output: Generated sub-trajectory τ g

1: Sample a initial state: s ∼ D

2: Get initial noisy sub-trajectory:

x k = [ s, s 1 , ..., s L -1 ] , s i ∼ N ( 0 , I )

3: Determine the condition y c ;

4: for k = K,..., 1 do

5: Calculate ϵ ( x k , y c , k ) (8);

6: Inverse sampling sequence x k -1 (6)

7: end for

8: Get actions, rewards, and costs from x 0 ;

9: Return: trajectory τ g

where ˆ π inv ( a | s ) = E s ′ [ ˆ f ( a | s, s ′ )] denotes the empirical inverse policy, which is a marginal of inverse dynamics over s ′ .

Then the distance of generated data distribution to the optimal one is bounded as:

Theorem 1 (Distribution shaping error bound) . Suppose that the optimal stationary state distribution satisfies that 1) its score function ∇ s log d ∗ ( s ) is L -Lipschitz and 2) its second momentum is bounded. Under Assumption 1 and 2, the gap of generated state-action distribution to the optimal stationary state-action distribution is bounded by

where C ( d ∗ ( s ) , L, K ) represents a constant determined by d ∗ ( s ) , L and K .

The proof is given in Appendix A.2. Theorem 1 indicates that using the proposed OASIS method, we can shape the dataset distribution towards a bounded neighborhood of the optimal distribution.

Given the generated data, we will then train a regularized offline safe RL policy by Eq. (3). Notice that the regularization term in the objective function in Eq. (3) is equivalent to an explicit policy constraint, and the coefficient w is the corresponding dual variable. Therefore, we make the following assumption on the distance between the learned policy π ϕ and the behavior policy.

Assumption 3. Denote the generated dataset as D g and the corresponding behavior policy as π g , given a fixed coefficient w , for the policy π ϕ optimized by Eq. (3), there exists a ε reg such that

Based on the above assumptions, we can derive the bound of constraint violation of the policy learned on the offline data generated by OASIS. The proof is given in Appendix A.3.

Theorem 2 (Constraint violation bound) . For policy π ϕ optimized by regularized-based offline safe RL on generated dataset D g , under Assumption 1 , 2 and 3, the constraint violation of the trained policy is bounded as:

where C ( d ∗ ( s ) , L, K ) represents a constant determined by d ∗ ( s ) , L and K .

The theoretical analysis sheds insights by answering two questions: (1) Why do we use the diffusion model for conditional distribution shaping, and (2) How does conditional distribution shaping benefit offline safe RL training? Theorem 1 shows that by using the conditional diffusion model as a data generator, the TV distance between optimal and generated state-action distribution is bounded; Theorem 2 shows that the safety performance of agent trained on the generated dataset is guaranteed with OASIS.

5 Experiment

In the experiments, we answer these questions: (1) How does the distribution of the dataset influence the performance of regularized offline safe RL? (2) How does our proposed distribution shaping method perform in offline safe RL tasks? (3) How well does the conditional data generator shape the dataset distribution? To address these questions, we set up the following experiment tasks.

Environments. We adopt the continuous robot locomotion control tasks in the public benchmark Bullet-Safety-Gym [96] for evaluation, which is commonly used in previous works [68, 28, 60]. We consider two tasks, Run and Circle , and three types of robots, Ball , Car , and Drone . We name the environment as Agent -Task . A detailed description is available in the Appendix B.

Datasets. Our experiment tasks are mainly built upon the offline safe RL dataset OSRL [74]. To better evaluate the tested algorithms with the challenging SDM problem, we create four different training dataset types, full , tempting , conservative , and hybrid . The tempting dataset contains sparse safe demonstrations, the conservative dataset lacks rewarding data points, and the hybrid dataset has scarcity in the medium-reward, medium-cost trajectories. We set different cost thresholds for different datasets. A detailed description and visualization of the datasets are available in Appendix B. For the main experiments presented in Table 1, we train on tempting dataset, with threshold κ = 20 .

0

200

400

600

BC

CPQ

COptiDICE

BEAR-Lag

BCQ-Lag

OASIS

Reward-Full

0

200

400

600

Reward-Conservative

0

200

400

600

Reward-Hybrid

0

20

40

BC

CPQ

COptiDICE

BEAR-Lag

BCQ-Lag

OASIS

Cost-Full

0

20

40

60

Cost-Conservative

0

20

40

60

Cost-Hybrid

(a) Ball-Circle

0

200

400

BC

CPQ

COptiDICE

BEAR-Lag

BCQ-Lag

OASIS

Reward-Full

0

200

400

Reward-Conservative

0

200

400

Reward-Hybrid

0

25

50

75

BC

CPQ

COptiDICE

BEAR-Lag

BCQ-Lag

OASIS

Cost-Full

0

25

50

75

Cost-Conservative

0

25

50

75

Cost-Hybrid

(b) Car-Circle

Baselines and OASIS. For baselines, we compared our method with both model-centric and datacentric baseline methods. Model-cenrtic : (1) Q-learning-based algorithms: BCQ-Lag [35, 86], BEAR-Lag [41, 86], and CPQ [26]; (2) Imitation learning: Behavior Cloning ( BC ) [26]; (3) Distribution correction estimation: COptiDICE [23], and (4) Sequential modeling algorithms: CDT [28] and FISOR [31]; Data-cenrtic : (5) Data augmentation: CVAE-BCQL : we train BCQ-Lag agents on the datasets generated by Conditional Variational Autoencoder (CVAE) [97]. The CVAE training set and the dataset generation conditions are set as the same with our OASIS. For OASIS implementation, we set three cost conditions ˆ C ≤ κ to align safety preference and guarantee data coverage. Code is available on our Github repository, checkpoints and curated datasets are available on our HuggingFace repository.

Metrics. We use the normalized cost return and the normalized reward return as the evaluation metric for comparison in Table 1 and 2. The normalized cost is defined as C normalized = C π /κ , where C π is the cost return and κ is the cost threshold. The agent is safe if C normalized ≤ 1 . The normalized reward is computed by R normalized = R π /r max ( M ) , where r max ( M ) is the maximum empirical reward return for task M within the given dataset. We report the averaged results and standard deviations over 3 seeds for all the quantity evaluations.

5.1 How can conditional data generation benefit offline safe RL?

Performance degradation with SDM problems. The comparison results on the tempting dataset are presented in Table 1 with the cost threshold κ = 20 before normalization. Results of BC show the mismatch between the behavior policy and the safe policy, as the cost returns significantly violate the safety constraints. The results of BCQ-Lag and BEAR-Lag show this mismatch further influences the regularized-based algorithms, leading to constraint violations. This is because the regularization term pushes the policy towards the unsafe behavior policy. The conservative Q function estimation method CPQ , exhibits a significant reward degradation in all tasks, which arises from the drawback of the pessimistic estimation methods that learn over-conservative behaviors. COptiDICE also fails to learn safe and rewarding policies, showing that even using distribution correction estimation is not enough to solve the SDM problem. For sequential modeling algorithms, CDT shows poor safety performance and FISOR tends to be over-conservative with poor reward performance. This is because both methods require a large amount of high-quality data while the trajectories with low cost and high reward are sparse in this task. The unsatisfactory performance of these model-centric algorithms further motivates the effective data-centric learning algorithm for offline safe RL.

Performance improvement using OASIS. From Table 1, we find that only our method OASIS can learn safe and rewarding policies by mitigating the SDM problem. In addition to the results on the tempting dataset, we also provide evaluation results within different types of datasets and constraint thresholds in Fig. 5a and Fig. 5b. We can observe that most baselines still fail to learn a safe policy within different task conditions due to the SDM issue. In contrast, our proposed OASIS method achieves the highest reward among all safe agents, which shows strength in more general cases.

1

6

63

(%)

400

500

600

700

Reward

1

6

63

(%)

0

10

20

30

40

50

60

Cost

OASIS

BCQ-Lag

COptiDICE

CDT

cost limit

High data efficiency of OASIS. In this experiment, we vary the amount of data for offline RL agent training to evaluate data efficiency. The evaluation results are shown in Fig. 6. The x-axis α represents the size of the RL agent training dataset. For OASIS, it denotes the size of the generated data. A subsequent BCQ-Lag agent is trained on this generated dataset to obtain the safe RL policy. For the baseline methods, we randomly sample trajectories from the original dataset to construct the training dataset.

The comparison results indicate that we can still learn a good policy using a small amount of high-quality data ( α < 2% ) generated by OASIS. In contrast, baseline methods show significant performance degradation when the data are sparse as the noisy data is of low quality.

This observation demonstrates that the agent can learn a good policy with high data efficiency given high-quality data with minimal safe dataset mismatch (SDM) issues. As OASIS offers a solution to shape the dataset distribution, it also reduces the required training dataset size while maintaining good performance, further

reaffirming the effectiveness of data-centric approaches in offline safe learning and highlighting that prioritizing quality is essential [70].

5.2 How can OASIS shape the dataset distribution?

Successful distribution shaping. To show the distribution shaping capability of the proposed OASIS and baseline CVAE , we generate the dataset under different conditions and visualize them in Fig. 7. When using different conditions, the expectations of reward and cost of the generated dataset change accordingly. This shows the strong capability of our method in distribution shaping. We also visualize the density of the generated data. In the Car-circle task, the robot receives high rewards when moving along the circle boundary and receives costs when it exceeds the boundaries on both sides, as shown in Fig. 7(c). The original dataset contains trajectories with various safety performances.

(c) Data density visualization

(a) Reward distribution

(b) Cost distribution

reward condition

cost condition

cost condition

reward condition

OASIS

CVAE

Task & dataset

1

2

3

4

5

6

0.1

0.25

0.5

0.75

0.9

0.1

0.25

0.5

0.75

0.9

0.9-0.09

0.16

0.41

0.75

0.76

0.7

0.9-

1.05

1.09

1.20

0.89

0.81

1.2

1.0

0.18

0.46

0.73

0.75

0.6

0.75 -

0.86

0.88

0.99

0.77

0.73

0.5

0.8

0.5-0.08

0.19

0.51

0.69

0.74

0.4

0.5 -

0.31

0.35

0.34

0.50

0.59

0.6

0.20

0.57

0.67

0.74

0.3

0.25-0.07

0.12

0.12

0.40

0.50

0.4

0.2

0.2

0.1-0.09

0.22

0.56

0.63

0.74

0.1

0.1-0.05

0.09

0.09

0.17

0.47

When using a low-cost condition, the generated data are clustered within the safety boundary to satisfy the constraints. When using a high-reward condition, the generated data points are closer to the circle boundary and receive higher rewards. In contrast, the baseline CVAE cannot successfully incorporate the conditions in data generation, resulting in almost similar datasets with different conditions as shown in Fig. 7(c). More experiment results including the visualization of generated dataset comparison and corresponding analysis are available in Appendix B.2.

5.3 Robust performance against denoising steps

We conduct an ablation study on the key hyperparameter of the proposed OASIS method. The experiment related to the denoising steps K is presented in Table 2. Performance does not change much with different values, which shows the robustness of the proposed OASIS method.

6 Conclusion

In this paper, we study the challenging problem in offline safe RL: the safe data mismatch between the imperfect demonstration and the target performance requirements. To address this issue, we proposed the OASIS method to employ a conditional diffusion model to shape the dataset distribution and benefit offline safe RL training. In addition to the theoretical guarantee of performance improvement, we also conduct extensive experiments to show the superior performance of OASIS in learning a safe and rewarding policy on many challenging offline safe RL tasks. More importantly, our method shows good data efficiency and robustness to hyperparameters, which makes it preferable for applications in many real-world tasks.

There are two limitations of OASIS: (1) Offline training takes longer: our method involves preprocessing the offline dataset to enhance quality, which requires more time and computing resources; (2) Achieving zero-constraint violations remains challenging with imperfect demonstrations. One potential negative social impact is that misuse of our method may cause harmful consequences and safety issues. Nevertheless, we believe that our proposed method can inspire further data-centric research in the safe learning community and help to adapt offline RL algorithms to real-world tasks with safety requirements.

Acknowledgment

The work is partially supported by Google Deepmind with an unrestricted grant. The authors also want to acknowledge the support from the National Science Foundation under grants CNS-2047454.

References

Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Kyle Stachowicz and Sergey Levine. Racer: Epistemic risk-sensitive rl enables fast driving with fewer crashes. arXiv preprint arXiv:2405.04714 , 2024.

Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Safety-aware causal representation for trustworthy offline reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters , 2024.

Zhili Zhang, Songyang Han, Jiangwei Wang, and Fei Miao. Spatial-temporal-aware safe multi-agent reinforcement learning of connected autonomous vehicles in challenging scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5574-5580. IEEE, 2023.

Xing Fang, Qichao Zhang, Yinfeng Gao, and Dongbin Zhao. Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) , pages 3417-3422. IEEE, 2022.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

Wenhao Ding, Laixi Shi, Yuejie Chi, and Ding Zhao. Seeing is not believing: Robust reinforcement learning against spurious correlation. Advances in Neural Information Processing Systems , 36, 2024.

Weiye Zhao, Rui Chen, Yifan Sun, Ruixuan Liu, Tianhao Wei, and Changliu Liu. Guard: A safe reinforcement learning benchmark. arXiv preprint arXiv:2305.13681 , 2023.

Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, and Wei Zhan. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. arXiv preprint arXiv:2309.09408 , 2023.

Wenhao Ding, Tong Che, Ding Zhao, and Marco Pavone. Bayesian reparameterization of reward-conditioned reinforcement learning with energy-based models. In International Conference on Machine Learning , pages 8053-8066. PMLR, 2023.

Dohyeong Kim, Yunho Kim, Kyungjae Lee, and Songhwai Oh. Safety guided policy optimization. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2462-2467. IEEE, 2022.

Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, and Guanya Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint arXiv:2401.17583 , 2024.

Wenli Xiao, Tairan He, John Dolan, and Guanya Shi. Safe deep policy adaptation. arXiv preprint arXiv:2310.08602 , 2023.

Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations , 2023.

Zhaorun Chen, Binhao Chen, Tairan He, Liang Gong, and Chengliang Liu. Progressive adaptive chance-constrained safeguards for reinforcement learning. arXiv preprint arXiv:2310.03379 , 2023.

Wenjun Zou, Yao Lv, Jie Li, Yujie Yang, Shengbo Eben Li, Jingliang Duan, Xianyuan Zhan, Jingjing Liu, Yaqin Zhang, and Keqiang Li. Policy bifurcation in safe reinforcement learning. arXiv preprint arXiv:2403.12847 , 2024.

Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters , 8(3):1295-1302, 2023.

Qian Lin, Zongkai Liu, Danying Mo, and Chao Yu. An offline adaptation framework for constrained multi-objective reinforcement learning. arXiv preprint arXiv:2409.09958 , 2024.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5, 2021.

Akifumi Wachi, Xun Shen, and Yanan Sui. A survey of constraint formulations in safe reinforcement learning. arXiv preprint arXiv:2402.02025 , 2024.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957 , 2022.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8753-8760, 2022.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351 , 2023.

Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301.12203 , 2023.

Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. arXiv preprint arXiv:2306.00603 , 2023.

Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700 , 2024.

Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems , 34:11260-11273, 2021.

Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 4680-4688, 2022.

Yao Liu, Pratik Chaudhari, and Rasool Fakoor. Budgeting counterfactual for offline rl. Advances in Neural Information Processing Systems , 36, 2024.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning , pages 19967-20025. PMLR, 2022.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems , 33:1264-1274, 2020.

Yihang Yao, Zuxin Liu, Zhepeng Cen, Peide Huang, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Gradient shaping for multi-constraint safe reinforcement learning. arXiv preprint arXiv:2312.15127 , 2023.

Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems , 36:4985-5009, 2023.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing offpolicy q-learning via bootstrapping error reduction. Advances in neural information processing systems , 32, 2019.

Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning , pages 907917. PMLR, 2022.

Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning , pages 25611-25635. PMLR, 2022.

Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline reinforcement learning. Advances in neural information processing systems , 36, 2024.

Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems , 33:3976-3990, 2020.

Ahmed Aloui, Juncheng Dong, Cat P Le, and Vahid Tarokh. Counterfactual data augmentation with contrastive learning. arXiv preprint arXiv:2311.03630 , 2023.

Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato, Kun Zhang, and Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-based data augmentation. arXiv preprint arXiv:2012.09092 , 2020.

Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems , 35:18143-18156, 2022.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pages 22-31. PMLR, 2017.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205.14691 , 2022.

Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based safe distributional reinforcement learning for multiple constraints. Advances in neural information processing systems , 36, 2024.

Sheng Xu and Guiliang Liu. Uncertainty-aware constraint inference in inverse constrained reinforcement learning. In The Twelfth International Conference on Learning Representations , 2023.

Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. arXiv preprint arXiv:2201.12427 , 2022.

Dohyeong Kim, Mineui Hong, Jeongho Park, and Songhwai Oh. Scale-invariant gradient aggregation for constrained multi-objective reinforcement learning. arXiv preprint arXiv:2403.00282 , 2024.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research , 18(167):1-51, 2018.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7, 2019.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems , 2020.

Zhepeng Cen, Yihang Yao, Zuxin Liu, and Ding Zhao. Feasibility consistent representation learning for safe reinforcement learning. arXiv preprint arXiv:2405.11718 , 2024.

Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Off-policy primal-dual safe reinforcement learning. arXiv preprint arXiv:2401.14758 , 2024.

Dongsheng Ding, Zhengyan Huan, and Alejandro Ribeiro. Resilient constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3412-3420. PMLR, 2024.

Jiayi Guan, Li Shen, Ao Zhou, Lusong Li, Han Hu, Xiaodong He, Guang Chen, and Changjun Jiang. Poce: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26243-26253, 2024.

Nicholas Polosky, Bruno C Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning , pages 1780117810. PMLR, 2022.

Kihyuk Hong, Yuhang Li, and Ambuj Tewari. A primal-dual-critic algorithm for offline constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 280-288. PMLR, 2024.

Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, et al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Zijian Guo, Weichao Zhou, and Wenchao Li. Temporal logic specification-conditioned decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2402.17217 , 2024.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345 , 2021.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems , 36, 2024.

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518 , 2024.

Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Information Processing Systems , 36, 2024.

Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333 , 2024.

Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023.

Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223 , 2023.

Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. arXiv preprint arXiv:2405.16907 , 2024.

Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Dit-Yan Yeung, and Chuang Gan. Adaptive online replanning with diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Rewarddirected conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems , 36, 2024.

Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991 , 2022.

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

Wei Xiao, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. arXiv preprint arXiv:2306.00148 , 2023.

Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877 , 2023.

Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems , 36, 2024.

Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems , 36, 2024.

Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research , 48(3):387417, 1998.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 91339143. PMLR, 2020.

Qian Lin, Chao Yu, Zongkai Liu, and Zifan Wu. Policy-regularized offline multi-objective reinforcement learning. arXiv preprint arXiv:2401.02244 , 2024.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Zhang-Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.13085 , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems , 33:15737-15749, 2020.

Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, and Ding Zhao. Learning from sparse offline datasets via conservative density estimation. arXiv preprint arXiv:2401.08819 , 2024.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems , 35:2287022882, 2022.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 , 2022.

Sven Gronauer. Bullet-safety-gym: Aframework for constrained reinforcement learning. 2022.

Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semisupervised learning with deep generative models. Advances in neural information processing systems , 27, 2014.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

Appendix

Table of Contents

A Proofs

A.1 Proof of Eq.(9)

By definition of the stationary state-action distribution,

The second inequality holds by triangle inequality for total variation distance.

In general, the stationary distribution of learned policy is in between of the empirical distribution of offline data d D and optimal d ∗ . Therefore, we can obtain

A.2 Proof of Theorem 1

Proof. By triangle inequality, we first decompose the TV distance between state-action distributions into a state distribution distance and a policy distance,

We then consider two parts separately.

For the stationary state distribution distance, we suppose that the optimal distribution d ∗ ( s ) has a L -Lipschitz smooth score function and a bounded second momentum. Meanwhile, note that the score function in Assumption 1 is closely related to the denoising model ϵ θ [98, 99]:

where ϵ θ ( s | y ) is the state marginal of the practical denoising model in Eq. (8). Therefore, by theorem 2 in [95], under Assumption 1, we have

where K is the number of denoising timestep, |S| is the dimension of the state space, and m 2 is the second momentum of d ∗ ( s ) . Therefore, aggregating the first two terms in RHS, we have

where C ( . . . ) is a constant w.r.t d ∗ , L, K .

Regarding the policy distance. By Pinsker's inequality,

Meanwhile, since the action is generated by the inverse policy, i.e., π g = ˆ π inv, by Assumption 2, we have

where the second inequality holds by Jensen's inequality.

Combining the Eq. (28)) and (32), we finish the proof of Theorem 1.

A.3 Proof of Theorem 2

We start from the Eq. (9). The policy distance can be further decomposed into

By Assumption 2 and 3 and Jensen's inequality, we have

Plug-in Eq. (28) and (38) into Eq. (21), we have

Meanwhile, notice that the optimal policy is constraint satisfactory, i.e.,

Therefore, we have

which finishes the proof of Theorem 2.

B Supplementary experiments

B.1 Trajectory reweighting for distribution shaping

In this section, we provide details about the trajectory reweighting experiment presented in Sec. 4.1. Following previous work [39, 89] in offline RL, we adopted datapoint reweighting in policy optimization, which can be formulated via importance sampling as:

where J w off ( π, λ ) is the objective function after reweighting. In this experiment, we utilize a Boltzmann energy function as adopted in [39] for offline RL tasks:

Here w ( τ ) means that all the state-action pairs in one trajectory share the same weight, which is related to the cost and reward returns C ( τ ) , R ( τ ) . We adopt α 1 = α 2 = 1 in the experiments shown in Fig. 3.

B.2 Supplementary data generation comparison results

Due to the page limit, we omit the visualization of reward and cost distribution using the CVAE method for data generation. Here we provide the results in Fig. 8. From the reward performance and the cost performance, we can observe that CVAE can hardly encode conditions into the data reconstruction, leading to similar results when setting different conditions. From the trajectory reconstruction results shown in Fig. 8(c), we can observe that the generated trajectories are almost the same as the original one. This feature is not desirable for our distribution shaping purpose. In contrast, our method OASIS can successfully shape the distribution as shown in Fig. 7, with the strong capability of the diffusion model in the condition-guided denoising process.

(c) Trajectory visualization

(a) Reward distribution

(b) Cost distribution

0.1

0.25

0.5

0.75

0.9

1.0

Original Data

0.9

0.42

0.44

0.42

0.40

0.39

Reconstruction

0.8

75

0.41

0.43

0.41

0.39

0.39

0.6

0.5

0.41

0.41

0.39

0.38

0.37

0.4

25

0.40

0.40

0.38

0.37

0.37

0.2

0.1-

0.39

0.38

0.38

0.37

0.36

0.0

0.1

0.25

0.5

0.75

0.9

1.0

0.9

0.51

0.54

0.55

0.55

0.55

0.8

75

0.52

0.54

0.55

0.55

0.55

0.6

0.5-

0.52

0.54

0.55

0.55

0.55

0.4

25 -

0.53

0.54

0.55

0.55

0.55

- 0.2

0.1 -

0.53

0.54

0.55

0.55

0.55

0.0

The comparison results of generated trajectories using OASIS and CVAE in the Drone-Circle task are presented in Fig. 9. This figure illustrates the generation results of OASIS and CVAE under two conditions: low-cost-medium-reward, and medium-cost-high-reward. Although CVAE can reconstruct the trajectories, it fails to integrate conditions into the generation process. In contrast, OASIS successfully controls the generated trajectories, avoiding the restricted area when conditioned on low-cost-medium-reward.

C Implementation details

C.1 Environment details

Due to the page limit, we omit some descriptions of experiments in the main context. Here we give more details about our experiment tasks. Both the Circle task and the Run task are from a publicly available benchmark [96].

OASIS

OASIS

CVAE

CVAE

Original datapoint

OASIS generated

CVAE generated

Low-cost, medium reward: (

𝑐 = 0.2, 𝑟 = 0.75

)

Medium-cost, high reward: (

𝑐 = 0.5, 𝑟 = 0.85

)

Circle tasks. The agents are rewarded for running along a circle boundary. The reward function is defined as:

where x , y are the positions of the agent with state s ′ , v x , and v y are velocities of the agent with state s ′ . radius is the radius of the circle area, and r robot ( s t ) is the specific reward for different robot.

The agent gets cost when exceeding the boundaries. The cost function is defined as:

where x lim is the boundary position.

Run tasks. Agents are rewarded for running fast along one fixed direction and are given costs if they run across the boundaries or exceed a velocity limit. The reward function is defined as:

The cost function is defined as:

where v lim is the speed limit, and y lim is the y position of the boundary, v t = [ v x , v y ] is the velocity of the agent with state s ′ , g = [ g x , g y ] is the position of a virtual target, x t = [ x t , y t ] is the position of the agent at timestamp t , x t -1 is the Cartesian coordinates of the agent with state s , x t is the Cartesian coordinates of the agent with state s ′ , and r robot ( s t ) is the specific reward for the robot.

Agents. We use three different robot agents in our experiments: Ball , Car , and Drone . The action space dimension, observation space dimension, and the timesteps for these six tasks are shown in Table 3.

C.2 Dataset details

We provide details about the dataset types we presented in the experiment part. The Full , Tempting , Conservative , and Hybrid datasets for Ball-Circle and Car-Circle tasks are shown in Fig. 10, 11, respectively. All the Tempting datasets associated with results in Table 1 are shown in Fig. 12.

Full dataset

Tempting

Conservative

Hybrid

800

800

800

800

rewardreturn

009

return

600

reward return

009

rewardreturn

600

400

reward

400

400

400

200

200

200

200

costlimit

costlimit

costlimit

costlimit

0

0

0

0

0

10

20

Costreturn

30

40

50

60

70

08

0

10

20

Cost return

30

40

50

60

70

80

0

10

20

Costreturn

30

40

50

09

70

80

0

10

20

Costreturn

30

40

50

60

70

80

Full dataset

Tempting

Conservative

Hybrid

500

500

500

500

福

reward return

400

return

400

reward return

400

reward return

400

00E

costlimit

reward

300

costlimit

300

cost limit

300

cost limit

200

200

200

200

100

100

100

100

0

0

D

0

20

Costreturn

40

60

08

100

0

20

Costreturn

40

60

80

100

0

Cost return

40

60

80

100

0

20

Costreturn

40

09

08

100

Ball-Run

Car-Run

Drone-Run

Ball-Circle

Car-Circle

Drone-Circle

700

1200

550

600

reward return

1000

rewardreturn

500

AS01

8

rewardreturn

500

800

400-

400

600-

300

350

400

300

200

200

cost limit

250

cost limit

100

cost limit

0

0

10

20

30405060

Costreturn

0

1015

Costreturn

20

25

35

40

0

0

20

4060

Cost return

80100

120

140

1000

800

009

900

rewardreturn

009

reward return

400

return

800

700

300

400

reward

600

200

500

净业

200

?

costlimit

100

。。。

costlimit

400

300

cost limit

0

0

200

10

20

Cost return

30405060

0804

0

20

Costreturn

40

60

80

100

0

20

40

60

80

100

Costreturn

C.3 Algorithm details

OASIS algorithm training diagram In this work, we use the cosine β schedule [100] to calculate β t , t = 1 , ..., K . Then we let α t = 1 -β t , and ¯ α t = ∏ t i =1 α i and denote the state dimension as m . With these notations, we show the training process of the OASIS data generator for one epoch in Algorithm 2.

Algorithm 2: OASIS (training)

1: Input: Original Dataset D , predefined β t and ¯ α t , diffusion core ϵ θ ( x k , y , k ) , learning rate lr , loss function L ( · , · ) .

2: for each sub-trajectory τ i ∈ D do

3: Extract the states from τ i : { ( s 0 , s 1 , . . . , s T ) } # [ T, m ] ;

4: Get the return conditions y = [ C, R ] associated with these sub-trajectories;

5: With probability p = 0 . 25 to mask the condition information as: y ← ∅

6: Get Gaussian Noise noise = N ( 0 , I ) # [ T, m ] ;

7: Randomly sample time t ∈ [0 , ..., K -1] ;

8: Calculate the forward sampling state x noise = ¯ α t ∗ τ i +(1 -¯ α t ) ∗ noise ;

9: Apply initial state condition x noise [0] ← s 0 ;

10: Reconstruct noisy sub trajectory x recon = ϵ θ ( x noise , y , k ) ;

11: Minimize the reconstruction loss θ ← θ -lr ∗ ∇ θ L ( x noise , x recon ) ;

12: end for

13: Output: Updated diffusion core ϵ θ ( x k , y , k ) ;

Network and hyperparameter details. For the dynamics model ˆ p , we utilize a MLP. For the denoising core, we utilize a U-net , which has also been used in previous works [82, 80]. The U-net is visualized in Fig. 13. The hyperparameters for our method are summarized in Table 4. More details are available in the code.

𝑡

𝒚(𝜏)

time

MLP

condition

MLP

𝒚

!"#

𝑡

!"#

𝒙

𝒕"𝟏

(𝜏)

Down layers

Mid layers

Up layers

Conv layers

𝒙

𝒕

(𝜏)

Baseline details. For the baseline methods BC , BCQ-Lag , BEAR-Lag , COptiDICE , CPQ , and CDT , we adopt the code base provided in the benchmark [74]. For the FISOR method, we use the code provided by the authors [31].

Computing resources. The experiments are run on a server with 2 × AMD EPYC 7542 32-Core Processor CPU, 2 × NVIDIA RTX A6000 graphics, and 252 GB memory. For one single experiment, OASIS takes about 4 hours with 200 , 000 steps to train the data generator. It takes about 1 . 5 hours to train a BCQ-Lag agent on this generated dataset for 200 , 000 steps.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: They are provided in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: They are provided in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are provided in Section 4.4, and the proof is provide in Section A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: They are provided in Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the anonymous link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: They are provided in Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: They are provided in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources are provided in Appendix C. More details about experiment results such as reweighting for distribution shaping are also provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Provided in the openreview form.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: They are provided in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not contain pre-trained language models, image generators, or scraped datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The codebase we utilize for baseline comparison is under Apache License 2.0. Related materials are cited in our code (See the README file in the code).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the safe dataset mismatch (SDM) problem, highlighting how low-reward or unsafe samples in datasets can harm offline safe RL. Conditional distribution shaping (OASIS) is proposed to mitigate this problem by generating high-reward and safe samples via diffusion models and promoting general offline safe RL algorithms with the generated data. This paper evaluates OASIS through extensive experiments across various safe RL tasks and different types of datasets with varying data distributions.\n\nSTRENGTHS:\n- The paper is well motivated. The influence of imbalance and biased data on offline safe RL is an important but underexplored problem. Solving the proposed problem through diffusion-based data generation is intuitive, reasonable and novel.\n- The empirical evaluation and ablation studies are comprehensive, carefully demonstrating the significance of the SDM problem and the effectiveness of the proposed approach.\n- Theoretical analysis provides certain guarantees for the proposed approach.\n- The paper is well written and well organized.\n\nWEAKNESSES:\n- Some technical and experimental details are a bit confusing. See Questions 1, 2, 3, 4.\n\n- Assumptions 1 and 2 seem kind of idealized to directly bound the distribution and policy discrepancy, since $\\epsilon_{score}$ and $\\epsilon_{inv}$ cannot be directly calculated or estimated. Admittedly, analyzing the distribution and policy discrepancy based on diffusion-generated data may be difficult and beyond the scope of this work. Maybe more discussion and explanation could help justify these assumptions.\n\n- Directly excluding mismatched data from training datasets could be an another intuitive approach for SDM problem. So it would be better to discuss the performance if tempting and conservative data were directly excluded from the full datasets, utilizing the remaining datasets (i.e., Full dataset - Hybrid dataset) for training. A comparison between these two approaches (i.e., adding matched data vs. reducing mismatched data) would be valuable.\n\nQUESTIONS:\n1. What is the size of the generated dataset $D_g$? I would like to confirm whether you directly train the policy on $D_g$ or a mixture of $D_g$ and the original datasets?\n2. How do you calculate the reward condition $\\hat R$ based on the given cost $\\hat C$?\n3. Why are the thresholds in Figure 5 different for different types of datasets? Did you use the same threshold when constructing different types of datasets? This is a concern because one tempting dataset constructed under the thresholds 20 may be considered tempting for one threshold 20 but not for another threshold 60.\n4. In Figure 6, how did you obtain the performance of baseline methods (e.g., CDT) under a specific $\\alpha$? Did you use the data of size $\\alpha$ generated by OASIS for these baselines? It would be helpful to provide more details about the data efficiency experiments.\n\nLIMITATIONS:\nThe limitation are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOffline Safe Reinforcement Learning (RL) is used to learn policies satisfying cost constraints from a given dataset. This proves to be a challenge when the dataset is biased in a certain way. This paper introduces a method to use the offline training dataset to capture the environment using a diffusion model that can generate data conditioned on our cost constraints and performance objectives. Experimental results show the approach generates data that better fit the constraints and outperform alternatives like dataset reweighting.\n\nSTRENGTHS:\n- The paper is well organized and clearly shows the strengths of diffusion models to generate offline data with the given cost/performance objectives.\n- Theoretical results show that the policies learned from the generated data are cost constrained given some reasonable assumptions on the model.\n- Extensive comparisons are made to different SoTA baselines in offline RL with and without data generation.\n\nWEAKNESSES:\n- Data generation and training can be a slow process due to the use of diffusion models which are known to be computationally heavy.\n- Selecting hyperparameters such as number of denoising steps might vary results significantly. For example, Table 2 has greatly varying policy costs for different values of $K$ (albeit similar performances).\n- While results are mostly consistent, it is hard to say when the proposed method prefers to act more conservatively with lower cost (or more riskily i.e., higher reward). This is reflected in the reward and costs in Table 1 (e.g., CarRun). A study on a hyperparameter change (apart from cost/return targets) to control this balance would be helpful.\n\nQUESTIONS:\n1. How are the initial states decided to generate the data from the diffusion model? Are they the same as the initial states of the training dataset or randomly sampled from the training dataset trajectories?\n2. How long is inference time for the OASIS model i.e., data generation time? Is that included in the training time (L999, App C.3)?\n3. How do we decide the “target” cost given the cost threshold to get the best performance? Is the study done in Fig. 7 required for each setting or is there a good heuristic to set these targets? How expensive is this study (Fig. 7) over target values (i.e., multiple inference runs)? On a related note, how were these targets set for Table 1 on OASIS and the compared baselines?\n4. Is it right to say the primary reason for the success of the proposed method over the diffusion baselines (like FISOR), a more realistic conditional generative model that yields optimal generated data satisfying our cost constraints?\n5. Are the learned labeling models (inverse dynamics, reward, and cost) only used for labeling the data generated from the diffusion model?\n6. In Fig. 6, are all models using data generated by the same diffusion model? OASIS handles the diffusion model training and data generation with BCQ-Lag as the actual Offline RL policy learning. This makes me a little confused. How are the curves related in Fig. 6? \n7. Why is hybrid an interesting dataset setting vs. full?\n8. Typos:\n    - L224 (Sec 4.4) Theoretical\n    -  L973 (App C.1) min instead of max\n\nLIMITATIONS:\nDiffusion models being computationally heavy (see Weakness)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OASIS, which uses a conditional diffusion model to reshape the dataset distribution and achieve effective offline safe RL learning. Theoretical analysis gives the error upper bound of distribution reshaping and constraint violation upper bound. A large number of experiments show that the proposed algorithm has significantly improved compared to offline safe RL baselines.\n\nSTRENGTHS:\n- The logic is clear and the paper is easy to understand.\n- The theoretical analysis is sufficient.\n- A large number of experiments prove the effectiveness of the proposed algorithm.\n\nWEAKNESSES:\n***W1:*** Some baselines are not introduced in related work, such as CVAE, FISOR, etc. This may cause difficulties in understanding.\n\n***W2:*** Typos, such as line 173 \"reweighing\" -> \"reweighting\"; wrong citations, such as line 290 \"COptiDICE [17]\" -> \"COptiDICE [16]\"\n\nQUESTIONS:\n***Q1:*** Compared with CVAE, OASIS shows that the conditional diffusion model has a better ability to generate according to the condition information, as shown in Figure 7(c), but this example is a bit simple. Can the author show more comparisons of the two similar to Figure 7(c)? For example, add the OASIS generation results to Figure 8(c)?\n\n***Q2:*** When showing the effectiveness of the newly generated dataset, in addition to showing the distribution of the generated state like Figure 7(c), it is also necessary to verify whether the annotations of the inverse dynamics model and reward & cost models are accurate. Can the author supplement the accuracy of these three models?\n\n***Q3:*** By comparing the results of OASIS and CDT, I came to a conclusion: in offline safe RL, both methods conditioned on cost and reward, generating data is more effective than generating policy. I wonder if the author agrees with this conclusion? This conclusion seems to be uncommon in the field of offline RL, or it may be because I am not familiar with distribution shaping methods.\n\nLIMITATIONS:\nSee the weaknesses and questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}