Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
Authors: Junyu Guo
Zhi Zheng
Donghao Ying
Ming Jin
Shangding Gu
Costas Spanos
Javad Lavaei
Venue: NeurIPS 2025 poster ()
Keywords: Offline Reinforcement Learning
Safe Reinforcement Learning
Diffusion Models
Primary area: reinforcement_learning
TL;DR: A new approach to learning offline safe reinforcement learning with high performance and safety guarantee

Abstract: Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints. We focus on an offline setting where the agent learns from a fixed dataset—a common requirement in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high-quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in real-world scenarios. We open-source our implementation at https://github.com/JamesJunyuGuo/DRCORL.

Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL
Junyu Guo*
Zhi Zheng
University of California, Berkeley
University of California, Berkeley
Donghao Ying
Ming Jin Virginia Tech
Shangding Gu
University of California, Berkeley
University Californa, Berkeley
Costas Spanos
Javad Lavaei
University of California, Berkeley
University of California, Berkeley
Abstract
Constrained reinforcement learning (RL) seeks nigh-performance policies under safety constraints. We focus on an offline setting where the agent learns from a fixed dataset- common requirement in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion- CrrtusionnRegulrizzee Constrained Offline Reinforcement Learning (DRCORL), which first uses diffusion model to capture the behavioral policy from offline data and then extracts simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high- quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in worldworld scenarios. We open-source our implementation at whhttsp:: / thub. com/ James JunyuGuu DRCORL
Introduction
Offline reinforcement learning (RL) has advanced mecciion-making by learning from pre-collected datasets without online interaction [Fujimoto et al. 2019 Levine et al. 2020). For real-world control tasks (e.g., autonomous driving, industrial control), safety equally critical. Safe RI. addresses this by imposing constraints, often formulated as a constrained Markov decision process (CMDP)
[Gu et al. 2024c Altman 2021], to ensure high performance without violating safety requirements. These can be hard constraints (no violation at each step) [Zheng et al. 2024 Ganai et al. 2024] or
soft constraints (expected total cost below threshold) [Chow
2018 Yang 2020 Koirala
2024 Guo et al. 2025 Shi et al. 2021] We focus on the soft cont-contr setting in this work. et al.
Offline safe RL faces two main hurdles: distribution shift and reward-safety trade-off. Firstly, the learned policy may deviate from the offline dataset's 's state state-action distribution, causing critic overestimation and extrapolation errors [Fujimoto et al.) 2019 Lyu et al. 2022 To address value overestimation, previous methods have either constrained the learned policy to remain close to the behavioral policy /Wu et al. 2019 Kumar al. [0019 conservatively penalized over-optimistic
out-of-distributt (OOD) state-action pairs Kostrikov et al. 2021 Lyu al. [2022] Xu et al. (2022)
*Coreesponing authors: jumyugguo24terreeeeyyycoo edu, shangding-guuberkeee wwwberkeley.cc.o edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Secondly, achieving high returns while strictly respecting safety becomes more challenging when these objectives conflict. Although constrained optimization methods [Achiam et al. 2017 Liu et al.) [022] handle this in online RI., they rely on on-policy data collection, making them not directly applicable to offline settings. Hence, the key question is:
How can we balance reward maximization and constraint satisfaction without risk- ing diitttrbbbtiion actions unsafe behavior in setting where no additional data can be collected?
To address this, we propose Seftusiion-Reulariizze Constrained Offline Safe Reinforcement Learning (DRCORL). First, DRCORL trains a diffusion policy to imitate the behavioral policy in the offline dataset; then it regularizes the learned policy via the diffusion model's score function removing the need for costly sampling from the diffusion model inference. Second, we apply gradient manipulation to balance reward optimization and cost minimization, effectively handling conflicts between these two objectives. Furthermore, the behavioral policy serves as a regularizer, discouraging OOD actions that may compromise safety. While prior works have independently explored gener- ative policy regularization and constraint-aware optimization, they typically treat these aspects as orthogonal. Our framework unifies them by showing that diffusion-based behavior modeling naturally induces a trust-region- regularization that complements gradient-level safety adaptation. This conceptual bridge highlights diffusion modeling not merely as behavioral prior, but as a principled mechanism for
learning.
We evaluate DRCORL on the DSRL benchmark [iu ct al. 2023a comparing against state-of-the-art offline safe RL methods. Experiments show that DRCORD. consistently attains higher rewards while satisfying safety constraints. Our main contributions are: @ We exploit diffusion- regularization to build a simple, high-speed policy with robust performance; and @ We introduce a gradient- manipulation mechanism for reward-cost trade-offs in purely offline settings, ensuring safety without sacrificing returns.
2 Preliminary
A Constrained Markov Decision Process (CMDP) [Altman] 2021
is defined by the tuple (8, A, P,r.c,,), c, where S and A represent the state and action spaces, respectively. The transi- tion kernel P speciifie the probability P(s'|s, a) of transitioning from state to state s' when action taken. The reward function is r S - IR, and the cost function is e: SxA
RR. The discount
factor is denoted by A policy is a function π S + (AA) that represents the agent's 's decision rule, i.e., the agent takes action a with probability {a]s) [s) in state s, and we define II as the set of all feasible policies. Under policy π, the value function is defined as V.- (p)
E Et= 7" o (st, ar] ~ pl. where E (rrcc) p is the initial distribution, and the expectation taken over all possible trajectories. Similarly, the associated Q-function is defined as (s, a) = o(s, a) + > [V." (s') for o E {r, c). In standard CMDP, the objective is to find a policy E II that maximizes the cumulative rewards V (p) while ensuring that the cumulative cost V. (p) remains within a predefined budget I.
In the offline setting, the agent cannot interact directly with the environment and instead relies solely on static dataset D = {(s,, a,, S/, ) consisting of multiple transition tuples, which is collected using a behavioral policy Th (a]s). This offline nature introduces the risk of distributional shift between the dataset and the learned policy. To mitigate this, additional constraint is often imposed to limit the deviation of the learned policy æ from the hehavioral policy Ts. resulting in the optimization problem
max [V s.t. (TI(T) S (P) 1 Wlll where Dit((plgg) the KI--divergence between two distributions defined as DKL. (pllg) Ezze (p((()/)((()))) We use the KI-ddiergeence to penalize the learned policy r's distance to the behavioral policy, though is actually not a distance measure. To address safety constraints, primal-dual methods| Ding et al. (202T), Paternain et al. [2022), Wu et al. (2224] typically reformulate the constrained optimization problem as follows:
max EE V (p)) XBE(V (p)) -1) s.t. (TT(TT S c,
where X 2 0 is surrogate for the Lagrange multiplier. When the safety constraint violated, the multiplier À increases to impose greater penalty the cost, thereby reducing the cost value.

---

3 Methodology
3.1
Diffusion Model for Policy Extraction
Our main idea is to fully exploit the offline dataset to obtain a behavioral policy, and use the behavioral policy to guide the training of the target policy. A policy ([ass) is a distribution on the action space. Previous work estimated the behavior policy with maximum likelihood estimation Fujimoto and
Gu [021] or leveraged a conditional variational autoencoder Kingmal [2013), Sohn et al. [2015].
Here, we exploit the concept of diffusion models (Sohl-Dickstein et al. [2015], Ho et al.] [2020], [Sonn] al. [020bb learn the unknown behavioral policy (u(a)s) given its strong generative capabilities.
Diffusion models have emerged as powerful generative tools to generate data sample 2o p(zo)
with few-shot samples. They work by using a forward process (Ire/20) to perturb the original distribution to a known noise distribution. Subsequently, this model generates the samples using the
reverse denoising process P& (1 (x)). The forward process can generally be written with a forward stochastic differential equation (SDE)
B(t) 2
dx
xdt + /B(t)dW,,
(3)
where 3()): [0. T + R+ is scalar function and the process { ee is a standard Brownian motion. Our forward process is the discretized version of SDE in Eq. (3) perturbing the original distribution to Gaussian noise. For example, if we choose variance preserving SDE for the forward diffusion process as in Ho et al. [2020), each step xy is perturbed with the noise 2 ~ NO, I) to obtain
It-1 = VaLTe + BBe where B, = 1 - a E 1). We denote ā. = I a and 3e = 1 - āt.
Therefore, we can rewrite 2, = vārzu + VBc, where & (O, I) follows the standard Gaussian distribution. The reverse denoising process is optimized by maximizing the evidence lower bound of the log-likelihowe [E[log Pyù (zo)] defined as (yg(so:))) loog (pos (No.T) ) (/((rrrrrrrr)) [=oo)]] We can then rewrite the loss function into the weighted regression objective in Eq. (4) and transform the problem
into training a denoising function e predicting the Gaussian noise ex:
((b) = Euu (()(e)))
<ell"], ~ ~N I),
where (t) is the weight function. Then the reversing denoising process can be formulated as dt-1
ya, (32²- VI-as <--a: Gp (2re,)) f)) + V where 2, -N, 7). Using this notion, we similarly use the diffusion model to diffuse the action space and sample actions given the current state with the reversing process. In Wang al. 22022. the authors learned diffusion policies from the offline dataset using guided diffusion. The diffusion policy here is (als) where Psi (a.-1 [0,,)) s) is a Gaussian distribution with mean myy (a,, {|s) and variance En (ar, f|s) See also Lu et al. (2023). [Hansen-Estruce et al. [2023. where diffusion policy is used for inference in policy eval valtin.oo. The limitation of these methods is that the diffusion models are slow in inference speed, even under the improved sampling scheme in Song al. [2020a], Lu al. [2022] the reverse denoising process normally takes at least 10 steps. Therefore, in this work we mainly use diffusion
(ar; 0, 7) II" Pe (a.- (a,, s).
models for pretraining and learning the behavioral policy m.. For state-action pair (s, a) the offline dataset DD we train our diffusion policy model by minimizing the loss
L = (s) De Eegunif(o.) [o(t)llc t|s) - c 12].
(5)
where d, = Vaya + BB and = ~ (O, I). We assume that the diffusion model can perfectly learn the behavioral policy Th. as shown in De Bortoli 2022]. due to the fact that the target distribution lies on compact manifold, the first-order Wasserstein distance between the learned policy and the target policy converges to 0 as the discretization length approaches 0.
3.2 Diffusion Regularization
The work Chen al. [2023] showed under the offline RL setting that one can train simple policy using the pretrained critic and diffusion policy imitating the behavioral policy. The key step is to use the reverse KL divergence to regularize the target policy to be close to the behavioral policy. The for- ward KL. is the KL.ddivergence DRI.(TT (x=x((r=m=l>)]lmm (-Is)][ (+{(s)) while the reverse KI. is ( (-Is)). As shown in Chen et al. [2023), the forward KI. leads to mode covering issue while backward KI. encourages mode- mode-secking behavior, although the latter is harder to optimize. Therefore, in this
3

---

work we also choose the reverse KL for regularization, We also constrain our policy family to simple Gaussian policy class II {To(a]s) = NN(a; Es Now, we denote the learned diffusion model's score function as 6 (az, Us) and the corresponding diffusion policy as (eg (a)s) Then, the reverse KI. between the policy mo (a|s) and the approximated behavioral policy 14g (a|s)
can be written as - (T((s())s)) + (= (-s)). For Gaussian policy, the first part self-entropy term H. can be directly computed in closed form. For A R2, we have
H(To(-1s) /
1 log(det(Eg(s))) + d log (og(2T) + d
log To (o
(6)
2
Using the reparameterization trick for Gaussian random variables, we can also rewrite the cross entropy term H (To00110))))))) (s), (w((1s) as
H (TT (-1s) (-1s)) (1,) log p (a|s)] = E.=. nll
log pn (mg + 21/22
(7)
Finally, to obtain the gradient with respect to the reverse KL divergence we have
DoL (To (- (+1s) = V (T
- VoH((To(-1s))
(8)
- Va log uy (ma (s) + 23/ 2|s) (a(mp(s) / 2) - Vo log(det 2i (s)).
In our implementation, the diffusion model serves as a behavioral density estimator, and we use its learned score function V log po (als) tractable surrogate for computing the gradient of the reverse-KL. regularization. The work Song et al. (2020b] shows that diffusion models essentially
estimate the score function:
V log 2 (x(2)) =
Co (zz. t).
(9)
Hence, we can substitute the denoising function ep into Eq. {8) to directly compute the gradient.
3.3 Safe Adaptation
To design our algorithm, we define the reward and cost optimization objectives respectively as follows:
Reward: max EANND well
[ViT( (s) (
(10)
B
Cost: max I DI TEll
(V) - 1) DKL (KL(T(HH))((((11))
(11)
We consider the reverse KI. divergence term in Eq. TOp
Polley Space
and Eq. (TT) as a regularization term to penalize the policy for deviating too far from the behavioral policy Tb. In prac- tical implementation, we use the learned diffusion policy Me to replace Tr and compute the gradient for optimiza- tion using Eq. (8). As discussed in Gu et al. [2024a], to address the conffict between optimizing reward and cost, it is essential to directly handle the gradient at each optimiza- tion step. We also adopt the gradient manipulation idea in Gu et al. [2024a]. but we do so under the offline setting where no longer have access updating our critic with the on-policy data. The gradient manipulation method can be generally described as follows. For each step where we need to optimize the reward, we update our gradient with 0 + 0 + 7/9r, and when we need to optimize the cost our gradient is updated with 0 <- 0 + 1e9e- The gradient ma- nipulation aims updating the parameter 0 with a linear combination of the two gradients as 3rgr + Bege- We can
Gradkend Maripulation
Optimel haliy Gradlent
Safe Pollcy Boundary
Optimize Reward
Figure 1: Illustration of the soft switch- ing between safe and unsafe policy re- gions.
use the angle o = cos-1
(g..00)) between the two gradients to judge whether the gradients are [1g. goll

---

conflicting or not, namely if o > 90º the two gradients are conflicting and otherwise they are not conflicting, Especially, the final gradient g is computed via the rule
if Φb E (0, 909)
(12)
if Øb E [90", 180%),
where g, = 9- [o Ie is 's projection on the null space of 9r and g. = 9c Ts- Dr is Or 's projection on the null space of gc. In Gu et al. [0244]] under the assumption of the convexity of the optimization target, one can ensure monotonic improvement using gradient manipulation. Therefore we also employ the gradient manipulation method to update our policy. With the procedures outlined above, we present our Algorithm| [] where the safe adaptation step visualized in Figure | and given detailed in Algorithm given in Appendix B Furthermore, we provide the following theoretical results derived from our algorithm, where the proof is deferred to Appendix [A.1
S x A + [0, Cmaxl is bounded and non-negative Let ((ass) be the output policy of Algorithm 7 Suppose that there
Proposition 3.1 (Cost Upper Bound), Assume that the cost function
exists Ediss O such that (* T-(()s)) S eais and DKL. (T6(-1)))) #t (11s)) S <diar. Let e
max, En (-la) [A (s,a)), a)). where AT (s, a) is the advantage function under policy n, defined as:
(x a) = (8, V2((), Then, it holds that:
(p) S (P) (Cmax + (
(13)
(1 (1-7)2
This proposition establishes that if the learned policy is constrained to remain within neighborhood of the dataset's ss behavior policy, its safety performance is at least guaranteed to match that of the behavior policy underlying the dataset.
To theoretically ground our algorithm, we ana- lyze its convergence properties in the tabular set- ting, where the state and action spaces are finite (i.e., (1, < oo). Under softmax parameter- ization, we derive convergence guarantees when optimizing the value function using the natural policy gradient (NPG) algorithm.
Algorithm 1 DRCORL
1: Input: Dataset DV, slack bounds h+,, h"
2: Pretrain diffusion model Ev
3: // Behavior Cloning
4: Pretrain Q Q
5: // Pretrain critic
Definition 3.1 (Softmax Parameterization). Un- der the tabular MDP setting, the policy follows a softmax parameterization, where the policy π parameterized by 0 is expressed as To(als) = E.'e (xp())).a)))
6: for each gradient step do
7: Sample mini-batch B
8: Update crities for reward and cost
+ R. The policy exp(6(s.a))
9:
f- SafeAAdaptation(....
10: Update f t- 0 + n9
11: end for
We use natural policy gradient method (Kakade and Langford [2022] to update policy, where the policy parameter 0 is updated as 0 +- (FF) V (p) at each iteration, where Fd = -E-ppaws V log (a)ss) Ve log (als)) T] is the Fisher information matrix, and the operator is the Moore-Penrose inverse.
Since the score network is trained on the empirical offline distribution, there exists a potential shift between the dataset and the evolving policy distribution. The regret bounds in Theorem 3.1 explicitly account for this effect through the term cottine which measures the approximation gap between these distributions. In practical implementation, this term remains small when the dataset adequately covers the behavioral policy, ensuring that safety guarantees degrade gracefully even under mild mismatch. Theorem 3.1. Let स be the weighted policy obtained after T iterations of Algorithm D with proper step-sizes. Suppose that the offline dataset has a size of [Du| O C (n((F1/8)) for some 8 E (0, 1), Fagitne (1 (1-)"/
where FF represents the critic function family, Then, with probability 2 - 8, the optimality gap and constraint violation satisfy thut
|STAI
V. (p) - VV (p)) < (Coflline) +
(1
(14)
(1-2)31
|S||A1 (1-7)3T
BIV (p) - b S ( +0
(15)
(
5

---

where Eoffline denotes the approximation error of policy evaluation induced by the offtine dataset D
For simplicity, we drop the reverse KL term to ensure the policy's closed-form update under NPG in our proof. The SafeAdapttiio step in Algorithm T is specified algorithm 2 givve in Appendix B We specify the definition of the weighted policy ग in Eq. (4T) given in Appendix A.2 We can interpret the theorem as follows: by selecting appropriate slack bounds h and h the policy optimization process will, over time, primarily remain in the reward optimization and gradient manipulation stages. As a result, the cost violation can be effectively upper bounded. Simultaneously. by ensuring that the policy is updated using the reward critic's gradients for the majority of iterations, we can guarantee that the accumulated reward of the weighted policy closely approximates that of the optimal policy.
Practical Algorithm
In this section, we present the detailed procedure for the implementation of our main algorithm DeftusionnRegularzzzel Constrained Offline Safe Reinforcement Learning (DRCORL), as outlined in Algorithm 1 where we provide the Adaptaattoi step outlined in Algorithm@ in Appendix This includes both the pretraining stage and the policy extraction stage.
4.1 Pretraining the Diffusion Policy and Crities
In the pretraining stage, we first use the offline dataset D to pretrain the diffusion policy (a (a|s) to simply imitate the behavioral policy by minimizing the loss in Eq. (5). Then we also pretrain the critic functions Q and Q? but we pretrain the reward critic with Implicit Q-Learning (IQL) Kostrikov et al. [ and pretrain the cost critic with TD learning with pessimistic estimation. We utilize IQI. to train the reward critic function by maintaining two Q-functions (Q", Q and one
value function V as the critic for the reward. The loss function for the value function V," is defined as:
LVz- E,,,, Es D' [4_ (min (s, a), QT, (s, a)) (s))l,
(16)
where [5( = |r - T(u < 0)lu², and T E [0, 1] is quantile. When 7 = 0.5, L3 simplifies to the L2 loss. When 5 > 0.5, LA encourages the learning of the 7 quantile values of Q. The loss function for updating Q7, is given by:
fqz Re..a.''DD (s),() + +s), QT, (s,a))))
(17)
This setup aims to minimize the error between the predicted Q-values and the target values derived from the value function V.". We employ IQL to pretrain the reward critic function, thereby approxi- mating the optimal Q-function Q without incorporating safety constraints. Additionally, we pretrain the cost critics using the temporal difference method and double-Q learning SSutton [1988]. However, in the offline RL setting, we adopt pessimistic estimation approach for the cost critic using a positive hyperparameter to avoid underestimation as stated in Eq. (18), thereby preventing the learning of unsafe policies. The cost critic can be updated by solving the optimization problem:
min E,.,,,,'' () + (s',aa)) - O- (s.a)))l c..a,a 1 (s,a)).
(18)
HEll
4.2 Extracting Policy
Now, we extract the policy Ty from the diffusion model 6y and the pretrained critic functions. Note that at this stage need to optimize the reward while preventing the unsafe performance. Therefore, we define two optimization targets, one for the reward and one for the cost. The reward optimization target defined as maximizing the critic value regularized by the KL-divergence with respect the behavioral policy, a smaller value of the temperature / refers to the higher conservativeness of our algorithm:
max E
(s, 3
[TO, mapls) (
(19)
where ((TO, Mu [s] denotes the KI-diverrence DKL. (mo(-)s)) [s)] (-s)) for abbreviation. Similarly, define the cost optimization target follows. We aim minimize the cost critic value regularized by the behavioral policy:
- -(Q 2 (s, - 1) - 3 ( Mpls)
max Eg
(20)

---

Using the result obtained in Section 3.2 with respect to 0 using the score function of the diffusion model ey and the entiic function. We take the Gaussian policy family under a constant variance. For example, with IT == (To(a)s) Na; mo(s), Es)s we can simplify the gradient to
we can obtain the gradient of Eq. (9) and Eq. (20)
[ (VaQ7 (s,a) tho(s, VOTO(S)]
9r = E Saawa
(21)
(- (-vaQ% (8, + ( V VO
9c = R B,aaT
where hv(s,a) =
Vo. cv (oe, [/s)[o=o- In Eq. 21 at denotes the action with perturbed noise,
a = Vāra + BrEr, & ~ 1), and the notation V ( denotes the gradient of the action given state with respect 0. By assuming m (s)) where = ~ (O /), we obtain that VOTO (s) = (sm + ( Our problem involves two competing objectives: maximizing reward (Eq. I0 and minimizing cost under safety violations (Eq. 20). To reconcile these goals, we adopt the gradient manipulation method from Gu et al. [0024a], originally proposed for online safe RL, as detailed Algorithm 2 This method introduces slack variables a h+. When (p( S 1--4", the policy is considered safe, and reward maximization is prioritized. If V (p) 2 I ht, we instead focus on cost minimization. Within the transition band / h < (p) S I 1 ht,, we apply gradient manipulation (Eq. 12) to balance the two objectives (Eqs. 19 and 20
(s))
key challenge is accurate safety assessment in the offline setting. where off-policy data may lead to cost underestimation. Ideally, the estimated critic satisfies Q Q + c, with zero-mean However, under Bx.a= [min (T, a)] S. min aww [2_ (s, a)], the error becomes biased, Ieading to underestimated cost values Thrun and Schwartz (2014). Temporal difference learning can amplify this bias. While conservative Q-laarning mitigates underestimation, it often yields sub-optimal policies. Crucially, unlike reward critics, where relative ranking suffices, underestimating cost critics can produce unsafe policies. To address this, we adopt conservative evaluation approach inspired by the UCB (Upper Confidence Bound) technique Hao al. (2019). Specifically, we train an ensemble of cost critics Q for = I..... E and define the UCB estimator as Q QUCB (s.a) = Q7 a) + k. Std (Q (s. a)), where (s, a) is the ensemble mean and k controls the confidence level. We then compute QUB (p) Ex-ppa QQ: .UBB (s, a)] and compare it against the cost budget to determine whether to prioritize reward or cost. This full procedure is outlined in Algorithm 2
5 Experiments
5.1 Performance On DSRL Benchmarks
Environments. We evaluate method on the offline safe RL benchmark DSRL Liu et al. [023a]. We conduct extensive experiments on Safety-Gymnasium Ray et al. (2019) and Bullet- Safety-Gym Gronauer [022]] We evaluate the score of different methods using the normalized returns and normalized costs. The normalized returns and costs are defined as Rnormalized = (Rg- Rmin) ) [Rmax- Rmin). Cnommalized = (C₂ mm c). is regularizer for the case when 1 = 0, and we set e = 0.1. The reward R2 is the accumulated return collected within an episode
R₂ = - Fr. Similarly, Cx = Din is the accumulated cost collected within an episode. Rmax. Rmin are the maximum and minimum accumulated returns of the offline dataset within an episode. We normalize the accumulated cost and return to better analyze the results. If the normalized cost is below 1.0, we can consider this policy as safe.
1=
Baseline Algorithms. We compare the performance of our algorithm against existing offline safe reinforcement learning (RL) algorithms under the soft constraints setting. The following six baseline algorithms are considered: 1) BC-Safe (Behavioral Cloning): cloning the safe trajectories within the offline dataset. 2) BCQ Fujimoto al. 2019]: this approach extends behavioral cloning with Q-learning framing safe reinforcement learning as constrained optimization problem. A Lagrangian multiplier Stooke et al.
[2020) is used to balance the reward maximization objective with the cost constraints. 3) BEARI Kumar et al. 2019:: an extension of BEAR that incorporates a Lagrangian multiplier Stooke et al. 2020 to control cost constraints, enabling safe policy learning. 4) CDT
(Constrained Decision Transformer) Liu et al. 2003b]: an adaptation of the Decision Transformer
7

---

Table I: Normalized return (t: higher is better) and cost (J: lower is better; threshold at 1) for each policy across tasks. Results are averaged over three cost limit scales, 20 evaluation episodes and random seeds. Policies with normalized cost < 1 (safe) are bolded; among these, the highest-reward safe policy per task is highlighted in blue. Policies with cost > 1 (unsafe) are shown in gray.
BC-Sal
BEARL
BCQ-Lap
CPQ
COpiiIcC
CDT
CAPS
CCAC
Our
Task
aeword
reward
reward
roward
reward
reward
CarGeall
0.38
0.71
4.45
3.27
0.68
0.45
2.31
0.65
3.75
1.35
6.52
0.51
PoinsBunoo
0.37
1.37
0.35
6.71
0.17
6.00
0.56
11.63
0.00
3.55
0.67
15055
0.16
1.00
0.71
4.27
0.81
PainCieeel
0.86
3.79
-033
17.84
0.45
8.13
0.23
6.77
0.85
18.30
0.51
0.00
0.50
0.14
0.62
7.58
0.53
1.40
PairGool
0.55
0.200
0.76
3
0.59
5.06
0.41
0000
0.52
5.26
0.08
4.35
11:00
0.53
0.77
5.00
0.85
PoinaGra22
0.50
3.15
0.60
12.33
0.65
10.60
0.34
5.10
0.38
4.62
0.52
1.45
0.23
1.91
0.90
4.88
0.52
(L00
Andel
-1.01
9.00
8.39
-1.01
0.59
0.91
0.97
0.36
0.39
0.55
HallChecaiiVel
0.95
0.71
0.61
HopperVel
0.21
0.36
124.05
14.91
0.34
7:83
0.21
1.12
0.68
Wallerbival
0.78
01.08
0.76
0.50
0.90
0.07
0.08
1.95
0.12
2.34
0.73
1.55
0.78
9.00
0.21
0.36
0.74
01300
Celinn
0.97
11:10
0.49
7.4%
0.35
0.92
6,93
0J08
0.99
1.10
0.86
01303
0.18
0.99
11.00
BalClekk
0.55
0.93
0.50
5,79
0.68
3.70
0.64
0.07
0.47
4.71
0.68
2,05
0.56
0.18
0.73
0.14
0.78
(L00
Carlirdo
0.73
2.588
0,70
0.47
0.73
7.34
0.80
0.48
Chen et al. [202]] for offline safe reinforcement learning, incorporating cost information into tokens to learn a safe policy. 5) CPQ (Conservative Policy Q-Learning) Xu al.) [2022]: this method uses conservative Q-learming to pessimisticalyy estimate the cost critic and updates the reward critic only with safe cost values, ensuring the policy adheres to safety constraints. 6) COptiDICA et al. [2022]: based on the DICE algorithm, this method learns a stationary distribution for the safe reinforcement learning problem and extracts the optimal policy from this distribution. 7) CAPS CAPsiChemingui et al.) [025]: optimizing different constraints with shared representations. 8) CCAC Guo et al. [2025] : safety RL algorithm matching state-action distributions and safety constraints. We present the comparison across different baselines in SafetyGym and BulletGym environments.
Result Analysis. Table ! summarizes the normalized accumulated reward and cost per episode across tasks. Overall, our algorithm consistently achieves high rewards while reliably maintaining safety constraints. Notably, our method significantly outperforms baselines in tasks such as CarGoall, CarGoal2, PointGoall, PointGoal2, and BallCircle, ensuring both optimal reward and constraint satisfaction. However, it is fair to acknowledge that BCQ-Lag shows slightly superior reward performance in the Walker2dVel task, though such performance does not generalize reliably to other tasks, often violating safety constraints. Similarly, CDT and CAPS achieve good safety in some scenarios, such as CarGoal2 and An/Vel, yet underperform significantly in reward optimization in other tasks. Overall, our method demonstrates robust balance between safety and reward optimization across diverse benchmarks, highlighting its practical applicability.
Computational Efficiency. We benchmark inference speeds across algorithms using 1, 1,000 sample inputs on the HalfCheetah task. Apart from the 6 baselines above, We also compare the computation efficiency against diffusion model-based algorithms TREBI Lin al. 2023 and FISOR Zheng et al. 2024). While baseline methods like BCQ and CPQ achieve the fastest inference due to their lightweight MLP-based actors, they compromise safety or reward performance. In contrast, our algorithm strictly adheres to cost constraints without sacrificing reward quality. Compared to diffusion or transformer-basee approaches our method are superior in inference speed, narrowing the gap between expressive generative models and efficient MI.P architectures. The result is in Figure (a).
5.2 Ablation Study
Impact of Different Cost Limits. We evaluate our algorithm's 's performance under varying cost limits 1= 10,20 20, and 30, analyzing the learned policies' behavior for each budget setting. The ablation results are presented in Figure Across all cost limit choices, our algorithm consistently achieves zero violations of the safety constraints, demonstrating its robustness to varying cost thresholds. This highlights the adaptability and reliability of our approach in maintaining safety compliance. While COptiDICE also strictly adheres to the cost limits, our method consistently outperforms it in terms of normalized return, demonstrating its superior ability to balance safety and reward optimization.
8

---

Temperature Parameter Setting, We compare three diffusion- diffusion-temperain temperature schedules: (i) constant,
3t = 0.02; (ii) square-root growth, 6 0.01 Di and (iii) linear growth, 6r 0.01 t + 0.04, held fixed within each epoch. As shown in Figure 3 (Appendix C). all three schemes yield similar performance. Additional temperature ablations are reported in Appendix C.T
Choice of Slack Variable. We introduce slack bounds that reward maximization applies when Vuorm I 1 - h and cost minimization when Vuorm 2 1 + ht Both h- and h+ are initialized to 0.2 and linearly decayed to zero; ablate over initial values h {0.1, 0.3, 0.5, 0.73. Figure (Appendix C illustrates these results, with further slack-value studies in Appendix
C.T C.T
Algorithm Performance Evaluation
1.0
Safe
Norre Ened
0.8
0.0
Inference Timm (secondyyKK semplest
1+32
1-20
(a)
Figure 2: (a) Computational Efficiency vs. Performance Trade-off. Normalized score (y-axis, combining reward and safety metrics) versus inference time for generating 1,000 actions (z-axis). (b) Normalized return and cost under varying cost limits (1 = 10, 20, 30). Since the cost is normalized relative the corresponding cost limits, the safety threshold is consistently 1.0. The dashed line represents the safety boundary.
Related Work
Diffusion Models in Offline RL. Diffusion models' powerful generative capacity for complex data
has made them increasingly popular for modeling diverse offline datasets, valuable either for planning
Janner et al. [2022]. He et al. [2024). Ajay et al. [2022] or expressive policy modeling Chi et al.
[2023] Wang et al. 2022]. Fang et al. [204]), Hansen-Estruch et al.) [2023). Chen et al. [2023] Lu et al. [002]]. Their main drawback is inherently slow sampling due to many reverse-diffusion steps. To accelerate inference, methods like DDIM et al. [2020a] reduce steps via subsequence sampling. while APM-Solver Lu et al. [2022) uses optimized ODE solver to reach 10 steps. Useful extensions to safety-critical settings include the cost- constranndd diffuser framework of Lin al. 2023], SafeDiffuser Xiao et al. [2233 and OASIS's conditional diffusion for dataset synthesis
and careful distribution shaping Yao et al. 2024].
Online Safe RL. Safe RL in online environments has been extensively surveyed in Gu et al. 2024c
common approach is constrained policy optimization -egio..nnee.......... CPO Achiam al. [2017] integrates TRPO (500ulmanlmn (2015) for robust updates, and CRPO Xu et al. [2021] obviates complex dual- variable tuning. Primal-dual Lagrangian methods dynamically adjust multipliers on the fly to enforce
safety criteria [Chow et al.] 2018 Calian et al., 20200 Ding et al. 2020 Ying et al. [202 Zhou et al.)
2023], and more recent algorithms aim to better balance cost and reward via gradient manipulation Gu et al. [2224a or refined trust-region formulations Kim et al. [2023 2024]. However, these methods depend on extensive environment interaction and accurate critic estimates, thus limiting
their practicality under stringent real-world safety and data collection cost constraints.
Safe Offline RL. Considering safety specifically the offline learning setting, Xu et al. 2022 Guan
et al. (223) penalized OOD and unsafe actions identified in the dataset. Diffusion based safe policy
learning was adapted by [in et al., (0033 for cost-constrained scenarios, and decision-transformee
architectures have been applied to safety considerations via et al. (203b)) Energy-based diffusion
policies can enforce hard constraints, for example, through weighted regression [Lu et al. 2023
Zheng et al.) 2024 Koirala et al. 2024 More recent approaches learn by modelling conditional

---

sequences as in Gong et al. [0025] Zhang et al. [2023] or by strategically sharing representations
across constraints for improved adaptation Chemingui et [2025]
Conclusion and Limitations
We present DRCORL, novel approach for offline constrained reinforcement learning that learns safe, sesh-perforance policies from fixed datasets. DRCORL employs diffusion models to faithfully capture offline behavioral policies and distills them into simplified policies for fast inference. To balance reward maximization and constraint satisfaction, we utilize a gradient manipulation strategy that adapts dynamically without extensive hyperparameter tuning. Extensive experiments across standard safety benchmarks show that DRCORL consistently outperforms existing offline safe RL methods, achieving superior rewards while satisfying safety constraints.
We also identify several limitations offering avenues for future work. First, DRCORL's pretraining stage comprising inffussion-mdde training and critic estimation, though producing lightweight policy at inference and avoids unsafe online exploration, incurs additional computational overhead. Future work should aim to optimize and parallelize the pretraining phase to further reduce its cost. Second, guaranteeing zero constraint violations remains challenging in offline RI., particularly when dataset quality and coverage vary. Developing methods with greater robustness imperfect data and tighter safety guarantees is a promising direction. Despite these challenges, our contributions lay a strong foundation for advancing both generalization and safety in constrained offline reinforcement learning. Although DRCORL inherits the stability of diffusion modeling, its effectiveness still depends on the representational quality of the offline dataset. In extremely sparse biased datasets, the diffusion model may under-tep safe modes. Future directions include integrating conservative density estimation or uncertainty-awee aware diffusion objectives to enhance robustness in lowcoveraae regimes. More generally, we view diffusion regularization as a flexible template: alternative generative priors such VAEs or normalizing flows could replace the diffusion estimator under the same theoretical framework, potentially trading expressivity for computational efficiency.
Finally, we hope this work provides unifying perspective linking generative policy modeling and safety-constrained optimization, paving the way for scalable and verifiable offline safe RL.
8 Acknowledgment
This material based upon work supported in part by the U.S. Army Research Laboratory and the U.S. Army Research Office under grant number W91 INF2010219, It was also supported by the Office of Naval research under grant number N000142412677 as well as NSF.
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference muchine learning, pages 22-31. PMLR, 2017.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-766 2021.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211. 2022.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Dan A Calian, Daniel J Mankowitz, Tom Zahavy, Zhongwen Xu, Junhyuk Oh, Nir Levine, and Timothy Mann. Balancing constraints and rewards with meta-gradient d4pg. In International Conference on Learning Representations, 2020.
Yassine Chemingui, Aryan Deshwal, Honghao Wei, Alan Fern, and Jana Doppa. Constraint- adaptive policy switching for offline safe reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 15722-15730, 2025.
10

---

Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023.
Jinglin Chen and Nan Jiang. Information-thheortiicc considerations in batch reinforcement learning. In International Conference Machine Learning, pages 1042-1051. PMLR, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:150444 15084-15097, 2021.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating trust region for offline reinforcement learning. arXiv preprint arXiv:2405. 10000./96900 19690, 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.0531/1 2022.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovie. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 33:8378-8390, 2020.
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International conference artificial intelligence and statistics, pages 3304-3312 PMLR, 2021.
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing- Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXiv:2405.20555, 2024.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Ze Gong, Akshat Kumar, and Pradeep Varakantham. Offline safe reinforcement learning using trajec- tory classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 16880-16887, 2025.
Sven Gronauer. Buller-safety-ge -ym: A framework for constrained reinforcement learning. Technical report, MediaTUM, 2022.
Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21099-21100 2006, 2024a.
Shangding Gu, Laixi Shi, Yuhao Ding, Alois Knoll, Costas Spanos, Adam Wierman, and Ming Jin. Enhancing efficiency of safe reinforcement learning via sample manipulation. NewrIPS, 2024b.
11

---

Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024c.
Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems, (6:33758-33788 2023.
Zijian Guo, Weichao Zhou, Shengao Wang, and Wenchao Li. cestraint-conditionnd actor-crite for offline safe reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Philippe Hansen-Estruch, llya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. ldql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprini 11X15:2304.10573 2023.
Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. Advances in neural information processing systems, 32, 2019.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances neural information processing systems, 36, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1116840-65541 2020.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint (xXiv:2205009911, 2022.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-274, 2002.
Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based based safe distributional reinforce- ment learning for multiple constraints. Advances in neural information processing systems, 36: 19908-19933 2023.
Dohyeong Kim, Mineui Hong. Jeongho Park, and Songhwai Oh. Scale-invariant gradient aggregation for constrained multi-objective reinforcement learning. arXiv e-prints, pages arXiv-2403, 2024.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, and Cody Fleming. Fawac: Feasibility informed advantage weighted regression for persistent safety in offline reinforcement learning. arXiv preprint arXiv:2412.08880. 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning arXiv preprint a1Xiv:2110.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy g-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Jongmin Lee, Cosmin Paduraru, Daniel Mankowitz, Nicolas Heess, Doina Precup, Kee-Fung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08955, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.06644 2020.
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shanggin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference Machine Learning, pages 21127-21152. PMLR, 2023.
12

---

Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306. X3v:2230600903,, 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang. and Ding Zhao. Con- strained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 21611-2166 1-6633.. PMLR, 2023b.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, and Ding Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5788 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825-22888 PMLR, 2023.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 35:1711-1724, 2022.
Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Trunsactions on Automatic Control, 68 (3):1321-1336, 2022.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 1rXtv:1910.017088, 7(1):2, 2019.
John Schulman. Trust region policy optimization. arXiv preprint arXiv: 2015.
Tianyu Shi, Dong Chen, Kaian Chen, and Zhaojian Li. Offline reinforcement leanning for autonomous driving with safety and exploration enhancement. arXiv preprint arXiv:2110.07067, 2021.
Jascha Sohl- Sohh-Diccstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2265.. PMLR, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in newral information processing systems, 28, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020b.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143. PMLR, 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 9-44, 1988.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 connectionist models summer school, pages 255-263. Psychology Press, 2014.
13

---

Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint (rXiv:2208.06193 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019.
Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shanggin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Off-policy primal-dual safe reinforcement learning. In The Twelfth International Conference Learning Representations, 2024.
Wei Xiao, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. arXiv preprint arXiv:2306. 00148, 2023.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753-8761 2022.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 11491. PMLR, 2021.
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 2020.
Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 777784411-74785 2024.
Donghao Ying, Yuhao Ding, and Javad Lavaei. A dual approach to constrained markov decision processes with entropy regularization. In International Conference on Artificial Intelligence and Statistics, pages 1887-1909, PMLR, 2022.
Qin Zhang, Linrui Zhang. Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301.
2023.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with reasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations, 2024.
Zixian Zhou, Mengda Huang, Feiyang Pan, Jia He, Xiang Ao. Dandan Tu, and Qing He. Gradient- adaptive pareto optimization for constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4443-11451, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Proof for the results and the statements made in the abstract and introduction are given in the main body and appendix section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation of the work discussed in the conclusion part of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Assumptions are mentioned right before the statements of the theoretical results. Proofs are mainly provided in the appendix due space limits.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
We provide all the essential experimental setting to reproduce the experimental results covered the main results, including the choice of hyperparameters, experimental configurations and so on. Algorithms can be implemented using the open-source benchmark OSRI [Lu al. 2024
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release and open-source the code and training logs after the review
process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//gudess/Cdeettbmisss wow.aaammmiioooooooooooooooee for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nips.cc/pput cttp://sccpbblccccuieeeeeees guides /ttp:///lai/iiiiie//////eueaaaaa
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify the settings of experiments in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We do not report error bars directly, instead five seeds on each task and report the average performance across five seeds to reduce randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We indicate the type of compute workers and the amount of compute required for each individual experimental runs in the Appendix C We also disclose that the full research project require the same compute reported in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp//////eeepppe.cccppp///////
Answer: [Yes]
Justification:
The paper strictly adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This work does not use such existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This work's core method does not involve LLM as any important, original or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Lemma 2], we can write the A Proofs of Theoretical Results
In this section, we provide the proof of the theoretical results in the main body, We make this assumption throughout the proof of Proposition 3.1 and Theorem 3.1
Assumption A.1. During the training process, assume the policy ne resides in the region Ill within the policy space. For all x E Ille, 8 E S, and a E A, the following holds:
log x(als)
S Edisr.
(22)
TT(als)
Equivalently, it means that for all s S and E A:
exp(-Edis) _ T[als)
exp(Edist).
(23)
(a(a)))
A.1 Proof of Proposition 3.1
Proof. Using the performance difference lemma (Agarwal al. 2021 difference the value functions under policies स and Thb as
1 vi (p) V (p) = Ea Ea ( (11) [4 (s, a)],
(24)
where d is the discounted stationary distribution defined as (s) = (1- 7) Pr(st = s|s0 p). The total variation distance (TV-distance) between two distribution is defined as = [A (als) (als) da, which is proportion the el d'-ditaance between the two distributions. Using the Pinsker's 's inequality [Csiszár and Körner [011]. we can bound the
TV-distance with the Kl.-divergence, i.e., for any two distributions μl, v, have that
(xll()lll)
Drv(allv) S
(25)
2
Hence, it holds that Drv(#1lm) S /Caia/2 /2. Then, following the same procedure as Achiam et al. 2017 Proposition 21, we can obtain that
1
27c
V.T (p) V.a (p) 2
E Ba B (1
B Bonds (PTv(
(1 7)2
(26)
Using Eq. (25). we can bound term II with
2 E Bowdy (Dvv(((()s)( (1 7)2
rel v 2edist (1 7)2
(27)
As for term I, we can bound it with
1
/ (s, (s,
7 Egwdy B. Ba (- (4 (s,
5
E Rondy
1 7
I (ms(als) [a]ss) Q* (8. a) da
E Band
1
E(D. 4mm
Cmax
E Exway / |[alls) (als)\dda
(28)
7)2
20 20max E Exndy [Drv((((s))[((([1)))
(1 7)2
Cmax V /2Edist
(1 7)2
22

---

Combining Eq. 28 with Eq. 27
we can finally obtain that
Cmax v 2ist
7 sdv V /2di
(Cmax 7c V
V (p) V. (p)
(1 1-- -7)²
(1 -)2
(1 - 7)2
which concludes the proof Proposition 3.1
A.2 Proof of Theorem 3.1
Our proof is based on theorem in Gu et al. 2024b] which is considering a online safe reinforcement learning setting. Before presenting the proof, we first introduce some notations and concepts that will be used throughout this section. We index the iterations of Algorithm by 2.2,....,
Let (s, a) and Q% (s, a) denote the estimators of the critic functions Q, (s, a) and Q (s, a), respectively, at the t-th iteration under the policy Tee
Denote the gradient at step for the reward optimization of and the gradient step t for the cost optimization as g'-
Let 1y represent the learning rate for the NPG algorithm.
We categorize the iterations of Algorithm into four cases based on the optimization scenarios:
1. Safe Policy Region, i.e., when Ve
< [ - h- We denote the set of iteration indices
corresponding to this case as Ssate-
2. Unsafe Region, i.c., when Ve. 2 14h We denote the set of iteration indices corresponding to this case as Sunsafe-
3. Gradient Manipulation Aligned Gradients, i.c., when the cost function is close the cost limit threshold and the angle between the updated gradients is less than 90%. We denote the corresponding set of iteration indices Salign
4. Gradient Manipulation Conflicting Gradients, i.e., when the cost function is close to the cost limit threshold and the angle between the updated gradients is greater than 90". We denote the corresponding set of iteration indices as Sconfiit-
For every 21, T), , the iteration index / must belong to one of the four sets: Safe. Sunsafes Salign. or Sconflic-
Assume the reward function S x A -
[0, M] and the cost function S A + [0, MM non-negative and bounded by M. This is standard assumption in the tabular setting.
We define the Bellman operator T for policy evaluation (applicable to both cost and reward functions) as:
max f(', ad)
af( r(s,a) + t7By.p((ss..))
(30)
a'EA
Proof. Under the softmax parameterization, the natural policy gradient update [akadee and Langford 2002 can be expressed as
To (als) To, (als) exp (n)Q (8, a)/(1
(s)
7))
(31)
Zo(s,a)
where the normalization constant (s) is defined as:
(10mar (8, (1,
Zx(s) = Σ no. (als) exp
(1-1 1-7
(32)
aEA
Since with the reverse KL term no longer have the close form softmax update under natural policy gradient algorithm. For simplicity we drop the KL term the theoretical proof here as under assumption A after the pretrain stage the KL term longer dominate the loss function, and mainly focus on the proof on the balance between reward and cost.
We admit that omitting the reverse KI. divergence term simplifies the derivation of a closed-fomm NPG update, yet the core insights from this analysis are expected to remain relevant the behavior of the full DRCORL algorithm. This is mainly because:
23

---

(i) The initial pretraining of the diffusion model (Section 3.1 and its subsequent use as regularizer (Section 3.2) aim ensure the learned policy To primarily operates within a region close to the behavioral policy Assumption A.1 formalizes this by bounding the KI. divergence.
(ii) In this regularized regime, the KI. term primarily acts to constrain exploration and prevent significant deviation from the data distribution. The fundamental trade-off's and convergence dynamics related to balancing expected rewards and costs still governed by the NPG updates on these value estimations. Thus, the analysis, while idealized, sheds light on the core learning dynamics concerning reward maximization under cost constraints.
Assumption A.2. Given offline dataset Du = {185, 0i, 04,8%, (i of size |D4| N, let the value function class be F and define the model class as (TfIS E E. We assume:
Realizability: The critic function Q", learned by optimizing Eq. T6 Eq. 17. and Eq. (8)
belongs the function class F, and TQ" resides the model class G. Moreover, we assume G = F.
Dataset Diversity: The offline dataset is diverse enough to ensure accurate offline policy evaluation. Specifically, we assume that:
C CIn([FF/8) (10((((((51100
=
(33)
Coffiiee -711
where Eoffine is the desired accuracy, and ō is the failure probability.
By invoking Chen and Jiang 2019 Theorem 3], we can show that under Assumption A.2 with probability at least 1 0, the following bounds hold:
Q 07* I| S Collime:
1Q º S Colline
(34)
Then, by Gu et al. 2024b Lemma A.2], we can show that the policy update with gradient manipula-
tion satisfies that:
For all & E Ssafe. the bound on the reward function given by:
V- (p) Te (p) S E
swaffo [DLL
(To,
(+1s) - DKI (oooo((sss))))) (11s)] (+|s)]
(35)
2zy S
3(1 + MM)
|0 Q l12
733
(1-7)²
Similarly, for all Sunsafes the bound on the cost function is:
TT (p) V. (p)
- E snd000 [DKL (TO- (To- (-1s) (-|s)) DKL DKL (-s))= (+|s))
1
(36)
2n S
3(1 + 7MM
1Qth0 Qa
(1 3)3
(1-
For è Saligns have the combined bound for reward and cost
1
To' (p) The (p)) (v (p) V. (p))
2
(DKL(FT- [To (-1s)) DKL [ (-|s)) + 2n/091/1111
< E snd
(37)
(1 7)3
+
3(1 + nM
1
Te, (s, a) 07 (s.a))
+ Q= (s, a) -Q=² (s, a)
21-
2
24

---

Finally for E Sconttict, it holds that
(gt ged Vo (p) V, v²( (
(g 9i)
(V (0)- - Ve. [p))
2
1
2|1gr|?
- doddo (DKL (TO (-1s)]|TO, (-15)) - DKL (TO (-1s))
11
2n (1-
(o-.0") 2110?
|S||Ai
(38)
21 12
3(1 + nM)
[1
2 (s)
(s,a) - Qr" (s, + 1 tht "s, Q (s, [ااا)
+
(1 11-7) 7)2
Summing the four equations, Eq. (35). Eq. (36). Eq. (37 and Eq. 38 we obtain that
Vo (p) Ve (p)) Sa
(V (p)
Ve (p))
S
(gt god
+
(V. (p) Ve (p))
(39)
21|gt|1
E Ba DKL. (TO- (-1s)] no, (-18) + 20|5||A|M22T
+ +0Q (1 7)3
1)
where EQ is the accumulated weighted critic error, defined as:
eQ Σ
3(++nM)
11 - 2.
3(1+
Q T. - -Q
(1 71-
(1 (1-
Saa
SE
3(1 + M)
1
2 11 (s) Q7 (s,a))12 + 1 10 (s, -.a² (s))))))
Σ
(1 7)-
(40)
3(1 + M)
[
(gf- g6)
)110 (s,a) - Q, (s, (s)
(1 71-
2||g 1
(gr: ged
0% (s,a) - Q (s. al) ||,]
+
2
2||g2|1
Now, we need to upper bound the weighted critic error eq. We assume that there exists a positive constant C' such that eq with 3CT(1+gM)
(1-7)² Cottline According to Gu al. [202bb Lemma A.6], by choosing reasonably large values for h+ and h we can ensure that:
SSunattel + Satigal + |Scontiet| 2 T/2.
For example, by setting h+ = 2 Now, we define the weighted policy ग follows
(1-7)²7 (Eaist + 4M + 6M) and h- = 0, this condition holds.
S SAA
ΣT (aTo(als)
#(als)
(41)
Σ
where the policy weights are assigned based on the categories of iterations: 1. Weight wa = 1 for
t E Saate 2. Weight w, = for E Sunsafe-
oo' for t E Semafiit. Under the weighted policy म, the following bound holds
3. Weight w, =
||o!|
true for the reward value function:
1
1
21 27/SS|AA/M2T
VT V <
E B DKL(TO-(-Iss)) (-1s)) +
+ +eQ
1
1
(1 7)3
2
2
(42)
4
27|S||A||M2T
3CT + (1.M)
<
T
Edist +
(1 71-7)3
(1 71)
Eoffline
25

---

Now, by choosing 17 = - /aaull (1))(11)//22/)))11111))) + 3CTTM(( - we can ensure that
32(27/S]|AIIM² +
cotttne
V
V
VT
(1-7)3
(1 7)2
(43)
=0
|S||Ii (1- (1-7) (1->)3T
+ (coMtnn).
Finally, the safety bound is given as:
(p) b
(ge gfd 2|1g2 1?
Σ
(Ve (p) b)++,
-p)
Σ (Ve (p) b)
(Ve (p) b)
T
2
rESu
1
Lg gft 2|!
<
Σ
Ve-e (p) -p) +) Σ
(ee
The (p) b)
+
Ve Tue (p) -b)
T
2
SE
LE &
S-h
Shi
4
(Ve V. (p) - -Vo (p)) + Σ
(V - Ve (p)) +
Σ
(v= (p) - Vo:10 (p))
+
T
tE.
tE
(V: |S||A
<
2h+ 20olline (Colllin)) +0
(1-
(44)
This completes the proof of Theorem 3.1
B Supplemental Materials for Algorithm
1
B.1 Clarification for the Main Body
Inconsistency critic update. In the training process, we specifically adopt conservative esti- mation for the cost critic, the differing update strategies for Q and Q are intentional and reflect their distinct roles in our framework. The cost critic Q plays a critical role enforcing safety, so we adopt pessimistic update to penalize uncertain or unsafe state-action particclllyl in low-coverage consiseen with prior work on conservative crities in offline RI.
In contrast, the reward critic Q guides policy optimization within the safe region. Applying pessimism here may lead to overly conservative behavior, reducing performance. We therefore use a standard update to maintain a balance between safety and reward. Our gradient manipulation mechanism relies on Q modulate the influence of each objective, making pessimistic cost critic essential for reliable safety adaptation. We appreciate the concern about value overestimation and will explore robust reward estimation techniques in future work.
Pretraining stage. We pretrain the diffusion model to accurately capture the behavioral policy from the offline dataset, enabling serve stable regularizer for the Gaussian policy during training. Unlike diffusion-based planning methods, our approach uses the diffusion model's 's score function to penalize -iittrr-iitiittionn actions, improving stability.
After pretraining, the diffusion model remains fixed. However, updating the critics during training is crucial, as they guide the gradient manipulation mechanism to balance reward maximization and safety. This separation ensures effective and stable policy optimization throughout training.
Gradient Manipulation Stage. the gradient manipulation method employed in our approach differs fundamentally from the naive weighted average of the two objectives. Specifically, we dynamically adjust gradient weights based on the extent of safety constraint violations, guided by slack variables. When the safety threshold significantly violated, parameter updates are driven primarily by the gradient of the safety objective. Conversely, when the current policy strictly adheres
26

---

to safety constraints, updates are performed using only the gradient of the performance objective. In intermediate scenarios, we judiciously combine both gradients.
To address the potential instability, our approach carefully assigns weights according to the angle between gradients, as detailed in Equation (12) of the main body. Instability typically arises when the angle between gradients exceeds 90%. Our weighting strategy effectively mitigates gradient conflicts, preventing gradient degradation and enhancing the stability and reliability of the optimization process. Additionally, our method does not require the reward and cost critics to share parameters. Instead, we temporarily freeze and save the policy parameters to implement gradient manipulation. For example, given two separate critics, QF and Q" we first store the initial parameters of the policy We then independently compute updates using each critic. By comparing the updated policy parameters with the initial parameters, we obtain the gradient values required for manipulation. Through this icenario-based approach to parameter updates, we effectively ensure training stability.
B.2 Safety Adaptation Step
We outline the Safe Adaptation step in Algorithm I specifically detailed in Algorithm 2 Our main theorem (Theorem 3.1 is derived based the safety adaptation procedure described in Algorithm 2 Our diffusion regularization method is compatible with both CRPO Xu et al. [2021] and the gradient manipulation method Gu al. [2024a]. implemented in Algorithm Both algorithms aim to
switch dynamically between reward optimization and cost optimization.
The key distinction lies in their approach to handling scenarios where the cost is close to the cost limit threshold. In Algorithm Algorithm incooporraes gradient manipulation in these scenarios, further stabilizing the training process by addressing conflicts between the objectives. Theorem 3.1 specifically considers the diffusion regularization algorithm equipped with the safe adaptation procedure outlined in Algorithm [] The fundamental difference between the two algorithms lies in the criteria for switching between reward and cost optimization objectives.
Algorithm 2 Gradient Manipulation Adaptation
Require: Dataset D
Require: Slack variable h+, h- and cost limit 1 Procedure: SafeAdaptation cs. Q" Q h+ h-)
if Q=. S 1 - h then Optimize reward by solving Eq. (19)
else if Q UCR (p) S + h then
Compute 9- and ge with Eq. (21
Gradient manipulation to obtain with Eq. 12
else
Ensure safety by solving Eq. (00
end if' end procedure
B.3 Hybrid Extension
Hybrid Agents. The assumption of the offline reinforcement learning setting be extended by allowing the agent to incorporate partial online interactions during the training episode. This extension enables further updates to the critic function, enhancing its ability evaluate safety conditions with greater accuracy. Since the problem remains within the scope of offline reinforcement learning, we restrict hybrid access two specific types:
Access to a simulator for evaluating the cost values V
Access limited number of trajectories collected by the current policy. which can be used to update the critics and policies, thereby partially mitigating the impact of distributional shift.
With the hybrid assumption We propose two distinct approaches for evaluating costs: Offline Evaluation and Online Evaluation. For the whole offline setting, we only use the critic functions learned from the offline dataset to evaluate the cost constraints, while for the hybrid agent we allow for online trajectory access.
27

---

Offline Evaluation
For the hybrid agents discussed section we consider two distinet forms of hybrid access to environment data.
In the fully offiine setting, estimate the cost value V (p) by randomly sampling batch of states B, from the static dataset. We assume that the state distribution in the dataset s ~ DD is sufficiently close to the target distribution p. The cost estimator defined as:
(D)
Σ (s). sEB,
B₂
To avoid hyperparameter tuning and additional budget constraints on the value function, we transform the value function into estimated episodic cost. Since the value function V can expressed as:
V (p) T-7 Bondy amo( (r(s, a)]:
define the estimated episodic cost V. (1 L, where L represents the episodic length.
Online Evaluation
Agents are allowed to collect a limited number of trajectories to evaluate the safety of the current policy. Based on this evaluation, the agent determines whether to prioritize optimizing the reward, jointly optimizing reward and cost, or exclusively optimizing the cost. This process serves as performance assessment of the learned policy during each episode.
Agents can roll out predefined number of trajectories to update the critic function in each episode. To ensure the setting remains consistent with offline reinforcement learning, the number of trajectories strictly limited; otherwise, would replicate fully online reinforcement learning setting. By leveraging this partial online data, the agent mitigates overestimation errors in the critic function, thereby improving its ability to evaluate and optimize the policy effectively.
B.4 Further Discussion on Safe Reinforcement Learning
In this section, discuss the difference between the hard constraint and soft constraint in safe reinforcement learning.
Hard Constraints. Existing works focusing hard constraints allow no violation safety conditions, i.e.
c(st, ae) = 0, A
(45)
Soft Constraints, variety works focus on the soft constraint setting, the safe constraint restricts the cost limit below a certain limit 1, either within episode or being step wise. Which can either can be episodic limit as
T
E
Σ ==
c(st, an)
E
Σ r (((s)) 1=
51,
(46)
or be the stepwise limit
Elc(st, < 1, (-13t)).
(47)
Still this type of problem allow for certain degree of safety violations in Xu et al. 2022 Lin et al. 22023,, but the soft constraints also allow for broader potential policy class to further explore higher reward. We choose the soft constraint as it allows for exploration search for higher rewards.
B.5 Discussion on Settings
Comparison with Offline Reinforcement Learning. In the context of safe reinforcement learning. simply pretraining the critic before extracting the policy insufficient for learning an optimal policy. This contrasts with approaches such those in Chen et al. [0024 2023), where pretraining a reward critic Q a) under the behavioral policy my using IQL Kostrikov et al. [2021] is sufficient. These methods do not require further updates the critic during policy extraction.
28

---

In safe reinforcement learning, however, optimization involves the reverse KL divergence term:
(el(s.a)
1
E aa
(48)
3
where pa(((s) represents the behavioral policy used to collect the offiie dataset. The optimal policy Tg for Eq. (48) is given by:
Te- (a|s) s(als) exp(3Qo(s,a))
(49)
Essentially, offline reinforcement learning algorithms aim to extract policy no that adheres to the energy-based form presented in Eq. (49) However, in the safe reinforcement learning setting, the optimal reward critic, when unconstrained, cannot be directly used for optimizing the reward objective. T'errefoee, it must be updated during the safety adaptation stage.
Comparison with Constrained Reinforcement Learning, While most safe reinforcement learning literature focuses on the online setting, the offline setting presents unique challenges for policy extraction. In the offline scenario, the agent's 's only access to the environment is through offline dataset D", which consists of transition tuples. Ideally, these transition tuples can be utilized to construct estimators for the transition dynamics P(s'1 a), the reward function f(s, a), and the cost function ê(s,a).
However, Markov decision processes (MDPs) are highly sensitive to even small changes in the reward function, requiring efficient exploitation of the offline dataset through conservative inference and penalization of out-of-distribution (OOD) actions. To address these challenges, our approach constrains the policy to remain within defined neighborhood of the behavioral policy m and adopts a pessimistic estimation the cost critic function, effectively mitigating the risk of unsafe implementations.
C Details of Experiments
C.1 Further Ablation Study
Temperature Parameter B. We explore two different types of 3 schedules regulate the trade-off between policy exploration and adherence to the behavioral policy.
Constant / Values. In this approach, , is maintained as a constant throughout all epochs. A low B value enforces conservative Q-learning by constraining the learned policy to remain close to the behavioral policy This setting prioritizes stability and minimizes divergence from the offline dataset.
Variant Values. Here, we employ a monotonic sequence increasing 3 values over different epochs. Following the pretraining phase, the weight optimizing the reward critic is progressively increased, or the weight on minimizing the cost critic is reduced. This dynamic adjustment encourages the policy explore diverse strategies, allowing it to optimize returns reduce costs effectively while gradually relaxing the conservativeness enforced during earlier stages of training.
In the initial training phase, set 3 to low value (starting at 0.04) to ensure the policy remains close to the behavioral policy, facilitating a stable foundation and minimizing out-o--distribution actions. As training progresses, we gradually increase 3, eventually reaching 1.0, to place greater emphasis on optimizing the critics. This linear scheduling allows for smooth transition from imitation to optimization, balancing exploration and exploitation effectively. Improper tuning of B can impact performance:
If 3 increases too rapidly: The policy may deviate prematurely from the behavioral policy, leading instability and potential safety violations due insufficient grounding in safe behaviors.
3 increases too slowly: The policy may remain overly conservative, limiting performance improvements and resulting in suboptimal reward outcomes.
Therefore, carefully planned 3 schedule is erucial to balance safety and performance. Our linear approach has demonstrated effectiveness across various tasks.
29

---

Nammllredd Daranerd
520
1052 Staps
1300
2003
80
500
1002
3300
2003
Steps
epoch
Figure 3: Training Curve Under Different Schedules. We compare the training performance of three different 3 schemes, under the square root growth we have the highest normalized reward with high stability.
Choice of Slack Variable. We set slack bounds relative to the normalized cost so that reward maximization applies when Voomalized
h- and cost minimization triggers when Vosmulizedd 2 1 1+ h During training. both h and h+ are linearly decayed from 0.2 to zero. An ablation study on the impact of slack values is shown in Figure 4 We tested different initial values for h+ = h = la 10.1, 0.3, 0.5, 0.73.
Revard
135
Slic--01
17.3
Slock
120
Slacki23
15.2
1.2
3.10
5:2
2.5
903
1000
3930
2000
2001
3000
3500
4070
3385
1940
Texining
2000
2530
3000
3001
4050
Figure 4: Slack Ablation
Choice of cost limit. We evaluated our algorithm on Offl offlineCarRun-vO using three cost limits (1 E {10, 20, 30)) with five random seeds each. Figure presents the normalized reward and cost. The normalized reward remains consistent across different limits, and the learned policy reliably keeps the cost below the safety threshold.
We present the general hyperparameter setting in Table 2 For hyperparameters that do not apply to the corresponding algorithm, use the back slash symbol "(" to fill the blank.
Remark C.1. In Table 2 the update steps refer to the total number of gradient descent updates performed. The evaluation steps indicate the frequency of policy evaluation, measured in terms of gradient descent steps. The actor architecture (MLP) is specified as a list representing the hidden layers, where the input corresponds critic architecture (MLP) is represented as list defining the hidden layers, with the input being the state-action pair and the output being a scalar value. The parameter represents the update rate between the target critic and the critic in double- learning.
state s and the output is an action a = x (()). Similarly, the
30

---

Cest
144
Figure 5: Cost Limit Ablation
Table 2: Summary of hyperparameter configurations for different algorithms.
Hyperparamettess
BC-Safe
BEARL
BCQ-Lag
CPO
COpúDICE
CDT
CAPS
CCAC
Ours
Device
Cuda
Coda
Cuda
Coda
Cuda
Cuda
Cuda
Cuda
Cuda
Batch Size
512
512
512
512
512
2018
512
512
256
Update Steps
100000
300000
100000
100000
100000
100000
100000
100000
2050
Eval Steps
2500
2500
2500
2500
2500
2500
2500
2500
1025
Threads
6
Num werkers
8
8
8
8
Actor Architectuue(MI..P
[256,256]
[256,256]
[256,256] 256]
[256,256]
[256,256]
[256,256]
[256.256]
[256,256]
Critic Architecture(MM.P))
[256,256]
256]
[256,256]
[256,256]
[256,256]
[256,256]
[256,256]
Actor Learning rate
001
001
001
001
.011
001
.001
.001
0006
Critic Learning rate
001
.001
.001
.001
.001
0006
Rpisode Length
1000
1000
1000
1000
1000
1000
1000
1000
1000
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
.005
.005
000
.00
.005
.005
.005
PID
[.L.003.000]
[.L.033.01]]
1
[11,003,001]
1
2.0
C.2 Choice of Policy Class
Standard Gaussian Policy Class: IT = {a ~ (mo(s), Eo(s))}, usually the covariance Ex(s) matrix is diagonal matrix.
Gaussian Policy Class with constant variance: 11 {a ([ o21)}, here the covariance matrix o²1 is state-independent.
Dirac Policy family: I1 {a mo(s)), we can approximate this as Gaussian policy with variance close to O.
C.3 Training Details
As illustrated in Figure experiments conducted in Safety-Gym environments and Bullet- Gymnasium environments. In this section, we present the training curves for each task. For every task, we run five independent seeds and plot, at each training step, the mean and standard deviation of both reward and cost.
31

---

AntVll
SafetyHaliChce-Gee SafetyHopperVe Vel (d)
SafetySwim- (e) Safety WalkerVel
Vel
merVel
Figure 6: Safety-Gym Environments
2.
Figure 7: HalfCheetah Reward
Figure 8: HalfCheetah Cost
2000
Figure Ant Reward
Figure 10: Ant Cost
Figure 11: Walker2D Reward
Figure 12: Walker2D Cost
32

---

65
65-
Figure 13: Balliie Reward
Figure 14: OfflineallCircee Cost
Figure 15: OffrineCrrCirlee Reward
Figure 16: OfflineCarCircle Cost
Figure 17: OfflinCCaRRun Reward
Figure 18: OfflineCarRun Cost
Wamren
250
Figure 19: OfflnnCarGoall Reward
Figure 20: OfflineCarGoall Cost
33

---

Figure 21: OffineCarGool2 Reward
Figure 22: OfflinCarGooa22 Cost
04
Figure 23: OfflinePointGoall Reward
Figure 24: OfflinPPointioall Cost
23
12
Why
00
-00
Figure 25: OfflinePointGoa22 Reward
Figure 26: OfflinePointGoa22 Cost
60
Figure 27: Poittreeoi Reward
Figure 28: OffiinePointircceel Cost
34

---

18
05
02
Figure 29: OfflinePointButtonl Reward
Figure 30: OfflinePointButtonn Cost
35


Reviews:
[{"review_id": "2PLAWaCcSm", "reviewer": "Reviewer_QXQe", "summary": "This paper introduces Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), a framework designed to learn safe and high-performance policies from offline datasets. The approach leverages a diffusion model to imitate the behavioral policy and uses reverse KL divergence to regularize a simplified Gaussian policy. In addition, it employs a gradient manipulation technique that adaptively interpolates between competing objectives. The method is evaluated on the DSRL benchmark, outperforming several existing offline safe RL methods in more than half of the tasks.", "questions": "- Could you comment more on DRCORL against non-diffusion-based offline RL approaches that are safety-aware but use simpler regularization (e.g., model ensembles or cost-conditioned Q-learning)? I would like to learn more about justifying diffusion modeling as necessary?\n\n- How does the diffusion model behave in cases where the offline dataset is sparse, multi-modal, or contains unsafe behaviors? Can the method robust enough to handle these situations which are more realistic in the real world, highlighted by some papers like Learning from Sparse Offline Datasets via Conservative Density Estimation. I have some concerns that the proposed method may be unstable when working on sparse data set.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- DRCORL employs a diffusion model to learn the complex behavioral policy from offline data, then extracts and regularizes a separate, simplified Gaussian policy using the diffusion model's score function. This design enables fast inference speeds. \n\n- The gradient manipulation technique offers an adaptive solution to the reward-safety trade-off, making the method more generalizable across different cost constraint conditions.\n\n- Extensive evaluations on 12 benchmark tasks from SafetyGym, BulletSafetyGym, and MuJoCo demonstrate that DRCORL achieves better performance while maintaining safety constraints, providing competitive scores against BCQ-Lag, CPQ, and COptiDICE.\n\nWeaknesses:\n\n- Not enough discussion or comparison of non-diffusion-based offline safe RL methods, such as ensemble-based uncertainty estimation or constrained supervised learning approaches, which may offer competitive performance without relying on generative models. Including these could better contextualize the advantages of DRCORL. \n\n- While DRCORL avoids inference-time sampling from the diffusion model, pretraining remains computationally expensive, especially for large-scale problems. In high-dimensional state-action spaces (e.g., manipulation with vision inputs or multi-agent systems), training diffusion models and critics can become very resource-intensive, limiting scalability and practical deployment. In addition, it may require a high-quality offline dataset and relies on good coverage of the behavioral policy to generate the diffusion models. Its performance may degrade under poor dataset conditions.", "was_revised": true, "final_justification": "Thank the authors for detailed replies. It addresses my concerns. After reviewing the replies of the authors to me and other reviewers, I would like to keep my score.", "rebuttal": "Thank you for engaging in the discussion and providing further clarification. We sincerely appreciate the reviewer’s thoughtful and constructive suggestions, which have helped improve the clarity and generalization perspective of our work. Your feedback has been constructive in strengthening our argument for DRCORL and its broader applicability.\n\n> **Question:** Could you comment more on DRCORL against non-diffusion-based offline RL approaches that are safety-aware but use simpler regularization (e.g., model ensembles or cost-conditioned Q-learning)? I would like to learn more about justifying diffusion modeling as necessary?\n\n**Response:** Thank you for raising this insightful question. In fact, several baselines included in our experiments are safety-aware and employ simpler regularization techniques. For example, CAPS [1] uses an $\\ell^2$ penalty (see Eq.(6)) to constrain the learned policy to remain close to the behavior policy, while BEAR [2] employs bootstrapping and MMD-based penalties to avoid out-of-distribution (OOD) actions. These approaches, like ours, share the core objective of regularizing the learned policy within a trust region around the behavior policy, which is a widely adopted strategy in offline RL.\n\nWhat differentiates our approach is the use of diffusion models to represent this regularization constraint. Diffusion models possess strong approximation capabilities, capable of modeling complex and multi-modal behavior distributions in a nonparametric and data-driven manner. In contrast to manually designed distance-based penalties, the diffusion-based approach learns the behavioral distribution directly and provides a more flexible and expressive way to regularize the learned policy. This allows DRCORL to scale more naturally to settings where behavior distributions are non-trivial.\n\n> **Question:** How does the diffusion model behave in cases where the offline dataset is sparse, multi-modal, or contains unsafe behaviors? Can the method be robust enough to handle these situations which are more realistic in the real world, highlighted by some papers like Learning from Sparse Offline Datasets via Conservative Density Estimation. I have some concerns that the proposed method may be unstable when working on sparse data set.\n\n**Response:** This is an important and valid concern. At present, we have evaluated DRCORL on a broad set of tasks from the DSRL benchmark, which primarily features uni-modal behavior distributions. We acknowledge that more realistic scenarios, such as multi-modal or sparse datasets with heterogeneous and potentially unsafe behavior can pose significant challenges. Evaluating DRCORL under such conditions is an active direction for future work. In particular, we are interested in testing vision-based or partially observed settings, where the data may naturally exhibit multi-modality.\n\nRegarding robustness to unsafe behaviors, the DSRL benchmark does include both safe and unsafe trajectories. Our experimental results show that DRCORL is able to learn safe policies despite the presence of unsafe samples. For example, in the BC-Safe baseline, imitation learning is conducted solely on safe trajectories, yet the resulting policies are often less safe than those learned by DRCORL. This empirical finding supports the robustness of our approach in handling mixed-quality data.\n\nThat said, we do recognize that the performance of our method is inherently tied to the quality and size of the offline dataset. Theoretical results (see [3]) assume that the diffusion model can successfully approximate the behavioral distribution given sufficient data. However, in highly sparse or low-quality datasets, such as those addressed in the conservative density estimation literature, this assumption may no longer hold, and more specialized solutions may be needed. We appreciate the reviewer’s pointer to this line of work and see it as a promising extension direction for improving the robustness of DRCORL in real-world deployments.\n\n**References:**\n\n[1] Chemingui, Yassine, et al. \"Constraint-adaptive policy switching for offline safe reinforcement learning.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 39. No. 15. 2025.\n\n[2] Kumar, Aviral, et al. \"Stabilizing off-policy Q-learning via bootstrapping error reduction.\" *Advances in Neural Information Processing Systems* 32 (2019).\n\n[3] De Bortoli, Valentin. \"Convergence of denoising diffusion models under the manifold hypothesis.\" *arXiv preprint* arXiv:2208.05314 (2022)."}, {"review_id": "5duFKSMXvw", "reviewer": "Reviewer_5D62", "summary": "The paper proposed Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. The diffusion model is used to compute the score function of offline data and used to compute policy gradient with reverse KL divergence regularization. Then the gadient maniputation tricks in Gu et, al. is used to update the constraint-satisfying policies.", "questions": "1. Why not use other score matching techniques mentioned in weaknesses 2 to learn the score function? I don't see any advantages in using the denoising score matching as the noise perturbation design is mainly beneficial to the generation process of diffusion models. Especially, the denoising score matching learns the score functions of the noise-perturbed data rather than the true score. \n\n2. How to compute the $t\\to 0$ in line 237?\n\n3. The pretrained model can only guaratee learning the score function of the expert policy accurately on the dataset distribution, but not on the current behavior policy distribution. How to handle these distributional shift when you query the score function of current policy using score network trained from the offline dataset?\n\n4. I might not be fully aware of the thoeritical contributions as no siginifcant parameters show up in the theoritical results. How is the theoritical results compared to other baseline algorithms, or none of the baseline algorithms can have such theoritical guarantees?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\n1. The writing is clear.\n2. comprehensive evaluations\n\n## Weakness\n1. The reverse KL regularization is not novel and has been studied for RL algorithms such as [1].\n2. The authors only learns the score function, which has connections to but is not fully equivalent to the claimed diffusion models, that is mainly for generation. There are a lot of alternative approaches to learn such score functions, such as [2, 3], so the diffusion model is not a significant component here. The title might be misleading.\n3. The theoretical results are a bit general. \n4. The main algorithm directly follows an online RL algorithm in Gu et, al. Adding regularization in offline RL is a standard technique when converting online RL to offline.\n\n[1] Chan, Alan, et al. \"Greedification operators for policy optimization: Investigating forward and reverse kl divergences.\" Journal of Machine Learning Research 23.253 (2022): 1-79.\n[2] Hyvärinen, Aapo, and Peter Dayan. \"Estimation of non-normalized statistical models by score matching.\" Journal of Machine Learning Research 6.4 (2005).\n[3] Song, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in artificial intelligence. PMLR, 2020.", "was_revised": true, "final_justification": "I have read the comments from other reviewers and AC. I will raise my initial score since the author's rebuttal resovled by major concern.", "rebuttal": "Thank you very much for your time and thoughtful comments. We sincerely appreciate your constructive feedback and provide the following clarifications to address your concerns.\n\n\n> **Question:** Why not use other score matching techniques mentioned in weaknesses 2 to learn the score function? I don't see any advantages in using the denoising score matching as the noise perturbation design is mainly beneficial to the generation process of diffusion models. Especially, the denoising score matching learns the score functions of the noise-perturbed data rather than the true score.\n\n**Response:** We thank the reviewer for raising this important point, we think the reviewer is asking about why do we need to use the noise perturbation to learn the score function, as mentioned in weaknesses2 we do not have to utilize diffusion model to learn the score function. \n\nIt is important to emphasize that the core role of the diffusion model in our framework is to approximate the behavioral policy, and we use this approximation to regularize the learned policy. The score function derived from the diffusion model is a means to this end, as it serves as a tractable surrogate for imposing a constraint that keeps the learned policy not far away from the behavioral one. We would like to clarify that our use of the score function primarily serves a practical purpose: it allows us to compute the gradient of the reverse KL divergence in Eq.(8) without needing to sample from the diffusion model, which is often computationally expensive. This significantly improves efficiency during policy optimization. \n\nWe can also compute the gradient of the reverse-KL by utilizing the diffusion model to sample multiple actions, then take the gradient of the estimator as the unbiased estimated gradient. The use of the score function simply serves to  improve the computation efficiency in the training process without having to use the diffusion model for sampling.\n\n\n\nIn summary, while we utilize the score function for computational reasons, the underlying regularization mechanism still relies on the diffusion model’s ability to capture the structure of the behavioral policy. We appreciate the reviewer’s suggestion regarding alternative score matching methods and will consider exploring more accurate or efficient options in future work.\n\n\n> **Question:** How to compute the $t\\to 0$ in line 237?\n\n**Response:** In our practical implementation, we discretize the$\\frac{1}{\\sqrt{\\bar{\\beta_t}}} \\epsilon_{\\psi}(a_t,t\\vert s)$ in Line 237 as an approximation.\n\n---\n\n> **Question:** The pretrained model can only guarantee learning the score function of the expert policy accurately on the dataset distribution, but not on the current behavior policy distribution. How to handle these distributional shift when you query the score function of current policy using score network trained from the offline dataset?\n\n**Response:** We acknowledge that the score network is trained on the empirical distribution of the offline dataset, which may differ from the current policy’s distribution. However, as stated in Theorem 3.1, we assume the dataset is sufficiently large such that the empirical distribution is close to the behavioral policy. This is reflected in the small $\\epsilon_{\\text{offline}}$, which appears explicitly in our regret bounds in Eq.(14) and Eq.(15), quantifying the distributional mismatch. Therefore, our theoretical analysis directly accounts for this issue and bounds its impact.\n\n> **Question:** I might not be fully aware of the theoretical contributions as no significant parameters show up in the theoretical results. How is the theoretical results compared to other baseline algorithms, or none of the baseline algorithms can have such theoretical guarantees?\n\n**Response:** We appreciate the reviewer’s perspective on the theoretical contributions. While the results may not involve complex parameter dependencies, we highlight several key contributions:\n\n1. Regret Bounds for Both Reward and Cost: Theorem 3.1 provides bounds on both the expected reward and constraint violation (cost) of the learned policy compared to the optimal one, which is relatively rare in the safe offline RL literature.\n2. Cost Upper Bound via Diffusion Regularization: Proposition 3.1 demonstrates that pretraining a diffusion model and regularizing the learned policy to stay near the behavioral policy helps upper bound the cost. This highlights the importance of diffusion-based regularization in safe RL.\n\nCompared to existing works:\n- Methods such as TREBI [1], OASIS [2], and CVPO [3] do provide regret bounds, but they are often based on strong assumptions, and some conclusions are not particularly significant or tight in practice.\n- FISOR [4] and CDT [5] only offer a closed-form representation of the optimal solution without any accompanying theoretical guarantees for the performance of the learned policy.\n- CRPO [6] and CPO [7] do include theoretical guarantees, but they are established in an online Safe RL setting, making them inapplicable to the offline setting we study.\n\nIn summary, our theoretical contributions are tailored to the offline safe RL context, and we believe they fill a meaningful gap by providing both regret and constraint violation guarantees without relying on strong assumptions from the online literature. We thank the reviewer for highlighting this point and will aim to further expand the theoretical scope in future versions.\n\n\n**References:**\n\n[1] Lin, Qian, et al. \"Safe offline reinforcement learning with real-time budget constraints.\" *International Conference on Machine Learning*. PMLR, 2023.\n\n[2] Yao, Yihang, et al. \"Oasis: Conditional distribution shaping for offline safe reinforcement learning.\" *Advances in Neural Information Processing Systems* 37 (2024): 78451-78478.\n\n[3] Liu, Zuxin, et al. \"Constrained variational policy optimization for safe reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2022.\n\n[4] Zheng, Yinan, et al. \"Safe offline reinforcement learning with feasibility-guided diffusion model.\" *arXiv preprint arXiv:2401.10700* (2024).\n\n[5] Liu, Zuxin, et al. \"Constrained decision transformer for offline safe reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2023.\n\n[6] Xu, Tengyu, Yingbin Liang, and Guanghui Lan. \"Crpo: A new approach for safe reinforcement learning with convergence guarantee.\" *International Conference on Machine Learning*. PMLR, 2021.\n\n[7] Achiam, Joshua, et al. \"Constrained policy optimization.\" *International Conference on Machine Learning*. PMLR, 2017."}, {"review_id": "KtXs12OOEq", "reviewer": "Reviewer_Fxxd", "summary": "This paper proposes a diffusion-regularized method for constrained offline RL. The key idea is to use a diffusion model to approximate the behavioral policy from offline data, then distill it into a lightweight Gaussian policy that enables fast inference. A gradient manipulation mechanism is further introduced to trade off between maximizing reward and minimizing constraint violations. The authors evaluate the method across DSRL benchmarks and claim superior performance in both reward and safety metrics compared to existing safe offline RL baselines.", "questions": "* Why use diffusion policies at all? Wouldn’t other generative models work just as well or better?\n* Is there any study of what happens when the diffusion model fails to fully capture the behavioral policy? How does this affect safety guarantees?\n* The method applies reverse KL regularization. Did the authors try forward KL or a symmetric variant?\n* The theoretical analysis drops the KL regularization—how valid are the bounds in practical settings where the reverse KL is strong?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Quality\nThe pipeline is solid, and the empirical results are comprehensive. The gradient manipulation approach is a reasonable adaptation of ideas. The paper also provided convergence analysis. However, I didn't check the correctness of the theories.\n\nHowever, I’m still unclear what the diffusion model is doing beyond replacing a behavior cloning model. Given that the final policy is a simple Gaussian actor and inference doesn’t sample from the diffusion trajectory, what exactly does the diffusion model help with—other than regularizing the Gaussian policy via score matching? The paper assumes the diffusion model perfectly recovers the behavior policy, but in practice this is far from guaranteed, especially in high-dimensional action spaces. That assumption weakens the theoretical safety bound.\n\nAlso, learning from the diffusion model simulated behavioral policy raises concerns about compounding error. If the score function is misestimated, wouldn't the final policy be regularized toward the wrong directions?  \n\n## Clarity\nThe paper is mostly well written, though the presentation gets dense in the methodology section. But the paper could benefit from a cleaner separation between components: what is learned in pretraining, what is used during policy optimization, and what is actually deployed at test time.\n\n## Novelty\nThe use of diffusion models as offline regularizers is not new. Prior work already explore this line. The key novelty here is combining the regularization with a soft gradient-switching mechanism adapted to safety-constrained offline RL. \n\nAlso, why using diffusion policy at all? If the goal is to regularize the final policy toward the offline data distribution, many simpler generative models (e.g., VAE, normalizing flow) could be used with faster training and easier analysis. The authors argue diffusion models have stronger generative capacity—but that strength is underutilized here since they only extract scores, not full samples. Some justification is needed.\n\n## Significance\nThe method works. Empirically it delivers strong reward under safety constraints, and the lightweight policy at inference is indeed faster than diffusion-based baselines like TREBI or FISOR. So from an engineering point of view, this is a practical and deployable method.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and thoughtful comments, we totally recognize the reviewer's contributions for the comments, and we would like to give credits to the reviewers. We will reflect on the reviewer's comments and improve our work based on the suggestions in the future. Here we would also like to provide the following clarifications to address the concerns you raise.\n\n> **Question:** Why use diffusion policies at all? Wouldn’t other generative models work just as well or better?\n\n**Response:** We choose diffusion policies for two key reasons. First, diffusion models are highly expressive and data-efficient, as they can approximate complex, multi-modal behavior distributions using relatively limited training data. This is particularly important in offline RL settings, where data is fixed and may be limited or unevenly distributed.\n\nSecond, diffusion models offer a computational advantage via score-based learning. Specifically, we leverage the score function to estimate the gradient of the reverse KL regularization term without needing to sample trajectories from the generative process. This avoids the high computational cost typically associated with sampling in diffusion models.\n\nWhile other generative tools like VAEs are also theoretically applicable, they require sampling multiple actions and computing expectations over the log-likelihood gradients, which can be less efficient and more prone to high variance. Moreover, in our experiments, diffusion-based regularization consistently yielded better empirical performance, particularly in terms of safety and stability compared to variational approaches we have tried in preliminary ablation studies.\n\nNevertheless, we do acknowledge that other generative approaches, including VAEs or flow-based models, may offer alternative trade-offs in modeling and optimization. For example, [2] applies constrained VAEs in safe RL, which we consider a promising future direction for extension and comparison.\n\n\n> **Question:** Is there any study of what happens when the diffusion model fails to fully capture the behavioral policy? How does this affect safety guarantees?\n\n**Response:** We acknowledge that when the offline dataset is sparse or highly heterogeneous, diffusion models may fail to accurately approximate the behavioral policy. While existing works such as TREBI [3] and OASIS [4] implicitly assume that diffusion models are sufficiently expressive to model the dataset distribution, to our knowledge, few studies have systematically analyzed what happens when this assumption breaks.\n\nIn our framework, if the diffusion policy fails to match the behavioral policy, the main impact is on the initial policy distribution and the regularization signal during training. However, DRCORL is designed to be robust to this mismatch: our safety adaptation procedure (outlined in the main algorithm) fine-tunes the learned policy while respecting safety constraints. Thus, even under imperfect modeling, the algorithm can still converge to a safe policy, though we may also encounter potential degraded performance.\n\n> **Question:** The method applies reverse KL regularization. Did the authors try forward KL or a symmetric variant?\n\n**Response:** We appreciate the reviewer’s question. We chose reverse KL regularization because of its mode-seeking nature, which is well-suited for constraining the learned policy near the behavioral policy,  especially when the dataset contains multiple modes. Reverse KL tends to avoid assigning probability mass to unsupported regions, thereby reducing the risk of unsafe generalization.\n\nThis design choice is also motivated by prior work, including [1], which analyzes the use of forward vs. reverse KL in policy optimization. We agree that symmetric divergences or other variants (e.g., Jensen-Shannon or $\\alpha$-divergence) are worth exploring and plan to investigate them in future work.\n\n> **Question:** The theoretical analysis drops the KL regularization—how valid are the bounds in practical settings where the reverse KL is strong?\n\n**Response:** This is an insightful question. We omit the KL regularization in the theoretical analysis to simplify the derivation and make the regret bounds more interpretable. However, in practice, the inclusion of the reverse KL term plays a critical role in stabilizing training and encouraging conservative policy updates. \n\nOur assumption is that the optimal policy lies in the neighborhood of the behavioral policy. Under this assumption, the solutions to the optimization problems, with or without KL regularization, will be close. Hence, while the bounds do not formally incorporate the KL term, we believe they still meaningfully capture the learning behavior of the algorithm in realistic settings where the reverse KL penalty is used.\n\nAbove all, we hope that we have addressed your questions. We greatly value your insights and would be happy to further discuss or clarify any remaining concerns in future exchanges.\n\n**References:**\n\n[1] Chan, Alan, et al. \"Greedification operators for policy optimization: Investigating forward and reverse kl divergences.\" Journal of Machine Learning Research 23.253 (2022): 1-79.\n[2] Liu, Zuxin, et al. \"Constrained variational policy optimization for safe reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2022.  \n[3] Lin, Qian, et al. \"Safe offline reinforcement learning with real-time budget constraints.\" *International Conference on Machine Learning*. PMLR, 2023.  \n[4] Yao, Yihang, et al. \"Oasis: Conditional distribution shaping for offline safe reinforcement learning.\" *Advances in Neural Information Processing Systems* 37 (2024): 78451-78478."}, {"review_id": "sbqYrmYUJ5", "reviewer": "Reviewer_uHfr", "summary": "The authors propose a two-step approach for constrained offline RL, where on the first step a diffusion behavioral cloning policy is learned, which is distilled into a safe offline policy on the second step of learning. This allows to use the expressivity of diffusion to capture the behavioral policy and a faster inference of standard policies. On the second step of training the authors use a gradient manipulation approach to handle simultaneous optimization over rewards and costs. The algorithm is then validate on a standard benchmark showcasing superior performance to other baselines", "questions": "1. Is pre-training necessary? Is it possible to have a one stage training with diffusion regularization? \n2. Proposition 3.1 and Theorem 3.1 seem to be very conservative. Personally, I wouldn’t prioritize adding these results to the main paper, but up to the authors. \n2. What is BC-safe in experiments? Is it the diffusion pretrained policy? If not I recommend including it the results table\n3. Why not include TREBI Lin et al. [2023] and FISOR Zheng et al. [2024] in the results table?", "limitations": "Limitations are properly discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. An interesting combination of ideas for a fast inference offline policy, for example, the modify diffusion regularisation was proposed by Chen et al. [2023], safe adaptation was proposed by Gu et al. [2024a] and the authors adapt it to the online setting, use IQL for pre-training\n2. Good experimental section showing the superior performance of their method.\n3. Taking care of inference costs of diffusions and proposing a method to overcome it.\n### Weaknesses\n1. Ablation study is rather short, I would expand it and sacrifice theoretical results\n2. The authors make design choices for using diffusion vs a different BC regularisation, for diffusion optimization itself, safety adaptation vs standard lagrangian, using UCB vs conservative Q learning. Ideally, this should be properly discussed and ablated. Why did the authors make these choices is not entirely clear. But it is possible that i missed something. \n3. The flow of the paper can be improved. I appreciate the idea of first going through theoretical considerations and then presenting practical algorithms, but I feel in this case it forces the reader to go back and forth between different sections. It’s not a major weakness, but a better flow would definitely sell the method better\n\nMy assessment is not final and I hope that the authors' rebuttal can alleviate some of my concerns.", "was_revised": true, "final_justification": "The authors alleviated some of my concerns and I trust the authors will incorporate the changes we discussed into the camera-ready version.", "rebuttal": "Thank you very much for your time and thoughtful comments. We sincerely appreciate your constructive feedback and provide the following clarifications to address your concerns. \n> **Is pre-training necessary? Is it possible to have a one stage training with diffusion regularization?**\n\n**Response:** We include a pre-training stage primarily to initialize the policy close to the behavioral policy, which serves as a good starting point for the subsequent regularized optimization. Notably, this pre-training phase is computationally inexpensive—accounting for only 50 out of 500 total training epochs—and is intended to stabilize early learning rather than determine final performance. While it is indeed possible to adopt a one-stage training procedure with diffusion regularization from scratch, we observe in our experiments that doing so yields only marginal differences in final performance. Thus, we choose pre-training for practical stability and simplicity without introducing unnecessary complexity to the optimization.\n\n> **Proposition 3.1 and Theorem 3.1 seem to be very conservative. Personally, I wouldn’t prioritize adding these results to the main paper, but up to the authors.**\n\n**Response:** We appreciate the reviewer’s perspective. While the theoretical results may appear conservative in form, we believe they serve two important purposes. Proposition 3.1 demonstrates that if the behavioral policy is of sufficiently high quality, then keeping the learned policy close to it (via diffusion regularization) guarantees an upper bound on the incurred cost. This motivates the design of our regularization framework and provides theoretical justification for diffusion modeling in safety-critical settings. Theorem 3.1, on the other hand, formalizes our learning guarantee: it shows that after $T$ rounds of training, the algorithm can achieve a near-optimal reward while maintaining approximate safety. Although these bounds are not tight, we believe they offer valuable insight into the structure and intuition of our approach. We will consider relocating them to the appendix if space constraints arise.\n\n> **What is BC-safe in experiments? Is it the diffusion pretrained policy? If not I recommend including it the results table.**\n\n**Response:** Thank you for the question. BC-Safe refers to a policy obtained via behavior cloning using only the safe trajectories from the offline dataset. In contrast, our diffusion-pretrained policy is trained on the entire dataset, which includes both safe and unsafe trajectories—resulting in higher cost despite achieving strong reward performance. BC-Safe is included in the results table to provide a lower-bound reference for safety-oriented imitation learning. To further clarify, we now also include the results of the pretrained policy below for comparison:\n\n| Reward (Cost) | Pretrained |\n|---------------|------------|\n| AntVel        | 0.99 (4.36) |\n| HalfCheetahVel| 0.98 (19.04) |\n| HopperVel     | 0.76 (12.14) | \n| Walker2DVel   | 0.78 (5.38) | \n| CarCircle     | 0.65 (11.16) | \n| BallCircle    | 0.72 (9.34) |\n| CarRun        | 0.96 (1.88) |\n\nThis table highlights that pretraining alone does not yield a fully safe policy, thereby justifying the need for our full DRCORL training procedure with safety-aware fine-tuning.\n\n> **Why not include TREBI Lin et al. [2023] and FISOR Zheng et al. [2024] in the results table?**\n\n**Response:** We thank the reviewer for this suggestion. Regarding FISOR [Zheng et al., 2024], it is designed for a hard-constraint setting where constraint violations are not allowed at any timestep, whereas our setup focuses on soft constraints where only cumulative cost must remain below a threshold. Due to this fundamental difference in problem formulation, a direct empirical comparison is less meaningful.\n\nAs for TREBI [Lin et al., 2023], we agree it is a relevant baseline as it uses diffusion models for safe planning. However, since we already include CDT—which uses transformers and trajectory-level modeling for safety adaptation—we initially opted not to include TREBI to avoid redundancy among generative model baselines. Nonetheless, we recognize its importance and will include TREBI in future versions of our experimental results to provide a more complete comparison. We appreciate the reviewer’s recommendation and will revise the experiment section accordingly.\n\nAbove all, we hope that we can address your questions and we would be happy to further discuss or clarify any remaining concerns in future exchanges."}]

Author rebuttal:

Meta-review: This paper proposes a safety-based offline RL approach that consists of a diffusion policy pretraining stage and constraint-based adaptation stage.

The reviewers held mostly positive opinions on the paper, but there were a variety of remaining concerns about the novelty, as it seemed to be a combination of existing ideas. As stated by Reviewer Fxxd, "The key novelty here is combining the regularization with a soft gradient-switching mechanism adapted to safety-constrained offline RL."

Overall I recommend acceptance as a poster, although I think a case could be made that the contributions aren't significant enough for acceptance.
Decision: Accept (poster)