Title: Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline Reinforcement Learning

Safe Reinforcement Learning

Diffusion Models
Abstract: Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints. We focus on an offline setting where the agent learns from a fixed dataset—a common requirement in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high-quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in real-world scenarios. We open-source our implementation at https://github.com/JamesJunyuGuo/DRCORL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a two-step approach for constrained offline RL, where on the first step a diffusion behavioral cloning policy is learned, which is distilled into a safe offline policy on the second step of learning. This allows to use the expressivity of diffusion to capture the behavioral policy and a faster inference of standard policies. On the second step of training the authors use a gradient manipulation approach to handle simultaneous optimization over rewards and costs. The algorithm is then validate on a standard benchmark showcasing superior performance to other baselines\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. An interesting combination of ideas for a fast inference offline policy, for example, the modify diffusion regularisation was proposed by Chen et al. [2023], safe adaptation was proposed by Gu et al. [2024a] and the authors adapt it to the online setting, use IQL for pre-training\n2. Good experimental section showing the superior performance of their method.\n3. Taking care of inference costs of diffusions and proposing a method to overcome it.\n### Weaknesses\n1. Ablation study is rather short, I would expand it and sacrifice theoretical results\n2. The authors make design choices for using diffusion vs a different BC regularisation, for diffusion optimization itself, safety adaptation vs standard lagrangian, using UCB vs conservative Q learning. Ideally, this should be properly discussed and ablated. Why did the authors make these choices is not entirely clear. But it is possible that i missed something. \n3. The flow of the paper can be improved. I appreciate the idea of first going through theoretical considerations and then presenting practical algorithms, but I feel in this case it forces the reader to go back and forth between different sections. It’s not a major weakness, but a better flow would definitely sell the method better\n\nMy assessment is not final and I hope that the authors' rebuttal can alleviate some of my concerns.\n\nQUESTIONS:\n1. Is pre-training necessary? Is it possible to have a one stage training with diffusion regularization? \n2. Proposition 3.1 and Theorem 3.1 seem to be very conservative. Personally, I wouldn’t prioritize adding these results to the main paper, but up to the authors. \n2. What is BC-safe in experiments? Is it the diffusion pretrained policy? If not I recommend including it the results table\n3. Why not include TREBI Lin et al. [2023] and FISOR Zheng et al. [2024] in the results table?\n\nLIMITATIONS:\nLimitations are properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion-regularized method for constrained offline RL. The key idea is to use a diffusion model to approximate the behavioral policy from offline data, then distill it into a lightweight Gaussian policy that enables fast inference. A gradient manipulation mechanism is further introduced to trade off between maximizing reward and minimizing constraint violations. The authors evaluate the method across DSRL benchmarks and claim superior performance in both reward and safety metrics compared to existing safe offline RL baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\nThe pipeline is solid, and the empirical results are comprehensive. The gradient manipulation approach is a reasonable adaptation of ideas. The paper also provided convergence analysis. However, I didn't check the correctness of the theories.\n\nHowever, I’m still unclear what the diffusion model is doing beyond replacing a behavior cloning model. Given that the final policy is a simple Gaussian actor and inference doesn’t sample from the diffusion trajectory, what exactly does the diffusion model help with—other than regularizing the Gaussian policy via score matching? The paper assumes the diffusion model perfectly recovers the behavior policy, but in practice this is far from guaranteed, especially in high-dimensional action spaces. That assumption weakens the theoretical safety bound.\n\nAlso, learning from the diffusion model simulated behavioral policy raises concerns about compounding error. If the score function is misestimated, wouldn't the final policy be regularized toward the wrong directions?  \n\n## Clarity\nThe paper is mostly well written, though the presentation gets dense in the methodology section. But the paper could benefit from a cleaner separation between components: what is learned in pretraining, what is used during policy optimization, and what is actually deployed at test time.\n\n## Novelty\nThe use of diffusion models as offline regularizers is not new. Prior work already explore this line. The key novelty here is combining the regularization with a soft gradient-switching mechanism adapted to safety-constrained offline RL. \n\nAlso, why using diffusion policy at all? If the goal is to regularize the final policy toward the offline data distribution, many simpler generative models (e.g., VAE, normalizing flow) could be used with faster training and easier analysis. The authors argue diffusion models have stronger generative capacity—but that strength is underutilized here since they only extract scores, not full samples. Some justification is needed.\n\n## Significance\nThe method works. Empirically it delivers strong reward under safety constraints, and the lightweight policy at inference is indeed faster than diffusion-based baselines like TREBI or FISOR. So from an engineering point of view, this is a practical and deployable method.\n\nQUESTIONS:\n* Why use diffusion policies at all? Wouldn’t other generative models work just as well or better?\n* Is there any study of what happens when the diffusion model fails to fully capture the behavioral policy? How does this affect safety guarantees?\n* The method applies reverse KL regularization. Did the authors try forward KL or a symmetric variant?\n* The theoretical analysis drops the KL regularization—how valid are the bounds in practical settings where the reverse KL is strong?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. The diffusion model is used to compute the score function of offline data and used to compute policy gradient with reverse KL divergence regularization. Then the gadient maniputation tricks in Gu et, al. is used to update the constraint-satisfying policies.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n1. The writing is clear.\n2. comprehensive evaluations\n\n## Weakness\n1. The reverse KL regularization is not novel and has been studied for RL algorithms such as [1].\n2. The authors only learns the score function, which has connections to but is not fully equivalent to the claimed diffusion models, that is mainly for generation. There are a lot of alternative approaches to learn such score functions, such as [2, 3], so the diffusion model is not a significant component here. The title might be misleading.\n3. The theoretical results are a bit general. \n4. The main algorithm directly follows an online RL algorithm in Gu et, al. Adding regularization in offline RL is a standard technique when converting online RL to offline.\n\n[1] Chan, Alan, et al. \"Greedification operators for policy optimization: Investigating forward and reverse kl divergences.\" Journal of Machine Learning Research 23.253 (2022): 1-79.\n[2] Hyvärinen, Aapo, and Peter Dayan. \"Estimation of non-normalized statistical models by score matching.\" Journal of Machine Learning Research 6.4 (2005).\n[3] Song, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in artificial intelligence. PMLR, 2020.\n\nQUESTIONS:\n1. Why not use other score matching techniques mentioned in weaknesses 2 to learn the score function? I don't see any advantages in using the denoising score matching as the noise perturbation design is mainly beneficial to the generation process of diffusion models. Especially, the denoising score matching learns the score functions of the noise-perturbed data rather than the true score. \n\n2. How to compute the $t\\to 0$ in line 237?\n\n3. The pretrained model can only guaratee learning the score function of the expert policy accurately on the dataset distribution, but not on the current behavior policy distribution. How to handle these distributional shift when you query the score function of current policy using score network trained from the offline dataset?\n\n4. I might not be fully aware of the thoeritical contributions as no siginifcant parameters show up in the theoritical results. How is the theoritical results compared to other baseline algorithms, or none of the baseline algorithms can have such theoritical guarantees?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), a framework designed to learn safe and high-performance policies from offline datasets. The approach leverages a diffusion model to imitate the behavioral policy and uses reverse KL divergence to regularize a simplified Gaussian policy. In addition, it employs a gradient manipulation technique that adaptively interpolates between competing objectives. The method is evaluated on the DSRL benchmark, outperforming several existing offline safe RL methods in more than half of the tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- DRCORL employs a diffusion model to learn the complex behavioral policy from offline data, then extracts and regularizes a separate, simplified Gaussian policy using the diffusion model's score function. This design enables fast inference speeds. \n\n- The gradient manipulation technique offers an adaptive solution to the reward-safety trade-off, making the method more generalizable across different cost constraint conditions.\n\n- Extensive evaluations on 12 benchmark tasks from SafetyGym, BulletSafetyGym, and MuJoCo demonstrate that DRCORL achieves better performance while maintaining safety constraints, providing competitive scores against BCQ-Lag, CPQ, and COptiDICE.\n\nWeaknesses:\n\n- Not enough discussion or comparison of non-diffusion-based offline safe RL methods, such as ensemble-based uncertainty estimation or constrained supervised learning approaches, which may offer competitive performance without relying on generative models. Including these could better contextualize the advantages of DRCORL. \n\n- While DRCORL avoids inference-time sampling from the diffusion model, pretraining remains computationally expensive, especially for large-scale problems. In high-dimensional state-action spaces (e.g., manipulation with vision inputs or multi-agent systems), training diffusion models and critics can become very resource-intensive, limiting scalability and practical deployment. In addition, it may require a high-quality offline dataset and relies on good coverage of the behavioral policy to generate the diffusion models. Its performance may degrade under poor dataset conditions.\n\nQUESTIONS:\n- Could you comment more on DRCORL against non-diffusion-based offline RL approaches that are safety-aware but use simpler regularization (e.g., model ensembles or cost-conditioned Q-learning)? I would like to learn more about justifying diffusion modeling as necessary?\n\n- How does the diffusion model behave in cases where the offline dataset is sparse, multi-modal, or contains unsafe behaviors? Can the method robust enough to handle these situations which are more realistic in the real world, highlighted by some papers like Learning from Sparse Offline Datasets via Conservative Density Estimation. I have some concerns that the proposed method may be unstable when working on sparse data set.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}