Title: Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
Venue: NeurIPS 2025
Accepted: no
Keywords: reinforcement learning; diffusion policy
Abstract: Diffusion policies, widely adopted in decision-making scenarios such as robotics, gaming and autonomous driving, are capable of learning diverse skills from demonstration data due to their high representation power. However, the sub-optimal and limited coverage of demonstration data could lead to diffusion policies that generate sub-optimal trajectories and even catastrophic failures. While reinforcement learning (RL)-based fine-tuning has emerged as a promising solution to address these limitations, existing approaches struggle to effectively adapt Proximal Policy Optimization (PPO) to diffusion models. This challenge stems from the computational intractability of action likelihood estimation during the denoising process, which leads to complicated optimization objectives. In our experiments starting from randomly initialized policies, we find that online tuning of Diffusion Policies demonstrates much lower sample efficiency compared to directly applying PPO on MLP policies (MLP+PPO). To address these challenges, we introduce NCDPO, a novel framework that reformulates Diffusion Policy as a noise-conditioned deterministic policy. By treating each denoising step as a differentiable transformation conditioned on pre-sampled noise, NCDPO enables tractable likelihood evaluation and gradient backpropagation through all diffusion timesteps. Our experiments demonstrate that NCDPO achieves sample efficiency comparable to MLP+PPO when training from scratch, outperforming existing methods in both sample efficiency and final performance across diverse benchmarks, including continuous robot control and multi-agent game scenarios. Furthermore, our experimental results show that our method is robust to the number denoising timesteps in the Diffusion Policy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes NCDPO, an improvement over DPPO for online diffusion policy optimization. DPPO is an RL algorithm which employs diffusion models to parameterize the policy and formulates the optimization as a bi-level MDP, one for the diffusion process and one for the environment MDP. Although the expressiveness of diffusion policies should be better than standard MLP policies, the authors found that DPPO typically underperforms MLP+PPO and attributed the phenomenon to the optimization difficulty of the bi-level MDP formulation. Building on this understanding, NCDPO formulates the diffusion process as a deterministic inference process by pre-determining the noises used during inference as conditions. In this way, the diffusion process is in fact *contracted* and therefore standard RL algorithms can be applied to optimize the policy. Since PPO still requires a stochastic policy, the authors also added a learnable noise to the final action, such that the probabilities of actions can be computed.   NCDPO is evaluated on a wide range of benchmarks, including Gym-Locomotion, Robomimic and a multi-agent task Google Research Football. Generally, NCDPO demonstrates superior performance and stable convergence compared to baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n- The presentation of this paper is clear and easy-to-follow.\n- Some recent and relevant papers on diffusion policy optimization are missing from the related work section and comparison. To name a few: \n    - QSM [1] and DAC [2], which optimize diffusion policies by aligning the score function with action gradients. \n    - DIME [3], which fine-tunes diffusion policies with back-propagation though diffusion steps and seems to achieve better performance. \n    - BDPO [4], which also formulates the diffusion policy optimization as a bi-level MDP but circumvents back-propagation through time. \n    - SDAC [5], which uses re-weighted score matching to steer the generation towards high-Q regions. \n\n- From the experiments on Gym-Locomotion, it seems both NCDPO and DPPO are not sample-efficient compared to standard RL algorithms and recent progresses in diffusion policy optimization. For example, SAC only requires 3M environment steps to achieve similar or superior performance that NCDPO needs 10M steps. Some other diffusion-based methods, like DIME and SDAC, only requires 1M steps. I would encourage the authors to expand the set of baseline methods and include 1) standard online RL algorithms (e.g. SAC, TD3); 2) offline-to-online methods and 3) diffusion-based methods (see below). \n\n[1] Learning a Diffusion Model Policy from Rewards via Q-Score Matching\n\n[2] Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning\n\n[3] DIME:Diffusion-Based Maximum Entropy Reinforcement Learning. \n\n[4] Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning\n\n[5] Efficient Online Reinforcement Learning for Diffusion Policy\n\nQUESTIONS:\n- Since you wish to formulate the generation process as deterministic inference, why not use ODE to solve the final actions or more straightforwardly, use flow policies? \n- How are table 1 and table 2 related to the main text? I don't see any references to these tables.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel and intuitive method to incorporate diffusion models into reinforcement learning (RL) by treating the diffusion process as a deterministic policy. The authors achieve this by augmenting the environment state with sampled noise, allowing the diffusion model to act like a deterministic function (similar to an MLP predicting the mean action). This reformulation enables the computation of action likelihoods and makes the approach compatible with standard policy gradient methods such as PPO. Experiments are conducted primarily on OpenAI locomotion tasks, comparing this method to several RL baselines suitable for diffusion architectures.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- **Conceptual Simplicity and Innovation**: The core idea is simple yet creative. By augmenting the state with noise, the authors transform a stochastic diffusion model into a deterministic policy representation. This re-interpretation is elegant and avoids the need for complex inference or training pipelines that are common in previous RL + diffusion model works.\n\n- **Practical Integration**: The proposed method integrates well with PPO without requiring significant architectural or algorithmic changes. This makes it practical and easy to adopt.\n\n\n**Weaknesses**\n\n1. **Writing and Presentation**\n   - The writing lacks clarity in several parts and could benefit from more careful editing. The paper's storyline is at times hard to follow.\n   - Figures are poorly formatted, and the overall typesetting does not meet the standard expected of top-tier conferences like NeurIPS.\n\n2. **Limited and Outdated Experimental Setup**\n   - The primary evaluation is conducted on the OpenAI locomotion benchmark, which is now considered outdated. These environments offer limited insight into the potential benefits of diffusion models, especially in 2025 when the field has largely moved toward more complex domains.\n   - The main premise of diffusion models being beneficial in high-dimensional or multimodal tasks is not tested. There are no experiments involving image-based observations or complex sensory inputs.\n   - The paper does not include a direct comparison between the proposed diffusion policy and a standard MLP policy (with and without noise state augmentation) under the same PPO setup. As such, it is unclear whether the diffusion model actually improves performance in these tasks.\n\n3. **Lack of Demonstration of Unique Advantages**\n   - The paper does not convincingly demonstrate a scenario where the proposed method is clearly better than existing approaches. All of these domains can be easily solved by an MLP and PPO. It would be particularly impactful to see the method succeed in tasks where standard policies struggle, such as those requiring multimodal output distributions or robust multi-step planning.\n\nQUESTIONS:\n**Areas to improve**\n\n- **Improve the Writing and Formatting**: \nThe paper would benefit greatly from a clear and concise writing style, improved logical flow, and better-organized figures and captions.\n\n- **Upgrade the Experimental Benchmark**: \nConsider evaluating the method on more modern benchmarks. These would better showcase the potential of diffusion models and offer more meaningful insights.\n\n- **Add Comparisons to MLP Policies**: \nA key missing component is a comparison to a standard MLP policy (with and without state augmentation) using PPO. This would help clarify whether the proposed reparameterization of the diffusion model provides practical benefits.\n\n- **Explore Multimodal or Visual Tasks**: Showing that your approach excels in environments requiring multimodal action generation or learning from complex observations would strongly reinforce your claims.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes a novel algorithm combining the idea of PPO with diffusion models for the purpose of sequential decision-making. It considers each denoising step as a differentiable transformation to obtain an estimate of action likelihood, through out the denoising steps in diffusion process. The results are evaluated using standard offline RL datasets, showing solid improvement.

STRENGTHS AND WEAKNESSES:
+ The idea of considering each denoising step as a differentiable transformation to obtain an estimate of action likelihood is novel. It leads to tractable optimization computing action likelihood estimates.
+ Evaluation shows that the proposed method outperforms baselines consistently in the evaluation. NCDPO outperforms baselines in both sample efficiency and final reward.
- The evaluation should add a few more baselines using diffusion policies, such as Decision Diffuser, Decision Stacks, and CEIL. 
- Some theoretical analysis to demonstrate NCDPO's benefit in terms of sample efficiency would be helpful.
- In addition to training RL on randomly initialized policies, can NCDPO further improve the performance of an already-trained RL policy?

QUESTIONS:
- How does the algorithm compare with additional diffusion baselines such as Decision Diffuser, Decision Stacks, and CEIL?
- In addition to training RL on randomly initialized policies, can NCDPO further improve the performance of an already-trained RL policy? 
- How does NCDPO compare with baselines on random datasets?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04

comments: SUMMARY:
This paper proposed a method to finetune diffusion policy using proximal policy optimization. The method is an improvement over DPPO[20] by using the noise as part of the states, so that the denoising process of the current and old diffusion policy share the same noise in finetuning. Experiments on RL and robotics benchmarks demonstrate the effectiveness of the proposed method

STRENGTHS AND WEAKNESSES:
Strengths:
1. It is technical sound to use the same noise for \pi_\theta and \pi_\theta_old in finetuning to reduce loss variation and improve sample efficiency. It is somewhat novel and effective. The results are significantly better than [20] as reported in this paper
2. The proposed method is a straightforward modification from DDPO and can be easily adapted to different variants of diffusion policy finetuning methods
3. The paper reports the experimental results on multiple benchmark datasets including OpenAI Gym and Robomimics. 

Weaknesses:
1. The novelty is limited. The proposed method is very similar to DDPO except that \pi_\theta and \pi_\theta_old share the same noise. 
2. Experimental results are not convincing enough. All experiments are in simulation and there's no real world experiments. 
3. The graphics quality and layout are not visually pleasing. The figure captions in Sec 6.4 is not informative. The paper seems to be completed in a rush
4. The title is uninformative. It provide little information about the proposed method and could just easily describe existing methods.

Overall the idea is technical sound and somewhat novel but the paper looks hastily written

QUESTIONS:
It would be more convicing if the authors could report the results on robot hardwards as in [20]

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-20