Diffusion Policy Policy Optimization

Authors: Allen Z. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, Max Simchowitz
Primary area: reinforcement learning
Keywords: reinforcement learning, diffusion policy

Abstract:
We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730257142750

thread_type:
official_review

<thread_object>
creator: reviewer_sdz9

<content>
summary:
This paper studies how to use PPO to finetune the diffusion-model based policy in an offline-to-online setting. By integrating denoising steps of diffusion model into task MDP, authors could utilize PPO to finetune the diffusion model for improved performance. Extensive efforts are put to find the best practice in fine-tuning diffusion policy, in terms of hyperparameters, network architecture and algorithmic variants. Finally,  the efficacy and the performance of the proposed framework  is demonstrated on various experiments in both simulated and real-world scenarios.

strengths:
This paper contains tons of discoveries related to fine-tuning diffusion policy via PPO. Specifically, numerous aspects like denoising steps to be fine-tuned, network architecture, GAE variants  and effects of expert data are all studied with experimental results reported. As fine-tuning diffusion policy for real-world robot is a very important field, these experiments are welcome and contain significant value  for the community. Last but not least, the authors also provide an empirical explanation for why their method obtains better results than others (i.e. structural exploration).

$~$

## Originality 
Utilizing RL method to finetune diffusion model is in great need. And the fruitful experimental results presented in this work  are of great value and originality.

## Quality

Many experiments and ablation studies are conducted and the best practices are summarized from them.

## Clarity

Unfortunately, this paper is not clearly written. In the main papers (1-10 pages), authors’ writing lacks a central logic  and the results are presented without  a focus. I have great trouble understanding the contribution and experimental results during reading. 

## Significance
The large amount of experiments conducted is of great value to the community, especially considering there are also real-world experiments.

weaknesses:
Unfortunately, I personally think the presentation of this paper is very poor. With the extensive and fruitful experiments results, the authors fail to present them in a concise and focused manner in the main pages and make me quite confused during reading and comprehending this paper.

Firstly, there are three components that are actively mentioned and studied: the diffusion policy, the PPO method and the best practice for combining them together. 

But based on my comprehension, the diffusion policy and PPO used in this work are of no novel changes & contributions.

Combining these 2 together and successfully applying on real-world robotic system, is with great value and potential hidden evils. But  the authors fail to present this to me  in a clear and concise manner: for example, there are  experiments comparing diffusion policy (DP) to gaussian parameterized policy, but the advantage of DP to vanilla parameterization is already well-established and not the contribution of this paper, in my opinion. Since this paper is already over-sized, I suggest the authors to relocate some contents into appendix and focus on your main contribution in the main pages



To summarize, I hope the organization and presentation of this paper could be improved such that the problem formulation, method design and experimental results could be tightly chained and focused on  your main contribution.

questions:
1. Could you change the title "Diffusion Policy Policy Optimization" to provide more contexts to readers? Current title is too short and abstract and your work is not as general as it indicates. 
2. I would  suggest deleting/relocating some contents in early chapters as your paper is already over-sized and some of these contents are already known.
4. in line 249, the definition of $\gamma_{\text{ENV}}^t$ seems to be incorrect? I thinks it should be $\gamma_{\text{ENV}}^{t^\prime-t}$? or are there any further treatments of this discount factor?
4. there lacks a pseudo-code for the whole algorithm, please provide one. For example, how is your V function trained?
5. for fine-tuning control policy in RL, most existing algorithms finish in less than 1e6 steps. But the results presented throughout this paper is conducted in 1e6~1e7 steps. Could you truncate the figures to 5e5 such that we could better compare the performance of various methods?

6. Could you provide more comparisons of DPPO against established offline-to-online methods like [1] and [2]?

>[1] Wang, Shenzhi, et al. "Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
>
>[2] Zhang, Haichao, Wei Xu, and Haonan Yu. "Policy Expansion for Bridging Offline-to-Online Reinforcement Learning." The Eleventh International Conference on Learning Representations.

7. Could you provide more explanations about the worse performance of DAWR, DRWR and Cal-QL? In appendix, the result of the performance of Cal-QL on  Can and Square are almost zero, which is surprising to me. Could you elaborate more on this? BTW, what is the performance of Cal-QL when only offline training is performed?

8. for DRWR, in the weight function, could you minus a V function from the estimated return such that its’ more aligned toward DPPO?
9. in line 988-994, you mentioned speedup and less GPU memory usage is obtained by fine-tuning only the last few K denoising steps. But the method is to  use a trainable $\theta$ for the last K steps and a frozen $\theta_{\text{FT}}$ for previous steps. How does this method improve the training speed and reduce GPU memory?

10. Sect. 6 of structural exploration is of much importance and novelty. Could you try ablating on this phenomenon for DP? for example, by tuning the denoising process such that DP itself performs less or more structural exploration and see if the performance is affected?

There are many interesting, fruitful discoveries and experiments in this work. But I find none of them is studied exhaustively or presented clearly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730338116044

thread_type:
official_review

<thread_object>
creator: reviewer_wtdz

<content>
summary:
The papers introduces Diffusion Policy Policy Optimization (DPPO), an algorithmic framework designed to fine-tune diffusion-based policies using policy gradient optimization (PPO) from reinforcement learning (RL). DPPO enables structured exploration, enhances training stability, and demonstrates the robustness and generalization during deployment. It improves policy performance across various benchmarks, including tasks with pixel-based observations and long-horizon rollouts, which have traditionally been challenging for previous RL methods.

strengths:
(1) The paper introduces a novel dual-layer MDP framework and the use of PPO to fine-tune Diffusion-Policy, enhancing its performance in various tasks.
(2) The paper proposes several best practices for DPPO, such as fine-tuning only the last few denoising steps and using DDIM sampling to replace some denoising steps, improving efficiency and performance.
(3) DPPO shows a strong performance in all benchmark tests, particularly in tasks involving pixel-based observations and long-horizon rollouts.
(4)  DPPO achieves highly efficient zero-shot transfer from simulation to real-world tasks in robotics, demonstrating a minimal sim-to-real performance gap.

weaknesses:
(1) Figure 10 shows that the pre-trained Diffusion Policy can model multimodality in the behavior policy. However, after fine-tuning, the Diffusion Policy gradually converges into a deterministic policy. In this case, does it lose the advantage of expressing multimodal action distributions that Diffusion Policy is known for?

(2) In the experiments, DPPO is only compared against other diffusion-based methods, without any comparison to existing state-of-the-art Offline2Online or Sim2Real algorithms, such as Uni-O4 [1] and O3F [2]. Clearly, there is a significant optimal gap in environments like HalfCheetah.

(3) The motivation for fine-tuning models is typically to reduce the number of online interactions and improve sample efficiency. However, DPPO does not seem to significantly enhance sample efficiency, as its convergence speed is similar to other online reinforcement learning methods.

(4) The v-D4RL [3] dataset could have been considered in the experiments for broader evaluation.

(5) Compared to QSM [4], DPPO does not introduce particularly novel concepts but rather provides engineering or empirical adjustments, such as fine-tuning certain denoising steps and replacing value estimation with advantage estimation.

[1] Kun Lei, et al. Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
[2] Seunghyun Lee, et al. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
[3] Cong Lu, et al. Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
[4] Michael Psenka, et al. Learning a Diffusion Model Policy from Rewards via Q-Score Matching

questions:
Please see the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550673812

thread_type:
official_review

<thread_object>
creator: reviewer_7q1r

<content>
summary:
This paper proposes a framework (DPPO) to enable fine-tuning pre-trained diffusion-based policies through policy gradient methods. They compare DPPO with rich baselines in common benchmarks. They conclude best practices for DPPO and conduct ablation studies to understand the effectiveness of DPPO.

strengths:
* This paper conducts rich experiments and analysis. They compare DPPO with other fine-tuning RL methods for diffusion policies and other fine-tuning RL policies. They also conduct several ablation studies and visualization to analyze the effectiveness of DPPO.
* This paper is well-written and well-structured. They conclude the best practice for DPPO and the benefits of such framework.

weaknesses:
* The novelty of the paper is limited. Previous works have proposed the idea that considers a denoising process as multi-step MDP. This paper simply integrate this MDP into the environmental MDP. The tricks to adapt PPO for better fine-tuning are also seen in the previous works like GAE except fine-tuning the last few steps.

questions:
* In the original diffusion policy paper, the performance in the transport is nearly 100% success rate, which conflicts with the results in the paper.
* How about the results compared with the baselines that introduce a residual policy?
* How about the results compared with the baselines that use RLPD or Cal-QL for the pre-trained diffusion policies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701326699

thread_type:
official_review

<thread_object>
creator: reviewer_yzhx

<content>
summary:
The authors propose a framework for finetuning a pretrained diffusion policy using on-policy RL, specifically PPO, and show that their method improves performance on long-horizon sparse-reward tasks and works on real hardware.

strengths:
- The authors explore the possibility of fine-tuning diffusion-based policies with on-policy RL and demonstrate some surprising results. Specifically, the authors find improved performance over baselines on long-horizon and sparse reward tasks. The authors also demonstrate how their method can leverage environment parallelization to train faster than off-policy RL methods. Given the abundance of GPU-accelerated simulators these days, I believe this to be a major strength as off-policy RL finetuning methods cannot leverage parallelization in the same way, leaving these high-throughput simulators as a relatively untapped resource until now. 

- The authors provide extensive comparisons to a variety of other RL finetuning methods, including two of their own novel baselines. The ablations are also thorough. 

- The paper is well written and the quality of the presentation is high.

weaknesses:
- Much of section 6, including the first few paragraphs up to and including Figure 8 are redundant and unnecessary. At this point there are a number of diffusion policy papers applied to various tasks as well as the original diffusion policy paper itself that show that diffusion models consistently outperform GMM and Gaussian policies at capturing the modes of a multi-modal distribution, and so this result is no longer surprising or novel. I'd have rather seen this space dedicated to showing more experimental results that were instead put in the appendix, such as the comparisons in Figure 12 to methods that finetune on additional expert data, or perhaps a more in-depth analysis on why DPPO does better on long-horizon sparse-reward tasks. "How does this method compare to just collecting and training (via BC or whatever else) on additional expert data?" was a question that came to mind while I read the results, which is an important one to discuss and one I imagine other readers will have. 

- While the technical contribution is fairly good, the novelty of this approach is relatively low. The proposed method follows the more general and somewhat repetitive trend of "take diffusion policy and try it on task X and potentially finetune with method Y" style papers in the robot learning community. 

Despite the somewhat low novelty, I believe that the impressive results and the fact that this paper is one of the first, if not the first, diffusion policy finetuning methods capable of leveraging high-throughput simulators is an important contribution to the robotics and ML communities.

questions:
- What intuition, if any, do the authors have on why DPPO does better on long-horizon sparse-reward tasks? Is there something special about finetuning with PPO as opposed to other finetuning strategies that helps? 

- In Figure 6, why does DPPO-UNet get outperformed by DPPO-MLP on state-based tasks? And why is DPPO-UNet not present on the pixel-based versions of the task? 

- On lines 264-265 the authors state *"We propose clipping the diffusion noise schedule at a larger-than-
standard noise level to encourage exploration and training stability."*. If more noise encourages exploration, would it not be more beneficial to use the variance-exploding line of diffusion models such as EDM instead of the variance-preserving ones? 

- In Figure 7 we see positive sim2real transfer from pretraining only, but negative transfer after fine-tuning. Do the authors have some intuition as to why this is the case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732079448317

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta response from the authors

comment:
We thank all the reviewers for their constructive feedback. To address the concerns, we have made a series of major revisions to the paper summarized below (the section and figure numbering are consistent with the revised version):

- [Experiment] In Sec. 5.2 and Appendix D.1, we have added additional results with the popular Franka Kitchen tasks from D4RL when comparing DPPO to other demo-augmented RL methods. Methods based on Q-learning work better in these tasks as expected, but they perform much worse in Robomimic tasks where higher precision is required.
- [Experiment] In Sec. 5.3, we have added additional results with UNet as the policy head for DPPO in Robomimic tasks with pixel input.
- [Experiment] In Appendix D.1, we have additional results using diffusion policies for RLPD, Cal-QL, and IBRL, showing that the issue with Q learning persists even with diffusion, corroborating our original findings when comparing to Q-learning-based diffusion RL methods in Sec. 5.1.
- [Experiment] In Appendix D.7, we have added an ablation study on structured exploration of DPPO, showing its contribution in faster fine-tuning.
- [Presentation] At the beginning of Sec. 4, we have expanded on the discussion of naively running policy gradient with diffusion and pointing out the experiments using exact likelihood in the appendix, thus better motivating the novel two-layer MDP setup considered in DPPO. 
- [Presentation] In Sec. 4 and Appendix A, we have made the distinction between DPPO’s and the policy gradient’s derivation from QSM [1] clear, highlighting the novelty of the DPPO formulation.
- [Presentation] At the beginning of Sec. 5, we have included rationales for considering each set of experiments to better motivate them and make the transition from the approach section to experimental results more coherent.
- [Presentation] In Appendix C, we have included pseudocode of DPPO and expanded the discussion on our GAE-based advantage estimation.
- [Presentation] To save space, we have moved the paragraph “Potential Impacts beyond Robotics” in the introduction to Appendix A, removed the original Fig. 2 showing the hardware trial, and trimmed discussions at a few places.

Please see the attached revised paper with the changes highlighted in blue. We also would like to address some of the shared concerns from the reviewers here.

## **Novelty of the overall DPPO approach**

We agree that the methodology proposed in this work admits a simple implementation and that it draws on recent ideas in the literature for fine-tuning diffusion models [1, 2]. While the instantiation of DPPO, together with the careful design decisions, is certainly one of our contributions, we argue that our more significant contribution is the **scientific finding** that this relatively simple method, based on policy optimization, **significantly outperforms other proposed methods in challenging robotic tasks**. In other words, our intention is not to combine DP + {an arbitrary RL method} and show that it is performant. Rather, it is to show that, with the right perspective, DP + {one of the simpler and more natural RL methods} matches and often outperformes all of the more complicated alternatives especially in difficult tasks. These contributions rely on a number of conceptual insights that inform the engineering details and which substantially differentiate our approach to those taken in prior works. We also stress that our submission was careful not to overclaim about which aspects were algorithmically novel, and instead emphasize developing a comprehensive scientific understanding of why this particular combination is (to borrow from the messaging of many other papers in the broader literature) “all you need.”

**Finding 1: Performance of DPPO / Surprising effectiveness of a Two-Layer MDP, despite horizon increase:** 
Our core finding is that DPPO, a relatively simple instantiation of policy optimization, drastically outperforms existing methods. We are open about the fact that Black et al. [1] interpret the denoising process as an MDP for fine-tuning image diffusion models. However, unrolling the denoising process in diffusion policies had not been considered in prior work. This is because unrolling the MDP increases the number of effective MDP steps by a factor of K, and we know that long-horizon RL problems are frequently more challenging. This led to other methods exploiting techniques either from offline RL [2] or ones inspired by diffusion guidance methods [3]. We also note that the authors of QSM tried a similar approach to ours but based on REINFORCE, and found it to have low performance. We ourselves thought our approach might not work, and indeed, its performance was rather unremarkable for the first few months of experimentation. Thus, the fact that such a method indeed is the **most performant** comes as a major surprise to us as authors, and we hope, to the community.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a technique for fine-tuning diffusion policies with a policy gradient method. Previously, policy gradient updates of diffusion policies were conjectured to have training instability. With the new technique proposed, the authors show that the resulting algorithm called DPPO can fine-tune to superior policies both in simulations and real robots. Extensive comparison is provided. This paper has the potential to be adopted by many papers in the future and as such I recommend this paper for a spotlight presentation.

Minor comments:

> using the policy gradient (PG) method

Change to “Using a policy gradient method”

> As the first glance

Change to “At first glance”