Diffusion Policy Policy Optimization
Authors: Allen Z. Ren
Justin Lidard
Lars Lien Ankile
Anthony Simeonov
Pulkit Agrawal
Anirudha Majumdar
Benjamin Burchfiel
Hongkai Dai
Max Simchowitz
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
diffusion policy
Primary area: reinforcement learning
TL;DR: We introduce DPPO, an algorithmic framework and set of best practices for fine-tuning diffusion-based policies in continuous control and robot learning tasks.

Abstract: We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task.

Published as a conference paper at ICLR 2025
DIFFUSION POLICY POLICY OPTIMIZATION
Allen Z. Ren', Justin Lidard', Lars L. Anthony Simeonov Pulkit Agrawal?, Anirudha Majumdar', Benjamin Burchfiel", Hongkai Dai Max Simchowitz 3.5
Prrinceton University 'Harvard University "Massacuueets Institute of Technology 'Toyota Research Institute "Carnegie Mellon University
ABSTRACT
We introduce Diffussion Policy Policy Optimization, DPPO, an algorithmic frame- work including best practices for fine-tuning diffusion-based policies (e.g. Diffu- sion Policy (Chi et al., 2024b)) in continuous control and robot learning tasks us- ing the policy gradient (PG) method from reinforcement learning (RL). PG meth- ods are ubiquitous in training RL policies with other policy parameterizations: nevertheless, they had been conjectured to be less efficient for diffusion-bart policies. Surprisingly, we show that DPPO achieves the strongest overall perfor- mance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL. fine-tuning and the dif- fusion parameterization, leading structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of limulation-traines policies on robot hardware in a long-hore horizon,,,, multi-stage manipulation task. Webpage with code: diffusion- oppooo.....pooopooooooooooooo gi github. io.
INTRODUCTION
Large-scale pre-training with additional fine-tuning has become a ubiquitous pipeline in the de- velopment of language and image foundation models (Brown et al., 2020; Radford et al., 2021; Ouyang et al., 2022; Ruiz al., 2023). Though behavior cloning with expert data (Pomerleau, 1988) is rapidly emerging
dominant paradigm for pre-training robol policies (Florence et al., 2019; 2022; Zhao et al., 2023; Lee et al., 2024; et al., 2024), their performance can subopti- mal (Osa et al., 2018) due to expert data being suboptimal expert data exhibiting limited coverage of possible environment conditions. As robot policies entail interaction with their environment, rein- forcement learning (RL) (Sutton and Barto, 2018) is natural candidate for further optimizing their performance beyond the limits of demonstration data. However, RL fine-tume one-tuning can be nuanced for pre-trained policies parameterized a leading parameterization for action policies (Chi et al., 2024b; Reuss et al., 2023; Pearce et al., 2023), due in large part their high training stability and ability to represent complex distributions (Rombach et al., 2022; Poole et al., 2022; Kong et al., 2020; Ho et al., 2022).
diffusion models (Ho al., 2020), which have emerged as
Contribution (DPPO). We introduce Diffiusion Policy Policy Optimization (DPPO), a generic framework as well as set of carefully chosen design decisions for fine-tuning diffusion- -based robot learning policy via popular policy gradient methods (Sutton et al., 1999; Schulman et al., 2017) in reinforcement learning.
The literature has already studied improving/fie-tuninn using RL (Psenka et al., 2023; Wang et al., 2022; Mansen-Estrucl et al., 2023), and has applied policy gradient (PG) to fine-tuning non-interactive applications of diffusion models such as text-to- image generation (Black et al., 2023; Clark et al., 2023; Fan et al., 2024). Yet PG methods have been believed to be inefficient in training Diffusion Policy for continuous control tasks (Psenka et al., 2023; Yang et al., 2023). On the contrary, show that for a Diffusion Policy pre-trained from expert demonstrations, our methodology for fine-tuning via PG updates yields robust, high- performing policies with favorable training behavior.
diffusion-baase based policies (Diffusion Policy)

---

Published as a conference paper at ICLR 2025
DPPO: Diffusion Policy Policy Optimization
Environment MDP
Velog mfoe - 144-s)
Policy
Diffusion MDP
Diffusion
Gradlent Update
MDP
Structured exploration /
Training stability
Policy robustness
We introduce DPPO, Diffission Policy Policy Optimization, that fine-tunes pre-trained Diffusion Policy using policy gradient. Extensive experiments in simulation and hardware show DPPO affords structured exploration and training stability during policy fine-tuning, and the final policy exhibits strong robustness and generalization. DPPO improves policy performance across benchmarks, including ones with pixel observations and with long-horizon rollouts that have been very challenging to solve using previous RL methods.
Figure 1:
Contribution (Demonstration of DPPO's Performance). We show that for fine-tuming a pre- trained Diffusion Policy, DPPO shows strong training stability across tasks and marked improve- ments in final performance in challenging robotic tasks in comparison to a range of alternatives, including those based on off-policy Q-learning (Wang et al., 2022; Hansen- Yang et al., 2023; Psenka et al., 2023) and weighted regression (Peng et al., 2019; Peters and Schaal, 2007; Kang et al., 2024), other eemo-augmented RI. methods (Ball et al., 2023; Nakamoto al., 2024; Hu al., 2023), well common policy parameterizations such Gaussian and Gaussian Mixture models.
al., 2023;
The above finding might be surprising because PG methods do not appear to take advantage of the unique capabilities diffusion sampling (c.g., guidance (Janner et al., 2022; Ajay et al., 2023)). Through careful investigative experimentation, however, we find unique synergy between RI. fine-tuning and diffusion- policies.
Contribution 3 (Understanding the mechanism of DPPO's success). We complement results with numerous investigative experiments that provide insight into the mechanisms behind DPPO's strong performance. Compared other common policy parameterizations, DPPO engages in structured exploration that takes better ad advantage of the "manifold" of training data, and finds policies that exhibit greater robustness perturbation.
provide evidence that
Through ablations. we further show that our design decisions overcome the speculated limitation of PG methods for tine-tuning Diffusion Policy. Finally, to justify the broad utility of DPPO, we verify its efficacy across both simulated and real environments, and in situations when either ground-truth states or pixels are given to the policy as input.
Contribution 4 ('ackliig challenging robotic tasks and settings). We show DPPO is effective in challenging robotic and control settings, including pixel observations and long-horizon manipulation tasks with sparse reward. We deploy policy trained in simulation via DPPO real hardware in zero-shot, which exhibits smoother behavior than the baseline and transfers better to the real-world.
RELATED WORK
Policy optimization and its application to robotics. Policy optimization methods update ex- plicit representation of an RL policy - typically parameterized by neural network - by taking gradients through action likelihoods. Following the seminal policy gradient (PG) method (Williams, 1992: Sutton et al., 1999), there have been a range of algorithms that further improve training stabil- ity and sample efficiency such as DDPG (Lillicrap et al., 2015) and PPO (Schulman al., 2015). PG methods have been broadly effective in training robot policies (Andrychowicz et al., 2020; Hwangbo et al., 2019; Kaufmann et al., 2023; Chen et al., 2022b), largely due to their training stability with high-dimensioonl continuous action spaces, as well as their favorable scaling with parallelized simu- lated environments. Given the challenges of scrathh exploration in long-horizon tasks, PG has seen great success in fine-tuning baseline policy trained from demonstrations (Rajeswaran et al., 2017: Torne et al., 2024; Peng et al., 2021). Our experiments find DPPO performing on- polpolic PG often achieves stronger final performance in manipulation tasks, especially ones with long horizon
2

---

Published as a conference paper at ICLR 2025
and high- digg--imensiooal action space, than off- off-policy Q-learning methods (Ball al., 2023; Nakamoto et al., 2024; Hu et al., 2023).
Learning and improving diffusion-based policies. Irffusion-basee policies (Chi et al., 2024b; Reuss ct al., 2023; Ankile ct al., 2024a; Ze et al.; Wang et al., 2024a; Sridhar et al., 2023; Pearce et al.,, 2023) have shown recent success in robotics and decision making applications. Most typically, these policies are trained from human demonstrations through a supervised objective, and enjoy both high training stability and strong performance in modeling complex and multi-modal trajectory distributions. As demonstration data are often limited and/or suboptimal, there have been many approaches proposed improve the performance of diffusion-based policies. One popular approach has been to guide the diffusion denoising process using objectives such as reward signal or goal conditioning (Janner al., 2022; Ajay et al., 2023; Venkatraman et al., 2023; Chen et al., 2024). More recent work has explored techniques including Q-learning and weighted regression, either from purely offline estimation (Chen et al., 2022a; Wang et al., 2022; Ding and Jin, 2023), and/or with online interaction (Hansen-Estruch et al., 2023; Psenka et al., 2023; Yang et al., 2023).
Policy gradient through diffusion models. RL techniques have been used to fine-tune diffusion models such as ones for text-to-image generation (Fan and Lee, 2023; Fan et al., 2024; Black et al., 2023; Wallace et al., 2023). Black et al. (2023) treat the denoising process as an MDP and apply PPO updates. We build upon these carlier findings by embedding the denoising MDP into the environmental MDP of the dynamics in control tasks, forming a two-layer "Diffusion Policy MDP". Though Psenka al. (2023) have already shown how PG can be taken through Diffusion Policy by propagating PG through both MDPs, they conjecture that it is likely to be ineffective due to large action variance caused by the increased effective horizon induced from the denoising steps. Our results contravene this supposition for diffusion-based policies in the fine-tuning setting.
3 PRELIMINARIES
Markov Decision Process. We consider Markov Decision Process (MDP) .MNN (S, A, Pa, P. R) with states E S, actions E A, initial state distribution Po. transition proba- bilities P, and reward R. At each timestep f, the agent (e.g., robot) observes the state 8; E S, takes an action 0₂ ~ Taa (x) E A, transitions to the next state S1+1 according to 81+1 ~ P(s++1 | 8t, at) while receiving the reward R(s,, az)". Fixing the MDP MENV. we let E= (resp. PT)) denote the ex- pectation (resp. probability distribution) over trajectories (Su, ag. .... ST, ar) with length T +'+1, with initial state distribution so Po and transition operator P. We aim to train policy to optimize the cumulative reward, discounted by a function y(-): J(To) = Po EE (tt) a.)l
Policy optimization. The policy gradient method (e.g., REINFORCE (Williams, 1992)) al- lows for improving policy performance by approximating the gradient of this objective w.r.t. the policy parameters: VoJ (TTO) = E |E12 Ve log (r)) at]], r2(6), 0z) := Er207(7) ((ST))((TT)))))))) a7) where re is the discounted cumulative future reward from time t, 7 is the discount factor that depends on the time-step, and V log (e(())))) denotes the gradient of the logarithm of the likelihood of at | st. To reduce the variance of the gradient estimation, a state- value function T (st) can be learned to approximate E[[r]. The estimated advantage function Åno (8t, 0+) = re(Se, 04) (s.) substitutes re(s),
Diffusion models. A denoising diffusion probabilistic model (DDPM) (Nichol and Dhariwal, 2021; Ho et al., 2020; Sohl-Dickstein et al., 2015) represents a continuous-valued data distribution p(-) = p(z") as the reverse denoising process of forward noising process q(a q(a*1x*1) (a* that iteratively adds Gaussian noise to the data. The reverse process is parameterized by a neural network eq (œ k), predicting the added noise e that converts Fy to 2g (Ho et al., 2020). Sampling starts with a random sample xx ~ ~NO, I) and iteratively generates the denoised sample:
k- ~ po h (x0)) := NNx==1;; "u (x co(x", k)), o2.).
(3.1)
Above, (+x(-) is a fixed function, independent of 0, that maps x² and predicted sq the next mean, and o is a variance term that abides by a fixed schedule from k = l,..., K. We refer the reader to Chan (2024) for an in-depth survey.
For simplicity, we overload R(-,+) denote both the random variable reward and its distrihution.
3

---

Published as a conference paper at ICLR 2025
Diffusion models policies. Diffusion Policy (DP; see Chi et al. (2024b)) is policy xe parame- terized by DDPM which takes in s as a conditioning argument, and parameterizes pa(a" | a s) in (3.1). DPs can be trained via behavior cloning by fitting the conditional noise prediction (a 8, k) to predict the added noise. Notice that unlike more standard policy parameterizations such unimodal Gaussian policies, DPs do not maintain an explicit likelihood of po s). In this work, we adopt the common practice of training DPs to predict an action chunk - sequence of actions few time steps (denoted T2) into the future comparison, our diffusion and non-diffusion baselines use the same chunk size.
promote temporal consistency. For fair
4 DPPO: DIFFUSION POLICY POLICY OPTIMIZATION
Directly applying policy gradient does not work. At first glance, applying policy gradient to a new policy parameterization should be straightforward - the update simply uses the likelihood of the sampled action under the policy, mg(aa)(s). However, diffusion as a multi-step denoising pro- cess introduces challenges: while the intermediate denoised action likelihood To(a 82, a,,,,,)))) 1 can be readily evaluated, the likelihood of the final denoised action <o(a/=0ss (se) only be approxi- mated (Song et al., 2020b). In Appendix D.6 we show that differentiating through the approximated likelihood leads training instability and fails to optimize the pre-trained policies. Next we take the perspective of "denoising process as MDP" to address the issue.
=
Ro
==
R1
St
S0 K
80
80
S
S
S
do
K
0g
Q1
0 an
K
Diffusion Policy
Qn
Q1)
01
a1
We treat the denoising process in Diffusion Policy as an MDP, and the whole environment episode can be considered as chain of such MDPs. Now the entire chain ("Diffusion Policy MDP", MDP) involves Gaussian likelihood at each (denoising) step and thus can be optimized with policy gradient. Blue circle denotes the state and red circle denotes the action in Mpp.
Figure 2:
Two-Layer "Diffusion Policy MDP". As observed in Black et al. (2023) and Psenka al. (2023), a denoising process can be represented as a multi-step MDP in which policy likelihood of each denoising step can be obtained directly. We extend this formalism by embedding the Diffusson MDP into the environmental MDP, obtaining a larger "Diffusion Policy MDP" denoted MDe. visualized the notation 5 to denote Dirac distribution and to denote a product
in Fig. 2. Below, distribution. Recall the environment MDP MENU == (S, A, Po. P, R) in Section 3. The Diffusion MDP MDP uses indices E(t, tK + (K LK-K k 1) corresponding to kt,, which increases in t but (to keep the indexing conventions of diffusion) decreases lexicographically with K - 1 2 k 2 0. The states, actions and rewards are
k: >
ki(t,k) = (St, a; 1+11 ) Tet(tk) = at. RET. . = -
(iss,, a?)
k ==
where the bar-action at T(1, k) is the action at after one denoising step. Reward is only given at times corresponding to when at is taken. The initial state distribution is Pu Pg & /0 I). corresponding to Sy ~ Po is the initial distribution from the environmental MDP and a (O, independently. Finally, the transitions
(st, (sn.a)) (Se41) x P(1441 St. ag) & 1(
8= T= T= [t)
P(5
3y. az)
That is, the transition moves the denoised action af at step f(t, k) into the next state when > 0, or
otherwise progresses the environment MDP dynamics with 1: = 0. The pure noise at is considered part of the environment when transitioning at l = O. In light of (3.1), ), the policy in Mpp is
no (oz(e,)) Bi((,))) To(aa a,t ,.))
Naaff (aaf: ssa(att
k + 1, s,)) of 1111) (4.1)
4

---

Published as a conference paper at ICLR 2025
Fortunately, (4.1) is Gaussian likelihood, which can be evaluated analytically and is amenable to the policy gradient updates":
VoJ F pa
Ve log (ār (s) ar)], F(5r,
E((7) (4.2) 728
Evaluating the above involves sampling through the denoising process, which is the usual "forward pass" that samples actions in Diffusion Policy; as noted above, the inital
be sampled from the enviroment via p Po @ (O,, where Po is from the environment MDP.
4.1 INSTANTIATING DPPO WITH PROXIMAL POLICY OPTIMIZATION
We apply Proximal Policy Optimization (PPO) (Schulman et al., 2017; Engstrom et al., 2019; Huang al., 2022; Achiam, 2018), popular improvement of the vanilla policy gradient update.
Definition 4.1 (Generalized PPO, clipping variant). Consider general MDP. Given an advantage estimator A(s a), the PPO update (Schulman et al., 2017) is the sample approximation to
(e(ar 8c)
Te (cz 8x)
Va ( min (A (st,at) az)
A (st, az) clip
, E, 1 1e))).
To (0e 81))
0u (at 8x)
where E, the clipping ratio, controls the maximum magnitude of the policy updated.
We instantiate PPO in our diffusion MDP with (s, a,t) f- (5, Our advantage estimator respects the two-level nature of the MDP: let "ENY (0.1) be the environment discount and "DEeoos E (0, 1) be the denoising discount. Consider the environment- return:
F(5,, āG) = Σ Ne (B anwro))
f =f(t,k)
e2t
since R(E) = 0 O. This fact also obviates the need of estimating the value le > 1 and allows us to use the following denoising-discoune advantage estimator":
A (ăi, ar) == DDEoIs (F(EE, 0z) FRa (SE(..)))
The denoising-discounten discounting has the effect of downweighting the contribution of noisier steps (larger k) to the policy gradient (see study in Appendix D.2). Lastly, choose the value estimator to only depend the "s" component of s: a (5)(),)) == Va (sz). which find leads more efficient and stable training compared to also estimating the value of applying the denoised action a!=1 (part of (((),)) as shown Appendix D.2.
Best Practices for DPPO. We summarize a number of best practices for DPPO; precise details are given in Appendix B. (1) We achieve substantial efficiency gains by fine-tuning the last few steps of the DDPM, whilst in many cases obtaining performance comparable to fine-tuning all steps. (2) For additional efficiency gains, one may fine-tune the Denoising Diffusion Implicit Model (DDIM) (Song et al., 2020a) instead.
(3) We propose clipping the diffusion noise schedule at larger-than- standard noise level to encourage exploration and training stability.
PERFORMANCE EVALUATION OF DPPO
We study the performance of DPPO in popular RL and robotics benchmarking environments, For comparisons, we consider (1) alternative RI. methods for fine-tuning diffusion policies (Section 5.1), (2) alternative RI. methods that leverage expert data (Section 5.2). (3) policy optimization using alternative policy parameterizations (Gaussian/GMMM (Section 5.3) additionally in multi-stage ma- nipulation tasks including hardware evaluation (Section 5.4). These comparisons highlight the ef- fectiveness of (1) policy gradient RL method for diffusion, (2) DPPO general framework
?(Psenka al., 2023) proposes similar derivation but does consider the denoising process MDP. See further clarification Appendix A.
In practice, we Generalized Advantage Estimation (GAF, Schulman et al. (2015)) that better balances variance and bias in estimating the advantage. We present the simpler form here.
5

---

Published as a conference paper at ICLR 2025
for pre-training plus fine-tuning, (3) diffusion RL parameterization (4) especially in challenging tasks, respectively.
Environments:
OpenAl Gym. We first consider three population OpenAI GYM locomotion bench- marks (Brockman et al., 2016): (Hopper-v2. Walker2D- v2, ah- All poli- cies are pre-trained with the full medium-level datasets from D4RL (Fu et al., 2020) with state input and action chunk size Ta = 4. We use the original dense reward setup in fine-tuning.
Robomimic. Next consider four simulated robot manipulation tasks from the RObOMIMIC benchmark (Mandlekar et al., 2021), (Lift, Can, Square, Transport, J. ordered in increasing difficulty. These
Environments:
representative of real-world robotic tasks, and Squa and Transport (Fig. 3) are considered very challenging for RL training. Both state and pixel pol- icy input are considered. State-based and pixel-based policies are pre-trained with demonstrations provided by ROBOMIMIC. We consider Ta 4 for Can, Li ft, and Square, and Ta 8 for Transport. They are then fine-tuned with sparse reward upon task completion.
Environments: Furniture-Benn -ench & real furniture assembly. Finally, we demonstrate solving longer-horizon, multi-stage robot manipulation tasks from the FURNITURE-BENCH (Heo et al., 2023) benchmark. We consider three simulated furniture assembly tasks, [One-leg, Lamp, Round-table). We consider two levels of randomness over initial state distribution, Low and Med, defined by the benchmark. All policies pre-trained with 50 human demonstrations col- lected in simulation and Ta = 8. They are then fine-tuned with sparse (indicator of task stage completion) reward. We also evaluate the zero-shot sim-to-real performance with One-leg.
Transport
One-leg
Lamp
Round-table
Figure 3: Long-horizon robot manipulations tasks including (left) the bimanual Transport from ROBOMIMIC and (right) BEENITURE-BENCE tasks (full rollouts visualized in Fig. 23).
5.1 COMPARISON TO DIFFUSION-BASED RI. ALGORITHMS
We compare DPPO to an extensive list of RL methods for fine-tuning diffusion- policies. DRWR and DAWR are own, novel baselines based on reward-weighte regression (Peters and Schaal, 2007) and advantage- advantage-weightee weighted regression (Peng et al., 2019). The remaining methods, DIPO (Yang al., 2023), IDQL (Hansen-Estruch et al., 2023), DQL (Wang al., 2022), and QSM (Psenka et al., 2023), are existing in the literature. We evaluate on the three OpenAl GYM tasks and the four ROBOMIMIC tasks with state input; see Appendix F.3 for further details.
Overall, DPPO performs consistently, exhibits great training stability, and enjoys strong fine-tuning performance across tasks. In the GYM tasks (Figure 4, top row), IDQL and DIPO exhibit com- petitive performance, while the other methods often perform worse and train less stably. DPPO is the strongest performer the ROBOMIMIC tasks (Figure bottom row), especially in the chal- lenging T Transport tasks. Surprisingly, DRWR, are strong baselines in [Li Et, Can, Square) but underperforms Transport, while all other baselines fare worse still. We postulate that the other baselines, using off-policy updates and propagating biased gradients from the Q function to the actor, suffers from even greater training instability sparse-reward ROBOMIMIC tasks given continuous action space plus large action chunk sizes (see furtuer studies in Appendix D.2).
5.2 COMPARISON TO OTHER DEMO-AUGMENTED RL ALGORITHMS
We compare DPPO with recently proposed RI. methods for training robot policies (not necessarily iiffuusionbbssed leveraging offline data, including RLPD (Ball et al., 2023), Cal-QL (Nakamoto et al., 2024), and IBRL (Hu et al., 2023). These methods add expert data in the replay buffer and performs off-policy updates (IBRL and Cal-QL also do pre-training), which significantly improves efficiency v.s. DPPO in HalfChettah--0
However, in searse-reward manipulation tasks including Can and Square, DPPO achieves much better final performance than all three methods; RLPD and Cal-QL fail to learn at all and IBRL saturates at lower success levels. See Appendix D. containing Fig. 11, for further discussion.

---

Published as a conference paper at ICLR 2025
4000
Ho
4003
WolkerDDv2
5200
HalfCheetah--2
Evaluation reward
200D
3000
3000
2000
2003
1000
1000
1000
1e7
167
0.0
0.2
0.0
0.8
1.0
Lift- State
1.0
Can SSate
Square State
1.0
Transport State
Success Rate
9.6
0.6
0.4
0.4
0.4
0.4
82
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
Environment steps
DPPO
DRWR
DAWR
DIPO
IDQL
DQL
QSM
Figure 4: Comparing to other diffusion- based RL algorithms. Top row: GYM tasks (Brockman
al., 2016) averaged over five seeds. Bottow row: ROBOMIMIC tasks (Mandlekar al., 2021), av- eraged over three seeds with state observation. DPPO curves are slightly thicker for better visibility.
5.3 COMPARISON TO OTHER POLICY PARAMETERIZATIONS
We compare DPPO with popular RL policy parameterizations: unimodal Gaussian with diago- nal covariance (Sutton et al., 1999) and Gaussian Mixture Model (GMM (Bishop and Nasrabadi, 2006)), using either MLPs Transformers (Vaswani al., 2017), and also fine-tuned with the PPO objective. We compare these to DPPO-MLP and DPPO-UNG which use either MLP or UNet as the network backbone. We evaluate on the four tasks from ROBOMIMIC (Lift, Can, Square, Transport) with both state and pixel input. With state input, DPPO pre-trains with 20 denoising steps and then fine-tune tunes the last 10. With pixel input, DPPO pre-trains with 100 denoising steps and then fine-tunes DDIM steps.
1.0
Square State
Transport State
1.0
Square Pirel
1.0
Transport- Pivel
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.0
1s5
165
0.0
0.4
0.8
1.2
1.6
2,0
0.0
0.8
1.6
322
4.0
Environment steps
OPPO-MLP
Gaussian-MLLP Grussian-Tennsfrmme
DPPO-VIT-MLP OPT--VIT-JNet
Goussian-ViT-MLL
DPPO-UNet
GMM- Tran-Tranformeer
Figure 5: Comparing to other policy parameterizatioon in the more challenging Square and Transport tasks from ROBOMIMIC, with state (left) or pixel (right) observation. Results are averaged over three seeds.
defer the results in Lift and Can to Fig. 20. With state input, DPPO outperforms Gaussian and GMM policies, with faster convergence to 100% success rate in Lift and Can, and greater final performance on Square and the challenging Transport, where it reaches > 90%. UNet and MLP variants perform similarly, with the latter training somewhat more rapidly. With pixel inputs, we use Vision-Traan Transformer-basseeeeeer and compare the resulting variants OPPO-ViT-MLE and to poor performance in state-based training). While the two are comparable Lift and Can, DPPO trains more quickly and to higher accuracy on Square, and drastically outperforms on Transport, whereas Gaussian does not improve from its 0% pre-trained success rate. To our knowledge, DPPO is the first RL algorithm to solve Transport from either state or pixel input to high (>50%) success rates.
Fig. 5 display results for the more challenging Square and Transport
(ViT) image encoder introduced in Hu et al. (2023) and an MLP head
(we omit GMM due

---

Published as a conference paper at ICLR 2025
5.4
EVALUATION ON FURNITURE-BENCH AND SIM-TO-REAL TRANSFER
One-leg
Round-table
2.66
0.6
0.4
0.4
0.4
0.2
0.0
1eg
1
3
4
5
6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps
DPPO-UNet
Gaussian-MLE
Low randomness
Medium randomness
Pre-training only
After fine-tuning
1.0
1.0
0.87
0.8
0.0
Success
0.6
0.47
0.6
0.45
Successs
0.0
0.53
9.
0.
0.0
Real
Real
0.0
Diffusion
Gaussian
0.0
Diffusion (DPPO)
Gaussian
Gaussian w/ BC loss
Figure 6: (Top) DPPO vs. Gaussian-MBn baseline simulated FURNITURE-BENCH tasks. Re- sults are averaged over three seeds. (Bottom) Sim-to-real transfer results in One-leg.
Here we evaluate DPPO on the long-horizon manipulation tasks from FURNITURE- BENCH (Heo et al., 2023). We compare DPPO Gaussian-MLP, the overall most effective baseline from Sec- tion 5.3. Fig. 6 (top row) shows the evaluation success rate over fine-tuning iterations. DPPO exhibits strong training stability and improves policy performance in all six settings. Gaussian-MLL collapses to zero success rate in three tasks with Med randomness (except for one seed Lamp) and Round-table with Low randomness. Note that for pre-training; we expect DPPO can leverage additional human data (better state space coverage) to further improve in Med, which is corroborated by ablation studies in Appendix D)3.
only using 50 human demonstrations
Sim-to-real transfer. We evaluate DPPO and Gaussian policies trained in the simulated One-leg task physical hardware zero-shot (i.e., no real data fine-tuning / co-training) over 20 trials. Please see additional simulation training and hardware details in Appendix F.8. Fig. 6 (bottom row) shows simulated and hardware success rates after pre-training and fine-tuning, Notably, DPPO improves the real-world performance to 80% (16 out of 20 trials). Though the Gaussian policy achieves a high success rate in simulation after fine-tuning (88%). it fails entirely on hardware (0%). Supplemental video suggests it exhibits volatile and jittery behavior. For stronger comparison, we also fine-tune the Gaussian policy with an auxiliary behavior-cloning loss (Torne et al., 2024) such that the fine-tuned policy is encouraged to stay close to the base policy. However, this limits fine- tuning and only leads to 53% success rate in simulation and 50% reality. Qualitatively, we find fine-tuned policies to be more robust and exhibit more corrective behaviors than pre-trained-only policies, especially during the insertion stage of the task; such behaviors are visualized in Fig. 24 with representative hardware rollouts.
5.5 SUMMARY OF ABLATION FINDINGS
Our ablation studies (c.f. Appendix D.2) find that: (1) for challenging tasks, using value estimator which depends on environment state but independent of denoised action is crucial for perfor- mance; we conjecture that this is related to the high stochasticity of Diffusion Policy; (2) there is sweet spot for clipping the denoising noise level for DPPO exploration, trading off between too little exploration and too much action noise; (3) DPPO is resilient to fine-tuning fewer-than-K denoising steps, yielding improved runtime and comparable performance; (4) DPPO yields improvements over Caussian-MLL basclines for varying levels of expert demonstration data, and achieves comparable final performance and sample efficiency when training from scratch in GYM environments.
UNDERSTANDING THE PERFORMANCE OF DPPO
The improvement of DPPO over popular Gaussian and GMM methods in Section 5.3 comes as a surprise initially DPPO solves much longer Diffusion Policy MDP (Section than the origi-
8

---

Published as a conference paper at ICLR 2025
nal environment MDP that other methods solve. This leads study the factors contributing to DPPO's strong performance through a series of investigate experiments below.
We use the Avoid environment from D3IL benchmark (Jia al., 2024), where a robot arm needs to reach the other side of the table while avoiding an array of obstacles (Fig. 7, top-lcft). The action space is the 2D target location of the end-effector. D3IL provides set of expert demonstrations that covers different possible paths to the goal line we consider three subsets of the demonstrations, MI, M2, and M3 in Fig. 7, each with two distinct modes. We pre-train MLP-based Diffusion, Gaussian, and GMM policies (T, = 4 unless noted) with the demonstrations. For fine-tuning, we assign (sparse) reward when the robot reaches the goal line from the topmost mode. Gaussian and GMM policies are also fine-tuned with the PPO objective.
Avoid Environment
Pre-training data
Training trajectories the beginning of fine-tuning GMM
DPPO
Gaussian
M2
Task: Reach goal line from topmost mode
M3
Figure 7: (Left) We use the Avoid environment from Jia et al. (2024) to visualize the DPPO's exploration tendencies. The task is to reach the green goal line from the topmost mode. (Right) Structured exploration. We show sampled trajectories the first iteration of fine-tuning for DPPO, Gaussian, and GMM after pre-training on three sets of expert demonstrations, M1, M2, and M3.
Structured, on-manifold exploration. Fig. 7 (right) shows the sampled trajectories (with exploration noise) from DPPO, Gaussian, and GMM during the first iteration of fine-tuning. DPPO explores in wide coverage around the expert data manifold, whereas Gaussian generates less structured exploration noise (especially in M2) and GMM exhibits narrower coverage. Unlike Gaussian policy that adds noise only to the final sampled action, diffusion adds multiple rounds of noise through denoising. Each denoising step expands the coverage with new noise while also pushing the newly denoised action towards the expert data manifold (Permenter and Yuan, 2023). Moreover, the combination of diffusion parameterization with the denoising of action chunks means that policy stochasticity in DPPO is structured in both action dimension and time horizon.
Benefit 1:
Training stability from multi-step denoising process. In Fig. 8 (left), tuning after pre- trainini with M2 and attempt to de-stabilize fine-tuning by gradually adding noise to the action during the fine-tuning process (see Appendix F.9 for details). We find that Gaussian and GMM's performance both collapse, while with DPPO, the performance is robust to the noise if at least four denoising steps are used. This property also allows DPPO to apply significant noise to the sampled actions, simulating an imperfect low-level controller to facilitate sim-to-real transfer in Section 5.4. In Fig. (right), we also find DPPO enjoys greater training stability when fine-tuning long action chunks, e.g., up to To = 16, while Gaussian and GMM can fail to improve at all.
Benefit 2:
fine-
Fig. visualizes how DPPO affects the multi-step denoising process. Over fine-tuning iterations, the action distribution gradually converges through the denoising steps the iterative refinement largely preserved, as opposed to, e.g., "collapsing" to the optimal actions at the first fine-tuned denoising step or the final one. We postulate this contributes to the training stability of DPPO.
Benefit 3: Robust and generalizable fine-tuned policy. DPPO also generates final policies robust to perturbations in dynamics and the initial state distribution. In Fig. 10, we again add noise to the actions sampled from the fine-tuned policy (no noise applied during training) and find that DPPO policy exhibits strong robustness the noise compared to the Gaussian policy. DPPO policy also converges to the Joptimal path from a larger distribution of initial states. This finding echoes theoretical guarantees that Diffusion Policy, capable of representing complex multi-modal

---

Published as a conference paper at ICLR 2025
100
M2
M2
DPPO
K=2
Gaussian
DPPO, K-3
Reward
GMM
DPPG, X-4
Ta-4
Gaussian GMM
T,=0
Noise level
T,=16
165
8.0
1.5
0.0
0.5
1.0
2.0
2.5
Environment stapa
Environment steps
Figure 8: Training stability. Fine-tuning performance (averaged over five seeds, standard deviation shown) after pre-training with M2. (Left) Noise is injected into the applied actions after a few training iterations. (Right) The action chunk size Ta is varied.
Denoising steps
DPPO
K-O
Gaussian
GMM
Fine-tuning Iterations
Figure 9: Preserving the iterative refinement. The 2D actions from 50 trajectories at the branching point through fine-tuning iterations after pre-training with M2. For DPPO, we also visualize the action distribution through the final denoising steps each fine-tuning iteration.
data distribution, can effectively deconvolve noise from noisy states (Block al., 2024), property used Chen et al. (2024) to stabilize long-horizon video generation.
DPPO
trajectories
w/ initialization
perturbation
Figure 10: Policy robustness after fine-tuning. Green dot / box indicates the initial state region.
7 CONCLUSION AND FUTURE WORK
We believe DPPO will become important component in the sewtiiiing-luu-fineee for training general-purpose real-world robotic policies. To this end, we hope in future work to further showease the promise of DPPO for simulation-to-reat transfer (Chen al., 2023b: Liang et al., 2020; Ren et al., 2023; Chi et al., 2024a) in which we fine-tune a vision-based policy that has been pre-trained and diverse expert data manifold, of which, as we have shown in Section 6, DPPO is well-suited take advantage for better exploration during fine-tuning We are also excited understand how DPPO can fit together with other decision decision-making tools such as model-based planning (Janner et al., 2022; Ding et al., 2024) and secision-makinn aided by video prediction (Chen al., 2024); these tools may help address the main limitation of DPPO its lower sample efficiency than off-policy methods and unlocking performing practical RL in physical hardware.
pipeline
variety of diverse tasks. We expect this pre-training provide large
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Lirui Wang and Terry Suh for helpful discussions in the early stage of the project. The authors were partially supported by the Toyota Research Institute (TRI). This article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.
REFERENCES
J. Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
A. Ajay, Y. Du, A. Gupta, B. Tenenbaum, T. S. Jaakkola, and P. Agrawal. Is conditional gener- ative modeling all you need for decision making? The Eleventh International Conference on Learning Representations, 2023.
M. Alakuijala, Dulac-Aroodd, Mairal, Ponce, and C. Schmid. Residual reinforcement learn- ing from demonstrations. arXiv preprint arXiv:2 08050, 2021.
O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.
Ankile, Simeonov, Shenfeld, and Agrawal. Juicer: Data-efficient imitation learning for robotic assembly. arXiv, 2024a.
L. Ankile, A. Simeonov, I. Shenfeld, M. Torne, and P. Agrawal. From imitation refinement- residual for precise visual assembly, arXiv preprint (xXiv:2407.16677 2024b.
P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. International Conference on Machine Leurning, pages 1577-1594. PMLR, 2023.
C. M. Bishop and M. Nasrabadi. Pattern recognition and machine learning. Springer, 2006.
K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforce- ment learning. arXiv preprint arXiv:2305. 13300 2023.
A. Block, A. Jadbabaie, D. Pfrommer, M. Simchowitz, and R. Tedrake. Provable guarantees for generative behavior cloning: Bridging low-level stability and high-level behavior. Advances in Neural Information Processing Systems, 2024.
G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1606.60066.4444 2016.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances in neural informa- tion processing systems, 2020.
S. H. Chan. Tutorial on diffusion models for imaging and vision. arXiv preprint 11X1v:2403.18103. 2024.
B. Chen, D. M. Monso, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint ar arXiv:2407.01392 2024.
H. Chen, C. Lu, C. Ying, H. Su, and Zhu. Offline reinforcement learning via high-fidelity genera- tive behavior modeling. arXiv preprint arXiv:2209.144 14548, 2022a.
T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning, 2022b.
Y. Chen, H. Li, and D. Zhao. Boosting continuous control with consistency policy. arXiv preprint 1rXiv:2310.006343, 2023a.
Y. Chen, C. Wang. L. Fei-Fei, and C. K. Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. arXiy preprint XXXi:2309.00987 2023b.

---

Published as a conference paper at ICLR 2025
C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy: for goal- conditioned dynamic manipulation of deformable objects. The International Journal of Robotics Research, 2024a.
C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robofics Research, 2024b.
K. Clark, Vicol, K. Swersky, and D. J. Fleet. Directly fine-tuning diffusion models on differen- tiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
Z. Ding and Jin. Consistency models rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309./6920 2023.
Ding, A. Zhang, Tian, and Q. Zheng. Diffusion world model. arXiv preprint 0 2024.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, Janoos, L. Rudolph, and A. Madry. Implemen- tation matters in deep rl: A case study ppo and trpo. In International conference on learning representations, 2019.
Y. Fan and K. Lee. Optimizing ddpm sampling with shortcut fine-tuning. arXiv preprint 1rXiv:2301.13322 2023.
Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 2024.
P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 2019.
P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning. PMLR, 2022.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven rein- forcement learning. arXiv preprint arXiv:2004.07219 2020.
Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02111 2024.
W. Goo and S. Niekum. Know your boundaries: The necessity of explicit behavioral cloning in offline rl. arXiv preprint arXiv:2206.0060 00695,, 2022.
T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01499, 2023.
P. Hansen-Estrudd I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit leaannnng as an actor-critic method with diffusion policies. arXiv preprint urXiv:2304. 10573, 2023.
M. Heo, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. arXiv preprint arXiv:2305. /282/, 2023.
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris. I. Osband, et al. Deep q-learning from demonstrations, In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural infor- mation processing systems, 2020.
12

---

Published as a conference paper at ICLR 2025
J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023303 2022.
H. Hu, S. Mirchandani, and D. Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint (rXiv:2311.02/990 2023.
Huang, R. Dossa, C. Ye, Braga, Chakraborty. Mehta, and G. AraAšjo. Cleanrl: High-quality single-fil file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 2022.
Z. Huang, L. Yang. X. Zhou, Z. Zhang, W. Zhang, X. Zheng, J. Chen, Y. Wang, C. Bin, and W. Yang. Protein-ligand interaction prior for binding-awaa aware 3d molecule diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Hwangbo, Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
M. T. Jackson, M. T. Matthews, C. Lu, B. Ellis, S. Whiteson, and Foerstrr. Policy- colicy-guided diffusion. arXiv preprint A.Xi::24040063555 2024.
M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 09990, 2022.
X. Jia, D. Blessing, X. Jiang, M. Reuss, A. Donat, R. Lioutikov, and G. Neumann. Towards di- verse behaviors: benchmark for imitation learning with human demonstrations. arXiv preprint arXiv:2402. 14606, 2024.
B. Kang, X. Ma, C. Du, T. Pang, and S. Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 2024.
E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion- level drone racing using deep reinforcement learning. Nature, 2023.
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976. 2020.
S. Lee, Y. Seo, K. Lee, P. Abbeel, and Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic geensemble. In Conference on Robot Learning, pages 1702-1712, PMLR, 2022.
Lee, Y. Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03188 2024.
K. Lei, 7. He, C. Lu, K. Hu, Y. Gao, and H. Xu. Uni-o4: Unifying online and offline deep inforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:2311.0335 2023.
J. Liang, S. Saxena, and O. Kroemer. Learning active aoik-orieteed exploration policies for bridging the sim-to-real gap. arXiv preprint arXiv:2006.01952 2020.
P. Lillicrap, Hunt, A. Pritzel, N. Heess, Frez, Tassa, D. Silver, and Wierstra. Contin- uous control with deep reinforcement learning. arXiv preprint 1XXx:1500000991, 2015.
Y. Lin. A. S. Wang, G. Sutanto, A. Rai, and E. Meier. Polymetis. https://
facebookresserde git..b..tt....... woobbioo/fate hel/fair/// ro/
hooppolooooooo pol ymet s/, 2021.
A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. stat, 2024.
J. Luo, Z. Hu, C. Xu, Y. L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine. Serl: A software suite for sample-efficient robotic reinforcement learning. arXiv preprint arXiv:2401.160) 16013, 2024.
13

---

Published as a conference paper at ICLR 2025
S. Luo, Y. Su, X. Peng. S. Wang, J. Peng, and J. Ma. Antigen-specific antibody design and op- timization with diffusion-basee generative models for protein structures. Advances in Neural Information Processing Systems, 2022.
V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
A. Mandlekar, D. Xu, J. Wong, Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint wwXiv:2108.03200 2021.
A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093: 2020.
M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Ma, C. Finn, A. Kumar, and Levine. Cal-qi: Calibrated offlin rl pre-training for efficient online fine-tuning, Advances in Neural Information Processing Systems, 36, 2024.
A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, 2021.
T. Osa, Pajarinen, G. Neumann, J. Bagnell, Abbeel, Peters, et al. An algorithmic perspec- tive on imitation learning. Foundations and Trends@) in Robotics, 2018.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 2022.
T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.
2023.
X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-polic reinforcement learning, arXiv preprint arXiv:1910.00117, 2019.
X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 2021.
F. Permenter and C. Yuan. Interpreting and improving diffusion models from an optimization per- spective. arXiv preprint arXiv:2306.04844 2023.
Peters and S. Schaal. Reinforcement learning by weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning, 2007.
D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1988.
B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1449 2022.
M. Popova, O. Isayev, and A. Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 2018.
M. Psenka, A. Escontrela, Abbeel, and Y. Ma. Learning a diffusion model policy from rewards via q-score matching, arXiv preprint arXiv:2312.11752. 2023.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In Inter- national conference machine learning, 2021.
A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
14

---

Published as a conference paper at ICLR 2025
A. Z. Ren, H. Dai, B. Burchfiel, and A. Majumdar. Adaptsim: Task-driven simulation adaptation for sim-to-real transfer. In Proceedings of the Conference on Robot Learning (CoRL), 2023.
M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprini arXiv:2304.025332 2023.
M. Rigter, J. Yamada, and I. Posner. World models via policy-guided trajectory diffusion. arXiv preprint arXiv:2312.08533, 2023.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolutio image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and conputer-assisted intervention (MICCAI), 2015.
N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2023.
S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.00 07524, 2024.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 02438, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint N.X.vv1770.0634,,000 2017.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, 2015.
J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Y. Song, J. Sohl-Dickstein D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based genera- tive modeling through stochastic differential equations. arXiv preprint M.Xiv:2011.134555 2020b.
A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navi- gation and exploration. arXiv preprint arXiv:2310.07896, 2023.
R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 1999.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model- control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, 2012.
M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and Agrawal. Reconciling real- ity through simulation: A real-to-sim-ttoreal approach for robust manipulation. arXiv preprint arXiv:2403.039.0 03949, 2024.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is you need. Advances in neural information processing systems, 2017.
M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.0880 08817, 2017.
Venkatraman, S. Khaitan, R. T. Akella, J. Dolan, Schneider, and G. Berseth. Reasoning with latent diffusion in offline reinforcement learning. arXiv preprint arXiv:2309.06653 2023.
15

---

Published as a conference paper at ICLR 2025
B. Wallace, M. Dang. R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong. S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12008, 2023.
Wang and E. Olson. Apriltag Efficient and robust fiducial detection. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016.
L. Wang, J. Zhao, Y. Du, E. H. Adelson, and R. Tedrake. Poco: Policy composition from and for heterogeneous robot learning. arXiv preprint arXiv:2402.02511, 2024a.
Wang. Q. Yang. J. Gao, M. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024b.
Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208. 06193, 2022.
R. Williams. Simple statistical gradient-followigg algorithms for connectionist reinforcement learning. Machine learning, 1992.
J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn. Robot fine-tuning made easy: Pre- training rewards and policies for autonomous real-world reinforcement learning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 4804 4811. IEEE, 2024.
L. Yang, Z. Huang, Lei, Y. Zhong, Y. Yang, C. Fang, S. Wen, B. Zhou, and Z. Lin. Pol- icy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122 2023.
Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation.
H. Zhang, W. Xu, and H. Yu. Policy expansion for bridging offline-to-onlin reinforcement learning. arXiv preprint 3320:2230..00935, 2023.
Z. Zhao, Kumar, Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
Z. Zhu, H. Zhao, H. He, Y. Zhong, S. Zhang, Y. Yu, and W. Zhang. Diffusion models for reinforce- ment learning: A survey, arXiv preprint arXiv:2311.01225 2023.
16

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
Related Work
3 Preliminaries
4
DPPO: Diffusion Policy Policy Optimization
4.1
Instantiating DPPO with Proximal Policy Optimization
5
Performance Evaluation of DPPO
5.1
Comparison to diffusion-basse RL algorithms
5.2
Comparison other demo-augmented RL algorithms
6
5.3
Comparison other policy parameterizations
5.4
Evaluation on Furniture-Bende and sim-to-real transfer
8
5.5
Summary of ablation findings
6
Understanding the performance of DPPO
7
Conclusion and Future Work
10
Extended Related Work
18
A.1 RL training of robot policies with offline data
18
A.2 Diffusion-based RL methods
18
B Best Practices for DPPO
20
C Additional details of DPPO implementation
20
Additional experimental results
22
D.I
Comparing to other demo-augmented RL methods
22
D.2 Ablation studies on design decisions DPPO
D.3 Effect of expert data
25
D.4
Comparing to other policy parameterizations in Avoid
26
D.5 Comparing to other policy parameterizations in the easier tasks from ROBOMIMIC
27
D.6 Comparing to policy gradient using exact likelihood of Diffusion Policy
27
D.7 Ablating Structured Exploration in DPPO
27
Reporting Wall-Clock Times
28
Additional Experimental Details
30
F1 Details of policy architectures used in all experiments
30
F2 Additional details of GYM tasks and training in Section 5.1
31
17

---

Published as a conference paper at ICLR 2025
F3
Descriptions of ceffusion-bassee RL algorithm baselines in Section 5.1
31
F.4
Descriptions of RL fine-tuning algorithm baselines in Section 5.2
32
E.5
Additional details of FRANKA-KITCHEN tasks and training in Section 5.2
33
F6
Additional details of ROBOMIMIC tasks and training in Section 5.3
34
F7
Descriptions of policy parameterization baselines in Section 5.3
34
F8
Additional details of FURNITURE-BENCH tasks and training in Section 5.4
34
F9
Additional details of Avoid d task from D3IL and training in Section 6
38
F.10
Listed training hyperparameters
38
A
EXTENDED RELATED WORK
A.1 RL TRAINING ROBOT POLICIES WITH OFFLINE DATA
Here, we discuss related work in training robot policies using RL augmented with offline data to help RL. better explore online in sparse reward settings.
One simple form is to use offline data to pre-train the policy, typically using behavior cloning, and then fine-tune the policy online. This is the approach that DPPO takes. Often, regularization loss is applied to constrain the fine-tuned policy to stay close to the base policy, leading to natural fine-tuned behavior and often better learning (Rajeswaran et al., 2017; Zhu et al., 2019; Torne et al., we find the on-manifold exploration
2024). DPPO does not apply regularization at fine-tuning helps DPPO maintain natural behavior after fine-tuning Section 5.4. Another popular approach is to learn residual policy with RL on top of the frozen base policy (Alakuijala al., 2021; Haldar et al., 2023). closer work to ours is Ankile et al. (2024b), which trains a one-step residual non- diffusion policy with on-policy RL on top of pre-trained chunked diffusion policy. This approach has the benefit of being fully closed-loop but lacks the structured on-manifold exploration of DPPO. Another hybrid approach from Hu al. (2023), which uses pre-traine pre-trained and fine-tuned policies to sample online experiences.
Another popular line of work, instead of training a base policy using offline data, directly adds the data in the replay buffer for online, off-policy learning in single stage (Hester et al., 2018; Vecerik al., 2017; Nair et al., 2020). One recent approach from Ball al. (2023), RLPD, further improves sample efficiency from previous off- off-policy methods incorporating, e.g., critic ensembling. Luo et al. (2024) demonstrates RLPD solving real-world manipulation tasks (although generally less challenging than ones solved by DPPO).
Other approaches, including Cal-QL, build on offline RL learn from offline data and then switch to online RL while still sampling from offline data (Nakamoto et al., 2024; Jansen-Esttruch et al., 2023; Yang et al., 2024). Often the distributional mismatch between offline data and online policy needs to be addressed: Cal-QL proposes calibrated conservative Q-learning that learns offline Q function that lower bounds the true value the learned policy; Lei et al. (2023) proposes ensemble behavior cloning during pre-training promote policy diversity (similarly in Wang al. (2024b)); Lee et al. (2022) proposes prioritizing online samples and then near-on-policy samples from the offline dataset during fine-tuning: Zhang al. (2023) proposes similar method akin to Hu et al. (2023) that uses both pre-trained and online policies for collecting new samples.
A.2 DIFFUSION-BASE RL. METHODS
This section discusses related methods that directly train improve diffusion-basee policies with RL methods. The baselines to which we compare in Section 5.1 are discussed below as well, and highlighted in their corresponding colors. We also refer the readers Zhu et al. (2023) for an extensive survey on diffusion models for RL.
Most previous works have focused on the offline setting with static dataset. One line of work fo- cuses on state trajectory planning and guiding the denoising sampling process such that the sampled actions satisfy some desired objectives. Janner al. (2022) applies classifier guidance that generates
18

---

Published as a conference paper at ICLR 2025
trajectories with higher predicted rewards. Ajay et al. (2023) introduces flassffierffree guidance that avoids learning the value of noisy states. There is another line of work that uses diffusion models as action policy (instead of state planner) and generally applies Q-learning. DQL (Wang et al., 2022) introduces Diffusion Q-Learning that learns a state-action critic for the final denoised actions and backpropagates the gradient from the critic through the entire Diffusion Policy (actor) denois- ing chain, akin to the usual Q-learning IDQL THansen-Estruce et al., 2023), or Implicit Diffusion Q-learning, proposes learning the critic select the actions at inference time for either training or evaluation while fitting the actor to all sampled actions. Kang et al. (2024) instead proposes using the critic to re- weight the sampled actions for updating the actor itself, similar to weighted regression baselines DAWR and DRWR introduced in our work. Goo and Niekum (2022) similarly extracts the policy the spirit of AWR (Peng et al., 2019). Chen et al. (2022a) trains the critic using value itera- tion instead based on samples from the actor. Finally, Jackson et al. (2024) explores using diffusion guidance to move offline data towards the target trajectory distribution.
We note that methods like DQL and IDQL can also be applied in the online setting. A small amount of work also focuses entirely on the online setting. DIPO (Yang et al., 2023) differs from DQL and related work in that it uses the critie to update the sampled actions ("action gradient") instead of the actor - the actor is then fitted with updated actions from the replay buffer. QSM, or Q-Score Matching (Psenka al., 2023), suggests that optimizing the likelihood of the entire chain of denoised actions can be inefficient (contrary to our findings in the fine-tuning setting) and instead proposes learning the optimal policy by iteratively aligning the gradient of the actor (i.e., score) with the action gradient of the critic. Rigter et al. (2023) proposes learning a diffusion dynamic model to generate synthetic trajectories for online training of a non-diffusion RL policy.
We note that almost all prior work in diffusion-based RL (offline or online) have relied on approxi- mating the action Q function and using to update the diffusion actor in some form - policy gradient update has been deemed challenging due to the multi step denoising process (Psenka et al., 2023; Yang et al., 2023). Inaccurate Q values may lead to biased updates to the actor, which can lead to training collapse as it starts with decent pre-training performance but quickly drops to zero success rate as seen in Fig. 4, also failing to recover since then due to the sparse-reward setup. While Q-learning methods generally achieve better sample efficiency when they can solve the task of in- terest, our focus has been largely on challenging long-horizon robot manipulation tasks where the training stability is much desired.
Lastly, we point out that the role of stochasticity of diffussion/consistennyy policy for exploration is also explored by Chen et al. (2023a), who find such stochasticity suffices for exploration without additional strategies needed. Our work discovers the similar effect, but also performs extensive investigative experiments in Section 6 and ablation studies in Appendix D.7 provide affirmative evidence for the distinct exploration strategies induced by diffusion.
Distinction from the policy gradient formulation in Psenka et al. (2023). There has been a different formulation introduced in Psenka et al. (2023) Sec. 3 that derives the policy gradient up- date for diffusion policy. The derivation is based on converting the gradient of the log likelihood of the final denoised action to the sum over log likelihood of individual denoising actions. This formulation, unlike DPPO, does not treat the multi-step denoising process as a MDP. In the policy gradient update, Psenka et al. (2023) sums over denoising steps and then takes expectation over environment steps. while DPPO's 's update (4.2) takes expectation over both denoising and environ- ment steps, potentially leading to better sample efficiency. Moreover, Psenka et al. (2023) does not propose leveraging PPO updates or other modifications to diffusion, and finds such vanilla form of policy gradient update to be ineffective. We formulate DPPO independent of their work and find DPPO highly efffcctive in fine-tuning settings while also being competitive in training from scratch (Appendix D.3).
Potential impact beyond robotics. DPPO is a generic framework that can be potentially applied to fine-tuning diffusion- based models in sequential interactive settings beyond robotics. These in- clude: extending diffusion iiffusion-bbsed text-to-image generation (Black et al., 2023; Clark et al., 2023) to multi-turn interactive setting with human feedback; drug design/discovery applications (Luo et al., 2022: Huang et al., 2024) with policy search on the molecular level in feedback with simulators (in the spirit of prior non-diffusion-basse drug discovery with RL (Popova et al., 2018)); and the
19

---

Published as a conference paper at ICLR 2025
adaptation of diffusion- based language modeling (Sahoo et al., 2024; Lou et al., 2024) to interactive (e.g. with human feedback (Ouyang et al., 2022)), problem-solving and planning tasks.
B BEST PRACTICES FOR DPPO
Fine-tune only the last few denoising steps. Diffusion Policy often uses up to K = 100 denoising steps with DDPM better capture the complex data distribution of expert demonstrations. With DPPO. we can choose to fine-tune only a subset of the denoising steps instead, e.g., the last K' steps. In Section 4.1 and Section 6 we find this speeds up DPPO training and reduces GPU memory usage without sacrificing the asymptotic performance. Instead of fine-tuning the pre-trained model weights 8, we make copy O.T - is frozen and used for the early denoising steps, while fhr is used for the last K' steps and updated with DPPO. Although maintaining the frozen 0 takes extra GPU memory (e.g., about 60MB in Ha fCheetah v2), we find that storing the extra intermediate denoised actions and their likelihoods due to higher K' can taking significantly more GPU memory (e.g.. 1.IGB extra if K' = 20 instead 10 in Hal fCheetah-v2).
Fine-tune DDIM. Instead of fine-tuning all K or the last few steps of the DDPM, one can also apply Denoising Diffusion Implicit Model (DDIM) (Song et al., 2020a) during fine-tuning, which greatly reduces the number of sampling steps K DDDIM < K, c.g., few as 5 steps, and thus poten- tially improves DPPO efficiency as fewer steps are fine-tuned.
k- ~ Pg
i DDIM (a (14) == NNaa-1 DDIM (a" ca(a", k)), no k: K-DDIM ....,.. (B.1)
Although DDIM is typically used as a deterministic sampler by setting 7) O in (B.1), we can use ? > for fine-tuning that provides exploration noise and avoids calculating Gaussian likelihood with a Dirac distribution. In practice, we set = for training (equivalent to applying DDPM Song et al. (2020a)) and then 7 = for evaluation. We reserve DDIM sampling for our pixel-based experiments and Long-horizon furniture assembly tasks, where the efficiency improvements are much desired.
Diffusion noise scheduling. We use the cosine schedule for introduced in Nichol and Dhariwal (2021), which was originally annealed to a small value on the order of 1 at 0. DPPO, the value of Fs also translates to the exploration noise that is crucial to training efficiency. Empirically, we find that clipping 0% to a higher minimum value (denoted exp mint c.g., 0.01 - 0.1) when sampling actions helps exploration (see sensitivity analysis in Appendix D.2). Additionally we clip σ to be least 0.1 (denoted prbb) when evaluating the Gaussian likelihood log 7o (ūg) which improves training stability by avoiding large magnitude.
Network architecture. We study both Multi-layer Perceptron (MLP) and UNet (Ronneberger et al., 2015) as the policy heads in Diffusion Policy. An MIP offers simpler setup and we find it generally fine-tunes more stably with DPPO. Moreover, since the UNet applies convolution to the denoised action, we can pre-train and fine-tune with different action chunk size T₃ (the number of environment timesteps that the policy predicts future actions with), e.g., 16 and 8. We find that DPPO benefits from pre-training with larger Ta (better prediction) and fine-tuning with smaller Ta (more amenable to policy gradient)*.
C ADDITIONAL DETAILS OF DRPO IMPLEMENTATION
Pseudocode. The pseudocode for DPPO is presented in Algorithm DPPO takes as input a diffusion policy ny trained using behavior cloning loss LBC. The policy is then fine-tuned using PPO-style loss (Schulman et al., 2017) with careful treatment of the denoising process (Section 4).
With fully-connected layers in MI., empirically we find that using different chunk sizes for pre-training and fine-tuning with ML. leads to training instability.
20

---

Published as a conference paper at ICLR 2025
Pre-training.
The diffusion policy ne is pre-trained using behavior cloning loss (Ho al., 2020):
LBC (t) = -(t || Eg(a, st. k)]l
(C.1)
where Doff is the offline dataset and Ee is the policy network predicting the sampled noise added to dg based on the noisy action. We use the cosine noise schedule from Nichol and Dhariwal (2021).
Algorithm DPPO
1: Pre-train diffusion policy To with offline dataset Dorr using BC loss (Bc Eq. (C.1).
2: Initialize value function Vg.
3: for iteration = 2, ... do
4: Initialize rollout buffer Dür-
5: Tea Tg.
6: for environment 1, 2, .... N in parallel do
7:
Initialize state {t(o.K) (so, a ) in MDP. for environment step = ..... T, denoising step h: K - ,,..., do Sample the next denoised action 0ī(t.K) of T&
8:
9:
10:
if k = O then
Run of the environment and observe Re(e,o) and S1(+11K)-
12:
else
13:
Set Ri(A,) = O and Bi(t,-11) = (st, az).
14:
Add (k, ()), (i((.))) R)((,A) to Dar- Compute advantage estimates A (sT(1,8=0), (z(), for Dar using GAE Eq. (C.2). for update = 2. num_ _ammt_tttteeeeee do for minibatch = 1, 2. .... B do
15:
Based on replay ratio No
17:
18:
Sample (A: ()(e.)), k.(,.)) R((,)) and A (SK(,)) ((())) from Dir- Compute denoising-discounted advantage AR(.)) A "0 (37(1,0) oz(t,0))
19:
20:
Optimize Ty using policy gradient loss Cg Eq. (C.3). Optimize V₂ using value loss Lg Eq. (C.4).
21:
22:
return converged policy m8.
Environment-step advantage estimation. We use Generalized Advantage Estimation (GAB) (Schulman et al., 2015) with parameter À for advantage estimation in Algorithm 1. GAE-λ ap- proximates the advantage function using the series
A A E ((1)) where de((,)) RE(e,)) + YENv V (5((e(111)5)))
Vo (SE())))
(C.2) RiCM + JENY Vo(5 Vo( and the Monte Carlo return of the episode relative to the baseline (A)=1 Et's Bit+LA) V ()((.,)) We refer the reader to Table 8 for additional details on GAE parameter selection and Section D.2 for ablations the choice of advantage estimator.
Notably, GAE-A interpolates between a one-step temporal difference (A^=0
Note that in Eq. (C.2) we are only concerned with = 0, i.c., the final denoising step. DPPO formulation, only need calculate the advantage for k 0 (i.e., environment steps), but not for intermediate denoising steps. We only need to apply denoising discounting to the calculated advantages so they can be applied to each denoising step k.
Fine-tuning.
During RL fine-tuning, we update the policy ma using the clipped objective:
To (5 2z)
3o (5 ai)
C9 = E Du min (a (5t, 0z)
A (3,, ā.) clip
- 1+)) (C.3)
Tam (5g āz)
Tea (3i, ai)
If choose to fine-tune only the last K' denoising steps, then sample only those from D
21

---

Published as a conference paper at ICLR 2025
Finally, we train the value function predict the future discounted sum of rewards (i.e., discounted returns):
T-
L E
Tex Rac+LK)) (=(
(C.4)
Similar to all baselines in Appendix F.3, we denote No and Ng & the replay ratio for the actor (diffusion policy) and the value critic in DPPO; in practice always set Ng Ng. Similar usual PPO implementations (Huang et al., 2022), the batch updates in an iteration terminate when the KL divergence between my and w0 reaches 1.
Large batch size. Since the gradient update in DPPO involves expectation over both environment steps and denoising steps, we use a larger batch size compared to, e.g., PPO training with Gaussian policy parameterization. Roughly of the fine-tuned denoising steps; in some cases like ROBOMIMIC we also observe that a much smaller batch size (close to that of Gaussian training) can be used and significantly improves sample efficiency.
use the batch size from Gaussian training times the number
Gradient clipping ratio. We also find the PPO clipping ratio, E, can affect the training stability significantly DPPO (as well in Gaussian and GMM policies) especially in pparse-rewand ma- nipulation tasks. In practice we find that, good indicator of the amount of clipping leading to opti- mal training efficiency, is to aim for clipping fraction (fraction of individual samples being clipped in batch) of 10% to 20%. For each method in different tasks, vary in {-1, .01, .001) and choose the highest value that satisfics the clipping fraction target. Empirically we also find that, us- ing a higher e for earlier denoising steps in DPPO further improves training stability in manipulation tasks. Denote ER the clipping value at denoising step k; and in practice we set E (==(K-1)) = 0.1Ek=0- and it follows exponential schedule among intermediate k.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 COMPARING TO OTHER DEMO-AUGMENTED RL METHODS
We compare DPPO with recently proposed RL methods for training robot policies (not necessarily diffusion-based) leveraging offline data, including RLPD (Ball et al., 2023), Cal-QL (Nakamoto et al., 2024), and IBRL (Hu et al., 2023). These methods add expert data in the replay buffer and performs off off-policy updates (IBRL and Cal-QL also pretrain with behavior cloning and of- fline RL objectives, respectively), which significantly improves sample efficiency vs. DPPO in Hal fCheetah-v2 (Fig. 11, top left).
Among the KRANKA-KITCHEN settings (Fig. 11, top right), we find RLPD and IBRL fail learn well especially with noisier demonstrations from Kitchen-Partial-- Kitchen Mi xed-v0. Cal-QL achieves competitive performance DPPO still achieves over- all the best performance especially with Kitchen www.chhhhnnhhaeeeeeeemmmeeoooeee e v0. We note that DPPO, us- ing any expert data during fine-tuning, can be sensitive to the pre-training performance; we find the incomplete demonstrations in Kitchen-Partial--
and
and Miich---Mi cause chal- lenge in fully modeling the multi-modality of the data even with diffusion parameterization and prevent DPPO from achieving mear-)perfect fine-tuning performance.
Nonetheless, we believe the expert demonstrations from ROBOMIMIC are most reflective of pre- training plus fine-tuning in robotics as all demonstrations complete the task despite the varying quality. Fig. 11 bottom row shows the performance of DPPO and baselines using either cleaner PH or noisier MH data in Can and Square; DPPO exhibits strong final performance. RLPD and Cal-QL fail to learn all. Although IBRL matches the success rates with DPPO in Square, find that it saturates at lower success levels while DPPO continues to reach 1000 success rates (not shown). DPPO also runs significantly faster in wall-clock time than the baselines as it leverages
22

---

Published as a conference paper at ICLR 2025
sampling from highly parallelized environments'; thus cap the number of samples 1e6 for the baselines in ROBOMIMIC, also since their performance saturates.
Evaluation reward
12000
HalfCheetah-v2
1.00
Kitchen-Completer/
1.00
KitchenPaariia-v0
1.00
Kitchen-Mixed-vug
0.75
0.75
0.75
Normalized
6000
Mm
9.50
3000
0.25
0.25
025
1ed
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Can State, PH
Square State,
Can
Square State
80.8
0.8
20.80
0.80
g0.60 1.60
0.60
0..
0.2
07.00
0.20
0.0
0.2
0.4
0.6
0.8
1eg
1.0
0.0
0.2
0.4
0.6
0.
1.0
-0.0
b.0
0.2
0.4
0.6
0.8
Ted
1.0
-0.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps
DPPO
RLPD
Cal-QL
IBRL
Comparing to other demo-augmented RL methods. Results are aver- aged over five seeds in HalfCheetah-v2 and three seeds in Ki Kitchen-Partinn -v0 Xitchen-Partte -v0, Can, and Square.
Figure 11:
We have also experimented with using diffusion policies for RLPD, Cal-QL, and IBRL, and the results are shown in Fig. 12. We use either action chunk size T, = or Ta 4. We see similar results as in Fig. using Gaussian policies that RLPD and Cal-QL fails to solve the task at all. We believe that the worse performance of Cal-QL is due to the offline RL objective (based on learning the state-action Q function) making learning precise continuous actions needed in ROBOMIMIC tasks very difficult, regardless of the policy parameterization, which corroborates original finding in Section 5.1 when comparing DPPO to A-Jearning-base diffusion RL methods. Compared to RLPD that trains with the SAC objective and expert data in the replay buffer, IBRL, using BC pre-training, is able to learn a base policy more effectively and uses it for online data collection. DPPO benefits from directly fine-tuning the pre-trained policy (instead of training new one using experiences from the pre-trained policy), and achieves similar or better sample efficiency before lеб steps compared IBRL, and converges to 100% success rates unlike IBRL saturates at lower levels (not shown).
Can,
State
Square, State
State
Square State
1.0
1.0
1.0
1.0
Success rate
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
1e6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8 1.0
0,0 0.2 0.4
0.6
0.8 1.0
0.0
0.2
0.4
0.8
1.0
Environment steps
DPPO, Ta=1
RLPD, Ta=1
Cal-QL, Ta=1
IBRL, Ta=1 IBRL, Ta=4
DPPO, Ta=4
RLPD, Ta=4
Cal-QL, Ta=4
Figure 12: Using diffusion policy for other demo-augmented RL methods. Results are averaged over three seeds.
'Off-policy methods (baselines) usually cannot fully leverage parallelized sampling as the policy is updated less often (c.g., 50 updates per 50 samples instead of update per 1 sample) and the performance can be affected.
23

---

Published as a conference paper at ICLR 2025
D.2 ABLATION STUDIES ON DESIGN DECISIONS IN DPPO
1. Choice of advantage estimator: In Section 4.1 we demonstrate how to efficiently estimate the advantage used in PPO updates by learning V(s)) that only depends on the environment state; the advantage used in DPPO formally
Å DENNISE (F(3,, @z) ((s))).
We now compare this choice with learning the value of the full state St(r.0) that includes environ- ment state 8; and denoised action af=1. We additionally compare with the state-action Q-function estimator used in Psenka al. (2023)6, Q(s.: af ), that does not directly the rollout reward in the advantage.
Fig. 13 shows the fine-tuning results in Hopper-v2 and Hal FCheetah-v2 from GYM, and Can and Square from ROBOMIMIC. On the simpler Hopper-v2 we observe that the two baselines, both estimating the value of some action, achieves higher reward during fine-tuning than DPPO's choice. However, in the more challenging tasks, the enviromment-state-
advantage used in DPPO consistently leads the most improved performance. We believe estimating the accurate value of applying a continuous and nigh-dimensional action can be challenging, and this is exac- erbated by the high stochasticity of diffusion based policies and the action chunk size. The results here corroborate the findings in Section 5.1 that off-policy Q-learning methods can perform well in Hopper v2 and Wa ke ker2D-v2, but often exhibit training instability in manipulation tasks from ROBOMIMIC.
Eveliation (ewaro
Hopper-v2
Doate
HallCheetahh-2
Square
2000
1000
Environment steps
Environment steps
Environment steps
Environment steps
(ts))
Vis..
Öts,
Figure 13: Choice of advantage estimator. Results are averaged over five seeds in Hopper-v2 and Ha #a1hCheeta and three seeds in Can and Square.
Denoising discount factor. We further examine how "DenoIse in the DPPO advantage estimator affects fine-tuning. Using a smaller value (i.e., more discount) has the effect of downweighting the contribution of earlier denoising steps in the policy gradient. Fig. 14 shows the fine-tuning results in the same four tasks with varying Torsoise E [0.5, 0.8, 0.9, 1]. We find Hopper-v2 and Hal @Cheethhhhhht Toenoise 0.8 leads to better efficiency while smaller "Desoise 0.5 slows training. The value does not affect training noticeably in Can. Square the smaller "DeNoise = 0.5 works slightly better. Overall in manipulation tasks, DPPO training seems relatively robust to
this choice.
Hopper-v2
HalfCheetah-v2 1,0
Can
Square
1.0
Rugluation reweard
Evelustionn reward
Sugoess gale
Success
3880
0.0
8.5
Iec
1.0
0.5
0.5
1.0
1.5
1ec
2.0
0.5
1.0
1.5
1ag
2.0
0.4
3
1ed
Environment steps
0.5
0.8
0.9
Figure 14: Choice of denoising discount factor. Results averaged five seeds in Hopper-v2 and fCheetah--2 and three seeds in Can and Scuare.
'Psenka et al. (2023) applies off-policy training with double Q-leaning (according to its open-source im- plementation) and policy gradient over the denoising steps. Note that this is baseline in Psenka et al. (2023) that conjectured be inefficient. We follow the same except for applying on-policy PPO updates.
24

---

Published as a conference paper at ICLR 2025
2. Choice of diffusion noise schedule. As introduced in Section 4.1, find it helpful to clip the diffusion noise de to a higher minimum value -00 to ensure sufficient exploration. In Figure 15, we min perform analysis varying on (.001, .1, -2 (keeping o miin .1 to evaluate likelihoods). Although in Can the choice of exp does not affect the fine-tuning performance, in Square exp higher min = 0.1 is required to prevent the policy from collapsing. We conjecture that this is due to limited exploration causing policy over-optimizing the collected samples that exhibit limited state- action coverage. We also visualize the trajectories at the beginning of fine-tuning in Avoid di task from D3IL. With higher mm. the trajectories still remain near the two modes of the pre-training data but exhibit a higher coverage in the state space we believe this additional coverage leads to better exploration. Anecdotally, we find terminating the denoising process early can also provide exploration noise and lead comparable results, but it requires more involved implementation around the denoising MDP.
Training trajectories at the beginning of fine-tuning
Can
Square
0.001
0.001
Syggess
0.1
Environment steps
Choice minimum diffusion noise. Results are averaged over three seeds. Note in Left, with higher minimum noise level, the sampled trajectories exhibit wider coverage the two modes but still maintain the overall structure.
Figure 15:
Choice the number fine-tuned denoising steps. We examine how the number of fine- tuned denoising steps in DPPO, K", affects the fine-tune performance and wall-clock time in Fig. 16. We show the curves of individual runs (three for each K') instead of the average as their wall-clock times (X-axis) are not perfectly aligned. Generally, fine-tuning too few denoising steps (e.g.. 3) can lead to subpar asymptotic performance and slower convergence especially in Can. Fine-tuning 10 steps leads to the overall best efficiency. Similar results also shown in Fig. 19 with Avoid task. Lastly, we note that the GPU memory usage scales linearly with K'.
We note that the findings here mostly correlate with those from varying the denoising discount factor, 'DEEOISEE Discounting the earlier denoising steps in the policy gradient can considered as soft version of hard limiting the number of fine-tuned denoising steps. Depending the amount of fine-tuning needed from the pre-trained action distribution, one can flexibly adjust ToeeNOse and K" achieve the best efficiency.
Hopper.22
HalfCheetah--2
Square
Eeelattion reward
Cilluation roware Evoll
2000
3300
Rapceett rate
SugcoRss Fate
3000
22000
Time (seconds)
K'
K'
K'=
K' =
Figure 16: Choice of number of fine-tuned denoising steps, K', Individual runs are shown. The curves are smoothed using a
filter.
D.3 EFFECT OF EXPERT DATA
We investigate the effect of the amount of pre-training expert data on fine-tuning performance. In Fig. 17 we compare DPPO and Gaussian in Hopper-v2, Square, and One- leg task from FUR- NITURE- -EECH, using varying numbers of expert data (episodes) denoted in the figure. Overall, find DPPO can better leverage the pre-training data and fine-tune high success rates. Notably, DPPO obtains non-trivial performance (60% success rate) on One-leg from only 10 episode of demonstrations.
25

---

Published as a conference paper at ICLR 2025
4032
Hopper-y2
Square
One-leg, Randomness
Evalustion reward
Exaluation reware
geluaaionn gowart
8000
1000
0.0
044
0.8
1.3
20
0.
Environment steps
Environment steps
Environment steps
OPPO-MLP. 0
MLL
MLP,
OPPO-MLR MLP, 10
Gaussian-MLP. 10
DP
Caunsien-MLF ML
OPPO-MLP. 20
Goussian MLP,
OPPO-MLP,
Gaussien-MPP 20
MPPD
Gaussian-MLFF 50
Gousstan-
OPPO-MLE P. 30
Gauesian-MLPP 30
DPPO-MIP, 200
Gaussian-MLPP 200
OPPO-MLP, 100
Gaussien-MLP. 100
OPPO-MLP,
Figure 17: Varying the number of expert demonstrations. The numbers in the legends indicates the number of episodes used in pre-training.
Training from scratch. In Fig. 18 we compare DPPO (10 denoising steps) and Gaussian trained from scratch (no pre-training on expert data) in the three OpenAI GYM tasks. As using larger action chunk sizes To leads to poor from-scratch training shown in Fig. 17, we focus on single-action chunks Ta 1 (and Tр 1) as is typical in RL benchmarking. Though we find Gaussian trains faster than DPPO (expected since DPPO solves MDP with longer effective horizon), DPPO still attains reasonable final performance. However, due to the multi-step (10) denoising sampling, DPPO takes about 6x wall-clock time compared Gaussian. We hope that future work will explore how to design the training curriculum of denoising steps for the best balance training performance and wall-clock efficiency.
Hopper-v2
Walker2D-v2
HalfCheetah-v2
Evaluation reward reward
4000
3000
4000
9000
2000
2000
1000
1000
1000
Environment steps
DPPO-MLP, T,-1
Gaussian-MLP, T,=1
Figure 18: No expert data / pre-training with GYM tasks. Results are averaged over five seeds.
D.4 COMPARING TO OTHER POLICY PARAMETERIZATTONS AVOID
Figure 19 depicts the performance of various parameterizations of DPPO (with differing numbers of fine-tuned denoising steps, K') to Gaussian and GMM baselines. We study the Avoid task from D3IL, after pre-training with the data from MI, M2, M3 described Section 6. We find that, for K' E {15, 20}. DPPO attains the highest performance of all methods and trains the quickest in terms of environment steps: on MI, M2, it appears to attain the greatest terminal performance as well. K' = 10 appears slightly better than, but roughly comparable to, the Gaussian baseline, with GMM and K' < 10 performing less strongly.
100
M1
100
M2
100
DP-0,K'-1
DPPO,K"3
DPPO, K'x5
Reward
3PPO,K'=10
DPP0,K'15
DPPO, K'=20
Gaussian
steps
Figure 19: Fine-tuning performance (averaged over five seeds, standard deviation not shown) after pre-training with M1, M2, and M3 in Avoid task from D3IL. DPPO (K 20), Gaussian, and GMM policies are compared. We also sweep the number of fine-tuned denoising steps K' in DPPO.
26

---

Published as a conference paper at ICLR 2025
D.5 COMPARING TO OTHER POLICY PARAMETERIZATIONS THE EASIER TASKS FROM ROBOMIMIC
Figure 20 compares the performance of DPPO Gaussian and GMM baslines, across variety of architectures, and with state and pixel inputs, in Lift and Can environments in the ROBOMIMIC suite. Compared to the Square and Transport (results shown in Section 5), these environments are considered to be "easier", and this is reflected in the greater performance of DPPO and Gaussian baselines (GMM still exhibits subpar performance). Nonetheless, DPPO still achieves similar or even better sample efficiency compared to Gaussian baseline.
1.0
Lift State
Can State
Lift Pixel
Can Pixel
0.8
0.8
0.
0.8
Success
0.6
0.6
0.6
0.6
0.
0.2
0.0
0.0
Environment steps
DPPO-MLP
DPPO-VIT-MLE OPPO-VIT-UNet
Geussion-VIT-Mu
DPPO-UNel
Gaussian Transformer
Transforner
Figure 20: Comparing to other policy parameterizations in the easier Li ft and Can tasks from ROBOMIMIC, with state (left) or pixel (right) observation. Results are averaged over three seeds.
D.6 COMPARING TO POLICY GRADIENT USING EXACT LIKELIHOOD OF DIFFUSION POLICY
Here we experiment another novel method (which, to our knowledge, has not been explicitly studied any previous work) for performing policy gradient with diffusion-based policies. Although diffu- sion model does not directly model the action likelihood, po(an]s) there have been ways to estimate the value, e.g., by solving the probability flow ODE that implements DDPM (Song al., 2020b). We refer the readers to Appendix. D in Song et al. (2020b) for a comprehensive exposition. We follow the official open-source code from Song et al.?, and implement policy gradient (single-level MDP) that uses the exact action likelihood xn (0r(st).
Fig. 21 shows the comparison between DPPO and diffusion policy gradient using exact likelihood estimate. Exact policy gradient improves the base policy in Hopper-v2 but does not outperform DPPO. It also requires more runtime and GPU memory as it backpropagates through the ODE. In the more challenging Can its success rate drops to zero. Moreover, policy gradient with exact likelihood does not offer the flexibility of fine-tuning fewer-thar thar-hhan-- denoising steps or discounting the early denoising steps that DPPO offers, which have shown in Appendix D.2 to often improve fine-tuning efficiency.
Hopper-v2
HalfCheetah.r2
Eralgation rewerr
5000
Sucgessa rate 1
3000
105
1000
Environment steps
DPPO
Exact Likelihood
Comparing to diffusion policy gradient with exact action likelihood. Results averaged over five seeds in Hopper-v2 and MalfCheetah-v2 and three seeds in Can.
Figure 21:
D.7 ABLATING STRUCTURED EXPLORATION IN DPPO
Here we provide additional evidence on how structured exploration of DPPO (Section 6) aids RL fine-tuning. While Fig. 19 compares DPPO with Gaussian and GMM policies and shows DPPO
httg::/gghhbb.cmmyyaagggnnneeee pytorch
27

---

Published as a conference paper at ICLR 2025
trajectories achieve wide coverage and stay near the expert data manifold, in Fig. 22 ablate such structured exploration within DPPO. We use DDIM (Song et al., 2020a) such that actions can this allows us to sample trajectories without adding any noise to intermediate denoising steps but only to the final denoised action (A: = 0), and compare that to DPPO with noise added to all denoising steps. In both cases, we consider the minimum noise level esp of 0.05 and 0.1. We see in Fig. 22 that with higher noise level, DPPO trajectories cover the expert data modes well without exploring aggressively into new modes, while in the case of only adding noise to the final step, the trajectories become less structured especially in M3.
be sampled deterministically
Then we run both exploration schemes in Can and Square from ROBOMIMIC, and Fig. 14 right shows the original DPPO setup achieves faster convergence than when noise is only added to the final step. This result, on top of results from Section 5.3 showing DPPO achiving better sample efficiency than Gaussian and GMM policies, showcases the benefit of structured exploration in fine- tuning.
Training trajectories beginning fine-tuning
1.0
Can- Stat-Sttt
Pro training data
DPPO (w/ only at k-o)
0.8
Oaza 0.05
ean
Ore 0.05
0.6
M1
Success
Squarre
M2
0.4
02
167
0.0
02
0.4
0.6
0.6
1.0
Environment steps
DPPO
DPPO (w/ noise only at k=
Figure 22:
Ablating Structured Exploration in DPPO. (Left) Sampled trajectories with noise added to all denoising steps vs. only to the last step k = 0 in Avoid. (Right) Results are averaged over three seeds in Can and Square.
E REPORTING OF WALL-CLOCK TIMES
Comparing to other diffusion-basse RL algorithms Section 5.1. Table and Table 2 shows the the wall-clock time used in each OpenAI GYM task and ROBOMIMIC task. In GYM tasks, on average DPPO trains 41%, 37%, and 12% faster than DAWR, DIPO, and DQL, respectively, which all require a significant amount of gradient updates per sample to train stably. QSM, DRWR, and IDQL trains 43%, 33%, and 7% faster than DPPO, respectively. ROBOMIMIC tasks are more expensive to simulate, especially with Transport task, and thus the wall-clock difference is smaller among the different methods. All methods use comparable time except for DIPO that uses slightly more on average.
Task
Method
Fopper-v2
Ma ker2D-v2
Ha 1 #Cheatsh-u2
DRWR
11.3
12.7
10.4
DAWR
30.4
30.7
27.1
DIPO
27.8
27.9
26.0
IDQL
16.3
16.1
15.5
DQL
20,5
20.5
17.6
QSM
9.6
9.9
9.7
DPPO
16.6
18.3
16.8
Table 1: Wall-clock time in seconds for a single training iteration OpenAT GYM tasks when comparing diffusion- based RL algorithms. Each iteration involves 500 environment timesteps in each of the 40 parallelized environments running on 40 CPU threads and a NVIDIA RTX 2080 GPU (20000 steps total).
28

---

Published as a conference paper at ICLR 2025
Task
Method
Lift
Square
Transport
DRNR
32.5
39.5
59.8
346.1
DAWR
38.6
46.0
70.5
354.3
DIPO
43.9
73.3
359.7
IDQL
33.8
63.7
DQL
QSM
DPPO
35.2
42.0
65.6
350.3
Table 2: Wall-clock time in seconds for single training iteration in ROBOMIMIC tasks with state input when comparing diffusion-based RL algorithms. Each iteration involves episodes (1200 environment timesteps for Lift and Can, 1600 for Square, and 3200 for Tr Transport) from each the 50 parallelized environments running on 50 CPU threads and a NVIDIA L40 GPU (60000, 80000, 160000 steps).
Comparing other policy parameterizations and architecture Section 5.3 and Section 5.4. Table 3 and Table 4 shows the wall-clock time used in fine-tuning in each ROBOMIMIC task with state or pixel input, respectively. Gaussian and GMM use similar times and Transformer is slightly more expensive than MIP. On average with state input, DPPO-MLP trains 24%, 21%, 24%, and 22% slower than baselines due to the more expensive diffusion sampling. DPPO-UNet requires more time with the extensive use of convolutional and normalization layers and trains average 49% slower than DPPO-MLP. On average with pixel input, DPPO- ViT-ML trains 14% slower than Gaussian-ViT-MLP simulation can be expensive. Table 5 shows the wall-clock time used in FURNITURE- BENCH tasks. DPPO-UNet trains 20% slower than Gaussian-MLan on average.
the difference is smaller than the state input case as the rendering in
Task
Method
LiSt
Can
Square
Transport
Gaussian-MLP
27.7
35.7
56.2
255.6
Gaussian Guussan--Trassfrrne Transformer
29.8
37.1
57.8
266.1
GMM MLP
28.0
36.2
55.2
254.5
GMM-Transfomner
29.5
37.4
58.1
260.2
DPPO MLP
35.6
43.3
65.0
350.5
DPPO-UNct
83.6
92.7
431.1
Table 3: Wall-clock time seconds for a single training iteration in ROBOMIMIC tasks with
state input when comparing policy parameterizations. Each iteration involves 4 episodes (1200 environment timesteps for Li and Can, 1600 for Square, and 3200 for Transport) from each of the 50 parallelized environments running 50 CPU threads and NVIDIA L.40 GPU (60000, 80000, 160000 steps).
Task
Method
Lift
Can:
Square
Transport
Gaussian MLP
153.6
173.1
277.0
770.0
DPPO ViT MLP
1949
202.5
328.5
871.3
Table 4: Wall-clock time in seconds for a single training iteration in ROBOMIMIC tasks with pixel input when comparing policy parameterizations. Each iteration involves 4 episodes (1200 environment timesteps for Lift and Can, 1600 for Square, and 3200 for Transport) from each of the 50 parallelized environments running on 50 CPU threads and a NVIDIA L40 GPU (60000, 80000. 160000 steps).
29

---

Published as a conference paper at ICLR 2025
Method
One-leg
Lamp
Rouzd-table
Gaussian-MLP
101.8
202.8
168.7
DPPO-UNet
148.4
258.2
188.6
Table 5: Wall-clock time in seconds for a single training iteration FURNITURE- tasks when comparing policy parameterizations. Each iteration involves 1 episodes (700 environment timesteps for One-leg, and 1000 for Lamp and Round-table) from each of the 1000 paral- lelized environments running NVIDIA L40 GPU (700000, 1000000, 1000000 steps).
F ADDITIONAL EXPERIMENTAL DETAILS
Task / Dataset
Ohs dim State
Obs dim Pixel
Act dim
T
Sparse reward ?
Hoppez V2
GYM
Ralker20-v2
No
dsllCheetah-v2
17
1000
No
Kitcchen-Uomplste-v0 Kitchen-Parttial-o0
60
280
Yes
FRANKA-KITCHEN
Kitchen-Nixed-v0
Lift
19
300
Yes
Can
23
300
Yes
ROBOMIMIC, state
Square
transpozt
LifL
9
96x
300
Yos
Can Square
96
300
Yes
ROBOMIMIC, pixel input
Transpost
2x96xx96
One leg
58
10
700
Yos
BURNITURE-BENCE
Tamp
44
1000
Yes
Round Lable
1000
Yes
MI
100
Yos
D3IL
M2
100
Yes
M3
100
Yes
Table 6: Comparison of the different tasks considered. "Obs dim State": dimension of the state observation input. "Obs dim State": dimension of the pixel observation input. "Act dim State": dimension of the action space. T: maximum number of steps in an episode. "Sparse reward ?": whether sparse reward is used in training instead of dense reward.
F.1 DETAILS OF POLICY ARCHITECTURES USED IN ALL EXPERIMENTS
MIP. For most of the experiments, we use a Multi-layer Perceptron (MLP) with two-layer residual connection the policy head. For diffusion-basee policies, also small MLP encoder for the state input and another small MLP with sinusoidal positional encoding for the denoising timestep input. Their output features are then concatenated before being fed into the MLP head. Diffusion Policy, proposed by Chi et al. (2024b), does not use MLP as the diffusion architecture, but we find delivers comparable (or even better) pre- treirrinnnng performance compared to UNet.
Transformer. For comparing to other policy parameterizations Section 5.3, we also consider Transformer as the policy architecture for the Gaussian and GMM baselines. We consider decoder only. No dropout used. learned positional embedding for the action chunk is the sequence into the decoder.
UNet. For comparing other policy parameterizations in Section 5.3, we also consider UNet (Ronneberger et al., 2015) as a possible architecture for DP. We follow the implementation from Chi et al. (2024b) that uses sinusoidal positional encoding for the denoising timestep input, except for using a larger MLP encoder for the observation input in each convolutional block. We find this modification helpful in more challenging tasks.
30

---

Published as a conference paper at ICLR 2025
VIT. For pixel-based experiments in Section 5.3 use Vision-Transfone Transformer(ViT) based image encoder introduced by Hu et al. (2023) before an MLP head. Proprioception input is appended to each channel of the image patches. We also follow (Hu et al., 2023) and use learned spatial emhedding for the ViT output to greatly reduce the number features, which are then fed into the downstream MLP head.
F.2 ADDITIONAL DETAILS OF GYM TASKS AND TRAINING IN SECTION 5.1
Pre-training. The observations and actions are normalized to [0. 1] using min/max statistics from the pre-training dataset. For all three tasks the policy is trained for 3000 epochs with batch size 128, learning rate of le-3 decayed to le-4 with cosine schedule, and weight decay of 1e-6. Exponential Moving Average (EMA) is applied with decay rate of 0.995.
Fine-tuning. All methods from Section 5.1 use the same pre-trained policy. Fine-tuning is done using online experiences sampled from 40 parallelized MuJoCo environments (Todorov et al., 2012). Reward curves shown in Fig. 4 are evaluated by running fine-tuned policies with a exp 0.001 (i.e.. without extra noise) for 40 episodes. Each episode terminates if the default conditions are met or the episode reaches 1000 timesteps. Detailed hyperparameters are listed in Table 7.
F.3 DESCRIPTIONS OF DIFFUSION-BALE BASED RL ALGORITHM BASELINES IN SECTION 5.1
DRWR:
This is customized reward-weighted regression (RWR) algorithm Peters and Schaal (2007) that fine-tunes a pre-trained DP with a supervised objective with higher weights on actions that lead higher reward-to-go
The reward is scaled with / and the exponentiated weight is clipped at Wmax- The policy is updated with experiences collected with the current policy (no buffer for data from previous iteration) and replay ratio of Ng. No critic is learned.
Cg To.50 minn(e Wmax) ) co
(aff, St, k)]pp
DAWR: This is customized serrantage-weighhee regression (AWR) algorithm Peng et al. (2019) that builds DRWR but uses TD-bootstrapped Sutton and Barto (2018) advantage estimation in- stead the higher-variance reward-to-ge to-go for better training stability and efficiency. DAWR (and DRWR) be seen as approximately optimizing (4.2) with Kullback-Leibler (KL) divergence constraint on the policy Peng et al. (2019), 9); Black et al. (2023).
The advantage is scaled with 3 and the exponentiated weight is clipped at Wmaxx Unlike DRWR, follow (Peng et al., 2019) and trains the actor in an off-policy manner: recent experiences are saved in replay buffer D, and the actor is updated with replay ratio of Ng.
L. pi se
Wmax) ||= Ea st. k3]]
The critic is updated less frequently (we find diffusion models need many gradient updates to fit the actions) with replay ratio of No-
L. = E [IlÁg(s,, a,) A(se. o.)1
where calculated using TD(A), with ADAWR and the discount factor TENY-
DIPO (Yang et al., 2023): This baseline applies "action gradient" that uses a learned state-action Q function update the actions saved in the replay buffer, and then has DP fitting on them without weighting.
Similar DAWR, recent experiences saved in replay buffer D. The actions (k 0) in the buffer are updated for MDIPP iterations with learning rate aDIPO-
at m+1A== a, m, + aDIPO V V QeQ m,k=0)) ), m = O,...,,Pie - 1.
The actor is then updated with a replay ratio of No.
L. E ||e (a Mo St, 1)
The critic is trained to minimize the Bellman residual with replay ratio of No. Double Q-learning also applied.
Ls = = [I(R. + TENv Q6 Fo(at=llss
( s(+1)) - Qo(,,, m- 0.
31

---

Published as a conference paper at ICLR 2025
IDQL (Hansen-Estruch et al., 2023): This baseline learns state- atate-action Q function and state V function to choose among the sampled actions from DP. DP fits on new samples without weighting.
Again recent experienees saved in a replay buffer D. The state value function is updated to match the expected Q value with an expectile loss, with a replay ratio of Ng.
Lv E [[TQq (s a.) < (sx)))
The value function used to update the Q function with replay ratio of No. The actor fits all sampled experiences without weighting, with a replay ratio No.
L = E [I( + "ENv (st+1) - Qp(s,, -.
Lo F.P [lle+ - co(a,. 8t, A)]l]]]
At inference time, MiDQ actions sampled from the actor. For training, Boltzmann exploration is applied based on the difference between Q value of the sampled actions and and the V value at the current state. For evaluation, the greedy action under Q chosen.
DQL (Wang et al., 2022): This baseline learns state-action Q function and backpropagates the gradient from the critic through the entire actor (with multiple denoising steps), akin to the usual Q-learning.
Again recent experiences saved in replay buffer D. The actor is then updated using both supervised loss and the value loss with a replay ratio of No-
Ce = E [|l= - =s(af, st, k)ll - CeDQLQ(ee #e (s)ss))))
where oppq.. is weighting coefficient. The critic is trained to minimize the Bellman residual with replay ratio of Ng. Double Q- Q-learning is also applied.
C. = E [I[(Re + TeNv "ENvQQ Ro (o(4+411544)) Q c
QSM (Psenka et al., 2023): This baselines learns state-action Q function, and then updates the actor by aligning the score of the diffusion actor with the gradient of the Q function.
Again recent experiences are saved in replay buffer D. The critic is trained to minimize the Bellman residual with replay ratio of Ng. Double Q-learning is also applied.
Le = P [R + JENV (sr++1, 7o 1
(s+1)) Q0( a 1?]
The actor updated follows with replay ratio of No.
L. = E c V Vo a.) (-Eo(af, st, k))]
where 0iQSM scales the gradient. The negative sign before Eq is from taking the gradient of the mean ju the denoising process.
F.4 DESCRIPTIONS OF RL FINE-TUNING ALGORITHM BASELINES SECTION 5.2
In this subsection, detail the baselines RLPD, Cal cal-QL, and IBRL. All policies ma param- eterized as unimodal Gaussian.
RLPD (Ball et al., 2023): This baseline is based on Soft Actor Critic (SAC, Haarnoja et al. (2018))
it learns an entropy-regularized state-action Q function, and then updates the actor by maximizing the Q function w.r.t. the action.
replay buffer initialized with offline data, and online samples are added to D. Each gradient update uses batch of mixed 50/50 offline and online data. An ensemble of Nutic crities is used, and at each gradient step two critics are randomly chosen. The critics are trained to minimize the Bellman residual with replay ratio No:
L. = E [[](R, + "/evv Qe (S141, (0(( - Qs at)
The target critic parameter of updated with delay. The actor minimizes the following loss with replay ratio of 1:
Lo E (s a.) aen log (o(a (s:]]
where Gen is the entropy coefficient (automatically tuned as in SAC starting 1).
32

---

Published as a conference paper at ICLR 2025
Cal-QL (Nakamoto et al., 2024): This baseline trains the policy and the action- value function Q" in an offline phase and then an online phase. During the offline phase only offline data is sampled for gradient update, while during the online phase mixed 50/50 offline and online data are sampled. The critic is trained to minimize the following loss (Bellman residual and calibrated Q-learning):
D =E" [1[ + TENN (s To (cc-1/s(+1)) (s(s), at)l
+ +eegl (IE2 max Qs az), (ss))) E² (Qo(s;, at)]):
where Begi a weighting coefficient between Bellman residual and calibration Q-Learning and V (se)) is estimated using Monte-Carlo returns. The target critic parameter ø' is updated with delay. The actor minimizes the following loss:
Lo E - Qa(s), ax) + ceent log (a [s:]]:
where Gent is the entropy coefficient (automatically tuned as in SAC starting at 1).
IBRL (Hu et al., 2023): This baseline first pre-trains a policy pe using behavior cloning, and for fine-tuning it trains RL policy he initialized as uú. During fine-tuning recent experiences saved in a replay buffer D. An ensemble of Nerti critics used, and at each gradient step two critics are randomly chosen. The critics are trained minimize the Bellman residual with replay ratio Ng:
L. EE [[[[R=
TENV
max d'E(a"",
QΦ (86+1, a') (s [
where aL Huw ($1+1) (no noise) and a Th (8c-1), and π&a the target actor. The target critic parameter ø' is updated with delay. The actor minimizes the following loss with a replay ratio of 1:
L. = -===E'(Qus000 (s0(())))))) ox)]
The target actor parameter 0' also updated with delay.
F.5 ADDITIONAL DETAILS OF FRANKA- KITCHEN TASKS AND TRAINING IN SECTION 5.2
Tasks. We consider three settings from the D4RL benchmark (Fu al., 2020): (1) Kit Kitcheen-Complete--vv containing demonstrations that
that complete the entire task (four subtasks), (2) Kitchen-Partial- containing some complete demonstrations and many ones complet- ing only subtasks, and (3) Kitchen -ixeed---i---d-------- containing incomplete demonstrations only.
Pre-training. The observations and actions are normalized to [0, 1] using min/max statistics from the pre-training dataset. No history observation (proprioception or mound-truth object states) is used. All policies are trained with batch size 128, learning rate le-4 decayed to 1e-5 with a cosine schedule, and weight decay le-6. DPPO policies trained with 8000 epochs. For IBRL and Cal-QL we follow the hyperparameters from the original implementations - IBRL proposes using (1) wider MLP layers and (2) dropout during pre-training, which we follow too. We use T' 4 for DPPO; we also tried to use the same action chunk size with IBRL, RLPD, and Cal-QL, but we find for all of them Tą leads to better performance.
Fine-tuning. With DPPO, policies are fine-tuned using online experiences sampled from 40 paral- lelized MuJoCo environments (Todorov et al., 2012), while the baselines use only one environment (matching their original implementations). Episodes terminates when they reach maximum episode lengths (shown in Table 6) or all four subtasks are completed. Detailed hyperparameters are listed in Table 8- follow the hyperparameter choices from the original implementations of the baselines.
Larger variance with DPPO in Fig. 11. In Fig. 11, it is shown that DPPO exhibits larger vari- ance in normalized score with Kitchen Maateen-Patiial Partial- than Cal-QL. This due to DPPO solv- ing either 3/4 or 2/4 subtasks in one seed (low variance within the evaluation episodes in one seed) but high variance over seeds, whereas Cal-QL has higher variance among evaluation episodes in seed but average over seeds it shows lower variance. This also highlights notable and Kitchen-Mixed-e
property of DPPO: Kitchen-Partiall completing subtasks, thus being highly multi-modal. Diffusion policy can sometimes struggle to
have trajectories only
33

---

Published as a conference paper at ICLR 2025
learn all the modes from pre-training, and since DPPO directly fine-tunes the pre-trained policy, can fail to converge to 100% success rate at fine-tuming tinettuning Cal-QL instead learns from all of- fline data during fine-tuning in an off off-poliey manner, thus less sensitive to pre-training performance. Nonetheless, with offline data completing tasks consistently despite varying quality (ROrOMIMIC and ki come ete--v, which, believe, are more realistic in the current paradigm of robot manipulation), DPPO demonstrates much better final performance than Cal-QL and other baselines in Fig. 11.
F.6 ADDITIONAL DETAILS OF ROBOMIMIC TASKS AND TRAINING IN SECTION 5.3
Tasks. We consider four tasks from the ROBOMIMIC benchmark (Mandlekar et al., 2021): (I) Lift: lifting a cube from the table, (2) Can: picking up a Coke can and placing it at a target bin, (3) Square: picking up square nut and place it rod, and (4) Transport: two robot arms removing a bin cover, picking and placing a cube, and then transferring a hammer from one container another one.
Pre-training. ROBOMIMIC provides the Multi-Human (MH) dataset with noisy human demon- strations for each task, which we use to pre-train the policies. The observations and actions are normalized to [0, using min/max statistics from the pre-training dataset. No history observation (pixel, proprioception, or ground-trute trunth object states) is used. All policies are trained with batch size 128, learning rate le-4 decayed to le-5 with cosine schedule, and weight decay 1e-6. Diffusion- based policies are trained with 8000 epochs, while Gaussian and GMM policies are trained with 5000 epochs we find diffusion models require more gradient updates to fit the data well.
Fine-tuning, Diffusion bbases, Gaussian, and GMM pre- pre-trained policies are then fine-tuned using online experiences sampled from 50 parallelized MuJoCo environments (Todorov et al., 2012). Suc- cess rate curves shown in Fig. 4, Fig. 5, and Fig. 20 are evaluated by running fine-tuned policies with a min 0.001 (i.e., without extra noise) for 50 episodes. Episodes terminates only when they reach maximum episode lengths (shown in Table 6). Detailed hyperparameters are listed in Table 9.
exp
Pixel training. We use the wrist camera view in Lift ft and Can, the third-person camera view in Square, and the two robot shoulder camera views in Transport. Random-shift data augmenta- tion is applied to the camera images during both pre-training and fine-tuning, Gradient accumulation used in fine-tuning so that the same batch size (as in state-input training) can fit on the GPU. De- tailed hyperparameters are listed in Table 10.
F.7 DESCRIPTIONS OF POLICY PARAMETERIZATION BASELINES IN SECTION 5.3
Gaussian. We consider unimodal Gaussian with diagonal covariance, the most commonly used policy parameterization in RI. The standard deviation for each action dimension, Gaa- is fixed during pre-training; we also tried to learn Gau from the dataset but we find the training very unstable. During fine-tuning Gan is learned starting from the same fixed value and also clipped between 0.01 and 0.2. Additionally we clip the sampled action to be within 3 standard deviation from the mean. As discusses in Appendix C. we choose the PPO clipping ratio E based on the empirical clipping fraction in each task. This setup is also used in the FURNITURE-EENCH experiments. We note that we spend significant amount of efforts tuning the Gaussian baseline, and our results with are some of the best known ones in RL training for long-horizon manipulation tasks (exceeding our initial expectations), c.g., reaching 100% success rate in Lamp with Low randomness.
GMM. We also consider Gaussian Mixture Model the policy parameterization. We denote MMMM the number of mixtures. The standard deviation for each action dimension in each mixture, doMM. is also fixed during pre-training. Again during fine-tuning FGMM learned starting from the same fixed value and also clipped between 0.01 and 0.2.
F.8 ADDITIONAL DETAILS OF FURNITURE-BEIE TASKS AND TRAINING IN SECTION 5.4
Tasks. We consider three tasks from the FURNITURE- BENCH benchmark (Heo et al., 2023): (1) One-leg: assemble leg of a table by placing the tabletop in the fixture corner, grasping and
34

---

Published as a conference paper at ICLR 2025
inserting the table leg, and screwing in the leg, (2) Lamp: place the lamp base in the fixture corner, grasp, insert, and screw in the light bulb, and finally place the lamp shade, (3) Round-table place a round tabletop in the fixture corner, insert and screw in the table leg, and then insert and screw in the table base. See Fig. 23 for the visualized rollouts in simulation.
Pre-training. The pre-training dataset collected in the simulated environments using a Space- Mouse", a DoF input device. The simulator runs at 10Hz. At every timestep, we read off the state of the SpaceMouse as 5a [Ar, Ay. Az, Aroll, Apitch, Ayaw], which is converted to a quaternion before passed to the environment step and stored as the action alongside the current observation in the trajectory. If | < & Vi for some small 0.05 defining the threshold for a no-op, we do pass it to the environment. Discarding no-ops is important for allowing
not record any action the policies to learn from demonstrations effectively. When the desired number of demonstrations has been collected (typically 50), we process the actions convert the delta actions stored from the SpaceMouse into absolute pose actions by applying the delta action to the current EE pose at each timestep.
The observations and actions are normalized to [-1, 1] using min/max statistics from the pre-training dataset. No history observation (proprioception or ground-truth object states) used, i.e. only the current observation is passed to the policy. All policies are trained with batch size 256, learning rate le-4 decayed to le-5 with a cosine schedule, and weight decay 1e-6. Diffusion-based policies are trained with 8000 epochs, while Gaussian policies trained with 3000 epochs. Gaussian policies can casily overfit the pre-trained dataset, while diffusion- based policies are more resilient. Gaussian policies also require very large MLP (~10 million parameters) to fit the data well.
Fine-tuning. Diffusion-barn and Gaussian pre-trained policies are then fine-tuned using online experiences sampled from 1000 parallelized IsaacGym environments Makoviychuk al. (2021). Success rate curves shown in Fig. 6 are evaluated by running fine-tuned policies with o cap = 0.001 (i.e.. without extra noise) for 1000 episodes. Episodes terminate only when they reach maximum episode length (shown in Table 6). Detailed hyperparameters are listed in Table 11. We find smaller amount of exploration noise (we set o exp and Gau to be 0.04) is necessary for the pre-trained policy achieving nonzero success rates at the beginning of fine-tuning.
One-leg
Lamp
Round-table
Figure 23: Representative rollouts from simulated BURNITURE-BENCE tasks.
Hardware setup robot control. The physical robot used is Franka Emika Panda arm. The policies output a sequence of desired end-effector poses in the robot base frame to control the robot. These poses are converted into joint position targets through differential inverse kinematics. We calculate the desired end-effector velocity as the difference between the desired and current poses divided by the delta time dt 1/10. We then convert this to desired joint velocities using the Jacobian and compute the desired joint positions with first-order integration over the current joint positions and desired velocity. The resulting joint position targets are passed a low-level joint impedance controller provided by Polymetis (Lin et al., 2021), running at 1kHz.
35

---

Published as a conference paper at ICLR 2025
Hardware setup- state estimation. To deploy state-based policies real hardware, utilize AprilTags (Wang and Olson, 2016) for part pose estimation. The FURNITURE-BEE 2023) task suite provides April'aags for each part and code for estimating part poses from tag detec- tions. The process involves several steps: (1) detecting tags in the camera frame, (2) mapping tag detections to the robot frame for policy compatibility, (3) utilizing known offsets between tags and object centers in the simulator, and (4) calibrating the camera pose using an AprilTag at a known position relative to the robot base. Despite general accuracy, detections can be noisy, especially during movement or partial occlusion, which the One-leg task features. Since the task requires high precision, we find the following to help make the estimation reliable enough:
(Heo et al,,
Camera coverage: We find detection quality sensitive to distance and angle between the camera and tag. This issue is likely due to the RealSense D435 camera having mediocre image quality and clarity and the relatively small tags. To remedy this, we opt to use 4 cameras roughly evenly spread out around the scene to ensure that at least one camera has solid view of a tag on all the parts (i.e., as close as possible with a straight-on view). To find the best camera positions, we start with having camera in each of the cardinal directions around the scene. Then, we adjust the pose of each to get it as close as possible to the objects while still covering the necessary workspace and capturing the base tag for calibration. Moving the robot arm around the scene to avoid the worst occlusion is also helpful.
Even with better camera coverage and placement, detection quality depends on having crisp images. We find proper lighting helpful to improve image quality. In particular, the scene should be well and evenly lit around the scene without causing reflections in either the tag or table.
Lighting:
Bad detections can sometimes cause the resulting pose estimate to deviate signifi- cantly from the true pose, i.e., jumping several centimeters from one frame to the next. This usually only happens on isolated frames, and thus before "acceeptig" given detection, we check if the new position and orientation are within 5 cm and 20 degrees of the previously ac- cepted pose. In addition, we apply low-pass filtering on the detection using simple exponential average (with a = 0.25) to smooth out the high- nighffrequency noise.
Filtering:
The objects have multiple tags that can be detected from multiple cameras. After performing the filtering step, we average all pose estimates for the same object across different tags and cameras, which also helps smooth out noise. This alone, however, does not fully cancel the case when a single detection has a large jump, as this can severely skew the average, still necessitating filtering step. Having multiple cameras benefits this step, too, as provides more detections to average over.
Averaging:
Caching part pose in hand: A particularly difficult phase of the task to achieve good detections is when the robot transports the table leg from the initial position to the tabletop for insertion. The main problems are that the movement can blur the images, and the grasping can cause occlusions. Therefore, we found it helpful to assume that once the part was grasped by the robot, it would not move in the grasp until the gripper opened. With this, we can "cache" the pose of the part relative to the end-effector once the object is fully grasped and use this instead of relying on detections during the movement.
Normalization pitfalls and clipping:
We generally use min-max normalization of the state observations to ensure observations are in 11. The tabletop part moves very little in the z-direction demonstration data, meaning the resulting normalization limits (the minimum and maximum value of the data) can be very close, Tmax Imin O. With these tight limits, the z-zmin Therefore, ensure noise in the real-world detection can be amplified greatly as Fnorm Jam ZZain"
that normalization ranges are reasonable. As an extra safeguard, clipping the data to [-1, 1] can also help.
Only estimate necessary states: Despite the One-leg task having 5 parts, only are manipu- lated. Only estimating the pose of those parts can eliminate a lot of noise. In particular, the pose of the legs that are not used and the obstacle (the U-shaped fixture) can be set to an arbitrary value from the dataset.
Visualization for debugging: We use the visualization tool MeshCat extensively for debugging of state estimation. The tool allows for easy visualizations of poses of all relevant objects in the
Phttps://g wtt.://wo//tcccceeeeeeeeee dev/ haat
36

---

Published as a conference paper at ICLR 2025
scene, like the robot end-effector and parts, which makes sanity-checking the implementation far easier than looking at raw numbers.
Hardware evaluation. We perform trials for each method. We adopt a single-blind model selection process: at the beginning of each trial, first randomize the initial state. Then, randomly select a method and roll it out, but the experimenter does not observe which model is used. We record the success and failure of each trial and then aggregate statisties for each model after all trials are completed.
DP Pro-trained
(A) Pre-trained Diffusion policy performs successful rollout
Policy pushes peg down without proper alignment with hole before releasing the peg, making topple over
Fine-tuned
(C) Fine-tuned DPPO policy performs successful rollout
(D) Initial peg alignment the policy corrects placement until properly inserted in the hole before letting
Figure 24: Qualitative comparison of pre-trained fine-tuned DPPO policies in real eval- uation. (A) Successful rollout with the pre-trained policy. (B) Failed rollout with the pre-trained policy due to imprecise insertion. (C) Successful rollout with the fine-tuned policy. (D) Successful rollout with the fine-tuned policy that requires corrective behavior.
Domain randomization for sim-to-real transfer. To facilitate the sim-to-real transfer, we apply additional domain randomization to the simulation training. We record the range of observation noises in hardware without any robot motion and then apply the same amount of noise to state observations in simulation. We find the state estimation in hardware particularly sensitive to the object heights. Also, we apply random noise (zero mean with 0.03 standard deviation) to the sampled action from DPPO to simulate the imperfect low-level controller; we find adding such noise to the Gaussian policy leads to zero task success rate while DPPO robust to it (also see discussion in Section 6).
BC regularization loss used for Gaussian baseline. Since the fine-tuned Gaussian policy exhibits very jittery behavior and leads to zero success rate in real evaluation, we further experiment with adding behavior cloning (BC) regularization loss in fine-tuning with the Gaussian baseline. The
37

---

Published as a conference paper at ICLR 2025
combined loss follows
BBC = Lo - Cgp Pau
log Beeminee (a) dz
k+1 st]],
A
where reeeeed is the frozen BC-only policy. The extra term encourages the newly sampled actions from the fine-tuned policy remain high-likelihood under the BC-only policy. We set aBc 0.1. However, although this regularization reduces the sim-to-real gap, it also significantly limits fine- tuning, leading to the fine-tuning policy saturating at 53% success rate shown in Fig. 6.
F.9 ADDITIONAL DETAILS OF AVOID TASK FROM D3IL AND TRAINING IN SECTION
Pre-training. We split the original dataset from D3IL based on the three settings, M1, M2, and M3; in each setting, observations and actions are normalized to [0. 1] using min/max statistics. All policies are trained with batch size 16 (due to the small dataset size), learning rate le-4 decayed to le-5 with a cosine schedule, and weight decay le-6. Diffusion-based policies are trained with about 15000 epochs, while Gaussian and GMM policies are trained with about 10000 epochs; manually examine the trajectories from different pre-trained checkpoints and pick ones that visually match the expert data the best.
Fine-tuning. Diffusion-based, Gaussian, and GMM pre-trained policies are then fine-tuned us- ing online experiences sampled from 50 parallelized MuJoCo environments (Todorov al., 2012). Reward curves shown in Fig. 8 and Fig. 19 are evaluated by running fine-tuned policies with the same amount of exploration noise used in training for 50 episodes; we choose to use the training (instead of evaluation) setup since Gaussian policies exhibit multi-modality only with training noise. Episodes terminate only when they reach 100 steps.
Added action noise during fine-tuning. In Fig. left, demonstrate that DPPO exhibits stronger training stability when noise added to the sampled actions during fine-tuning. The noise starts the 5th iteration. It is sampled from uniform distribution with the lower limit ramping up to 0.1 and the upper limit ramping up to 0.2 linearly in iterations. The limits are kept the same from the 10th iteration the end of fine-tuning.
F.10 LISTED TRAINING PERPARAMETERS
38

---

Published as a conference paper at ICLR 2025
Task(s)
Method
Parameter
GYM
Lif:, Can
Squace
Transport
Taay
0.99
0.999
0.999
0.999
0.1
0.1
0.08
0.1
T
4
8
Common
Ta
8
learning Critic learning applies)
(tuned le-5)
Actur MLP dims Critic MLP dims (if applics)
512.
[512, 512, 512]
[1024, 1024, 1024]
[1024, 1024, 1024]
[256, 256, 256]
DRWR
Wanax
Ny
Batch size
1000
Wanax AnawR No
0.95
DAWR
64
No
16
Buffer size
2000000
120000
120000
120000
le-4
Muro
10
DIPO
N9
1000000
Mixe No
20
10
10
IDQL
Na
Buffer
1000000
250000
250000
250000
Ratch size
1000
ongi.
Buffer cize
1000000
Batch size
1000
QSM
Buffer size
1000000
250000
250000
250000
Batch size
1000
GAE X No N₂
0.95
10
10
DPPO
10
10
10
0.01
Batch size
50000
7500
10000
10000
Table 7: Fine-tuning hyperparameters for OpenAl GYM and ROBOMIMIC tasks when compar- ing diffusion-based RL methods. We list hyperparameters shared by all methods first, and then method-specific ones.
39

---

Published as a conference paper at ICLR 2025
Task(s)
Method
Parameter
Dall laLeehetttth v2
Kitchen Complete vo Kitchen Paz:ial vD Kitchen Mixed v0
Common
0.99
Tp
RLPD
N
Nutik
10
5
5
Batch
256
Tp
Cal-QL
Batch size
256
Tp
Nomic Batch size
256
Tp
0.1
"nn GAE No
DPPO
0.99
0.95
10
10
10
No
10
10
10
Batch K
10000
5600
5600
5600
20
K'
Can,
Square, FH
Square,
0.999
Common
Ta
T.
No
RLPD
Natk
Batch size
256
fegl
Cal-QL
Batch size
2.56
N
Nenic
Batch
256
PPp ni
0.1
of mi
0.1
"foaxoss GAE X
0.9
0.9
0.99
0.99
0.95
DPPO
0.01
Batch size
6000
15000
8000
20000
20
Table 8: Fine-tuning hyperparameters for Hal fCheethh-v2 FRANKA-KITCHEN, Can, and Square (PH MH datasets) when comparing demo-augmented RI. methods. We list hyperpa- rameters shared by all methods first, and then meethod-specifee ones.
40

---

Published as a conference paper at ICLR 2025
Task
Method
Parameter
LiEL, Can
Squaze
Transpozt
0.999
4
4
Actor learning rale
le.4
le.5
le (decayed to le.
Critic learning rate
le-3
GAE A
0.95
Common
No
10
ID
0.01 fannealed in DPPO)
Gaussian, Common
Cia
0.1
0.1
0.08
Batch
10000
Gaussian-Me P
Model size
552K
2.15M
1.93M
Gaussian- Transformer GrussianTransfommer
Model size
675K
1.86M
1.87M
MUMM
GMM, Common
GGNM
Batch size
7500
10000
10000
GMM-MLP
Model size
1.15M
4.40M
4.90M
GMM-TTanstoennc
Model size
680K
1.87M
1.89M
"Porsosss
550 mig
0.1
0.08
DPPO. Common
0.1
K
20
K'
10
Batch size
75000
100000
100000
DPPO-MLP
Model size
576K
2.31M
2.43M
DPPO-UNet
Model
652K
1.62M
1.68M
Table 9: Fine-tuning hyperparameters for ROBOMIMIC tasks with state input when comparing policy parameterizations. We list hyperparameters shared by all methods first, and then method- specific ones. Since the different policy parameterizations use different neural network architecture, list the total model size here instead of the details such MLP dimensions.
Task
Method
Parameter
ft, Can
Square
Transport
YYpsv
0.999
Ta
Actor learning rate
le.4
le-5
le.5 (decnyed to le-6)
Critic learning rate
Common
N₂
10
N
10
10
0.01 (anneuled in DPPO)
Model size
1.03M
1.03M
1.93M
Gaussian Causian-ViT-ML. VIT-MIPP
(TCu
0.1
0.1
0.08
Batch size
7500
10000
10000
Model size
1.06M
1.06M
2.05M
0.08
DPPO-WVT-MLP
K
100
K"
(DDIM)
Batch size
37500
50000
50000
Table 10: Fine-tuning hyperparameters. for ROBOMIMIC tasks with pixel input when comparing policy parameterizations. We list hyperparameters shared by all methods first, and then method- specific ones. Since the different policy parameterizations use different neural network architecture, we list the total model size here instead of the details such as MLP dimensions.
41

---

Published as a conference paper at ICLR 2025
Task
Mothod
Paramoter
One leg
aam
Round Lable
"eew Ta
0.999
Actor learning rale Critic learning rite
le-5 5 (decayed le
le-3
GAE
0.95
Common
N
0.001
Model size
10.64M
10.62M
10.2MM
Gaussian-Min
(iau
Batch size
8800
Model size
6.86M
6.8M
6.81M
TiassooSE mm pmb
DPPO-UNet
K'
(DDIM)
Batch size
44000
Table 11: Fine-tuning hyperparameters for SURNITURE-BENCEE tasks when comparing policy pa- rameterizations. We list hyperparameters shared by all methods first, and then method ones.
42


Reviews:
[{"review_id": "GumMnoRIvt", "reviewer": "Reviewer_yZhx", "summary": "The authors propose a framework for finetuning a pretrained diffusion policy using on-policy RL, specifically PPO, and show that their method improves performance on long-horizon sparse-reward tasks and works on real hardware.", "questions": "- What intuition, if any, do the authors have on why DPPO does better on long-horizon sparse-reward tasks? Is there something special about finetuning with PPO as opposed to other finetuning strategies that helps? \n\n- In Figure 6, why does DPPO-UNet get outperformed by DPPO-MLP on state-based tasks? And why is DPPO-UNet not present on the pixel-based versions of the task? \n\n- On lines 264-265 the authors state *\"We propose clipping the diffusion noise schedule at a larger-than-\nstandard noise level to encourage exploration and training stability.\"*. If more noise encourages exploration, would it not be more beneficial to use the variance-exploding line of diffusion models such as EDM instead of the variance-preserving ones? \n\n- In Figure 7 we see positive sim2real transfer from pretraining only, but negative transfer after fine-tuning. Do the authors have some intuition as to why this is the case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The authors explore the possibility of fine-tuning diffusion-based policies with on-policy RL and demonstrate some surprising results. Specifically, the authors find improved performance over baselines on long-horizon and sparse reward tasks. The authors also demonstrate how their method can leverage environment parallelization to train faster than off-policy RL methods. Given the abundance of GPU-accelerated simulators these days, I believe this to be a major strength as off-policy RL finetuning methods cannot leverage parallelization in the same way, leaving these high-throughput simulators as a relatively untapped resource until now. \n\n- The authors provide extensive comparisons to a variety of other RL finetuning methods, including two of their own novel baselines. The ablations are also thorough. \n\n- The paper is well written and the quality of the presentation is high.", "weaknesses": "- Much of section 6, including the first few paragraphs up to and including Figure 8 are redundant and unnecessary. At this point there are a number of diffusion policy papers applied to various tasks as well as the original diffusion policy paper itself that show that diffusion models consistently outperform GMM and Gaussian policies at capturing the modes of a multi-modal distribution, and so this result is no longer surprising or novel. I'd have rather seen this space dedicated to showing more experimental results that were instead put in the appendix, such as the comparisons in Figure 12 to methods that finetune on additional expert data, or perhaps a more in-depth analysis on why DPPO does better on long-horizon sparse-reward tasks. \"How does this method compare to just collecting and training (via BC or whatever else) on additional expert data?\" was a question that came to mind while I read the results, which is an important one to discuss and one I imagine other readers will have. \n\n- While the technical contribution is fairly good, the novelty of this approach is relatively low. The proposed method follows the more general and somewhat repetitive trend of \"take diffusion policy and try it on task X and potentially finetune with method Y\" style papers in the robot learning community. \n\nDespite the somewhat low novelty, I believe that the impressive results and the fact that this paper is one of the first, if not the first, diffusion policy finetuning methods capable of leveraging high-throughput simulators is an important contribution to the robotics and ML communities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KPh5iEZKJB", "reviewer": "Reviewer_7Q1R", "summary": "This paper proposes a framework (DPPO) to enable fine-tuning pre-trained diffusion-based policies through policy gradient methods. They compare DPPO with rich baselines in common benchmarks. They conclude best practices for DPPO and conduct ablation studies to understand the effectiveness of DPPO.", "questions": "* In the original diffusion policy paper, the performance in the transport is nearly 100% success rate, which conflicts with the results in the paper.\n* How about the results compared with the baselines that introduce a residual policy?\n* How about the results compared with the baselines that use RLPD or Cal-QL for the pre-trained diffusion policies?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This paper conducts rich experiments and analysis. They compare DPPO with other fine-tuning RL methods for diffusion policies and other fine-tuning RL policies. They also conduct several ablation studies and visualization to analyze the effectiveness of DPPO.\n* This paper is well-written and well-structured. They conclude the best practice for DPPO and the benefits of such framework.", "weaknesses": "* The novelty of the paper is limited. Previous works have proposed the idea that considers a denoising process as multi-step MDP. This paper simply integrate this MDP into the environmental MDP. The tricks to adapt PPO for better fine-tuning are also seen in the previous works like GAE except fine-tuning the last few steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGWjLgVus2", "reviewer": "Reviewer_WTdz", "summary": "The papers introduces Diffusion Policy Policy Optimization (DPPO), an algorithmic framework designed to fine-tune diffusion-based policies using policy gradient optimization (PPO) from reinforcement learning (RL). DPPO enables structured exploration, enhances training stability, and demonstrates the robustness and generalization during deployment. It improves policy performance across various benchmarks, including tasks with pixel-based observations and long-horizon rollouts, which have traditionally been challenging for previous RL methods.", "questions": "Please see the weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper introduces a novel dual-layer MDP framework and the use of PPO to fine-tune Diffusion-Policy, enhancing its performance in various tasks.\n(2) The paper proposes several best practices for DPPO, such as fine-tuning only the last few denoising steps and using DDIM sampling to replace some denoising steps, improving efficiency and performance.\n(3) DPPO shows a strong performance in all benchmark tests, particularly in tasks involving pixel-based observations and long-horizon rollouts.\n(4)  DPPO achieves highly efficient zero-shot transfer from simulation to real-world tasks in robotics, demonstrating a minimal sim-to-real performance gap.", "weaknesses": "(1) Figure 10 shows that the pre-trained Diffusion Policy can model multimodality in the behavior policy. However, after fine-tuning, the Diffusion Policy gradually converges into a deterministic policy. In this case, does it lose the advantage of expressing multimodal action distributions that Diffusion Policy is known for?\n\n(2) In the experiments, DPPO is only compared against other diffusion-based methods, without any comparison to existing state-of-the-art Offline2Online or Sim2Real algorithms, such as Uni-O4 [1] and O3F [2]. Clearly, there is a significant optimal gap in environments like HalfCheetah.\n\n(3) The motivation for fine-tuning models is typically to reduce the number of online interactions and improve sample efficiency. However, DPPO does not seem to significantly enhance sample efficiency, as its convergence speed is similar to other online reinforcement learning methods.\n\n(4) The v-D4RL [3] dataset could have been considered in the experiments for broader evaluation.\n\n(5) Compared to QSM [4], DPPO does not introduce particularly novel concepts but rather provides engineering or empirical adjustments, such as fine-tuning certain denoising steps and replacing value estimation with advantage estimation.\n\n[1] Kun Lei, et al. Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization\n[2] Seunghyun Lee, et al. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble\n[3] Cong Lu, et al. Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations\n[4] Michael Psenka, et al. Learning a Diffusion Model Policy from Rewards via Q-Score Matching", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhgHozuoxS", "reviewer": "Reviewer_sDz9", "summary": "This paper studies how to use PPO to finetune the diffusion-model based policy in an offline-to-online setting. By integrating denoising steps of diffusion model into task MDP, authors could utilize PPO to finetune the diffusion model for improved performance. Extensive efforts are put to find the best practice in fine-tuning diffusion policy, in terms of hyperparameters, network architecture and algorithmic variants. Finally,  the efficacy and the performance of the proposed framework  is demonstrated on various experiments in both simulated and real-world scenarios.", "questions": "1. Could you change the title \"Diffusion Policy Policy Optimization\" to provide more contexts to readers? Current title is too short and abstract and your work is not as general as it indicates. \n2. I would  suggest deleting/relocating some contents in early chapters as your paper is already over-sized and some of these contents are already known.\n4. in line 249, the definition of $\\gamma_{\\text{ENV}}^t$ seems to be incorrect? I thinks it should be $\\gamma_{\\text{ENV}}^{t^\\prime-t}$? or are there any further treatments of this discount factor?\n4. there lacks a pseudo-code for the whole algorithm, please provide one. For example, how is your V function trained?\n5. for fine-tuning control policy in RL, most existing algorithms finish in less than 1e6 steps. But the results presented throughout this paper is conducted in 1e6~1e7 steps. Could you truncate the figures to 5e5 such that we could better compare the performance of various methods?\n\n6. Could you provide more comparisons of DPPO against established offline-to-online methods like [1] and [2]?\n\n>[1] Wang, Shenzhi, et al. \"Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024).\n>\n>[2] Zhang, Haichao, Wei Xu, and Haonan Yu. \"Policy Expansion for Bridging Offline-to-Online Reinforcement Learning.\" The Eleventh International Conference on Learning Representations.\n\n7. Could you provide more explanations about the worse performance of DAWR, DRWR and Cal-QL? In appendix, the result of the performance of Cal-QL on  Can and Square are almost zero, which is surprising to me. Could you elaborate more on this? BTW, what is the performance of Cal-QL when only offline training is performed?\n\n8. for DRWR, in the weight function, could you minus a V function from the estimated return such that its’ more aligned toward DPPO?\n9. in line 988-994, you mentioned speedup and less GPU memory usage is obtained by fine-tuning only the last few K denoising steps. But the method is to  use a trainable $\\theta$ for the last K steps and a frozen $\\theta_{\\text{FT}}$ for previous steps. How does this method improve the training speed and reduce GPU memory?\n\n10. Sect. 6 of structural exploration is of much importance and novelty. Could you try ablating on this phenomenon for DP? for example, by tuning the denoising process such that DP itself performs less or more structural exploration and see if the performance is affected?\n\nThere are many interesting, fruitful discoveries and experiments in this work. But I find none of them is studied exhaustively or presented clearly.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper contains tons of discoveries related to fine-tuning diffusion policy via PPO. Specifically, numerous aspects like denoising steps to be fine-tuned, network architecture, GAE variants  and effects of expert data are all studied with experimental results reported. As fine-tuning diffusion policy for real-world robot is a very important field, these experiments are welcome and contain significant value  for the community. Last but not least, the authors also provide an empirical explanation for why their method obtains better results than others (i.e. structural exploration).\n\n$~$\n\n## Originality \nUtilizing RL method to finetune diffusion model is in great need. And the fruitful experimental results presented in this work  are of great value and originality.\n\n## Quality\n\nMany experiments and ablation studies are conducted and the best practices are summarized from them.\n\n## Clarity\n\nUnfortunately, this paper is not clearly written. In the main papers (1-10 pages), authors’ writing lacks a central logic  and the results are presented without  a focus. I have great trouble understanding the contribution and experimental results during reading. \n\n## Significance\nThe large amount of experiments conducted is of great value to the community, especially considering there are also real-world experiments.", "weaknesses": "Unfortunately, I personally think the presentation of this paper is very poor. With the extensive and fruitful experiments results, the authors fail to present them in a concise and focused manner in the main pages and make me quite confused during reading and comprehending this paper.\n\nFirstly, there are three components that are actively mentioned and studied: the diffusion policy, the PPO method and the best practice for combining them together. \n\nBut based on my comprehension, the diffusion policy and PPO used in this work are of no novel changes & contributions.\n\nCombining these 2 together and successfully applying on real-world robotic system, is with great value and potential hidden evils. But  the authors fail to present this to me  in a clear and concise manner: for example, there are  experiments comparing diffusion policy (DP) to gaussian parameterized policy, but the advantage of DP to vanilla parameterization is already well-established and not the contribution of this paper, in my opinion. Since this paper is already over-sized, I suggest the authors to relocate some contents into appendix and focus on your main contribution in the main pages\n\n\n\nTo summarize, I hope the organization and presentation of this paper could be improved such that the problem formulation, method design and experimental results could be tightly chained and focused on  your main contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)