Diffusion Policy Policy Optimization

## Abstract

Abstract We introduce Diffusion Policy Policy Optimization , DPPO , an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy [ 20 ] ) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations;
nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations.
Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io .

## 1 Introduction

Large-scale pre-training with additional fine-tuning has become a ubiquitous pipeline in the development of language and image foundation models [, , , ].
Though behavior cloning with expert data [] is rapidly emerging as dominant paradigm for pre-training *robot policies* [, , , , ],
their performance can be suboptimal [] due to expert data being suboptimal or expert data exhibiting limited coverage of possible environment conditions.
As robot policies entail interaction with their environment, reinforcement learning (RL) [] is a natural candidate for further optimizing their performance beyond the limits of demonstration data. However, RL fine-tuning can be nuanced for pre-trained policies parameterized as diffusion models [], which have emerged as a leading parameterization for action policies [, , ], due in large part to their high training stability and ability to represent complex distributions [, , , ].

> Contribution 1 *(*DPPO*)*. We introduce *Diffusion Policy Policy Optimization* (DPPO), a generic framework as well as a set of carefully chosen design decisions for fine-tuning a diffusion-based robot learning policy via popular policy gradient methods [, ] in reinforcement learning.

The literature has already studied improving/fine-tuning diffusion-based policies (*Diffusion Policy*) using RL [, , ], and has applied policy gradient (PG) to fine-tuning non-interactive applications of diffusion models such as text-to-image generation [, , ]. Yet PG methods have been believed to be inefficient in training Diffusion Policy for continuous control tasks [, ]. On the contrary, we show that for a Diffusion Policy pre-trained from expert demonstrations, our methodology for *fine-tuning* via PG updates yields robust, high-performing policies with favorable training behavior.

Figure: Figure 1: We introduce DPPO, *Diffusion Policy Policy Optimization*, that fine-tunes pre-trained Diffusion Policy using policy gradient. DPPO treats the denoising process as a Markov Decision Process (“Diffusion MDP”) allowing the task reward signal (from “Environment MDP“) to easily propagate through Diffusion Policy (Fig. 3). Policy gradient update then leverages the tractable Gaussian likelihood at each denoising step. Extensive experiments in simulation and hardware show DPPO affords structured exploration and training stability during policy fine-tuning, and the final policy exhibits strong robustness and generalization.
Refer to caption: x1.png

> Contribution 2 *(Demonstration of *DPPO*’s Performance).* We show that for *fine-tuning* a pre-trained Diffusion Policy, DPPO yields consistent and marked improvements in training stability and often final policy performance in comparison to a range of alternatives, including those based on off-policy Q-learning [, , , ] and weighted regression [, , ], other demo-augmented RL methods [, , ], as well as common policy parameterizations such as Gaussian and Gaussian Mixture models with PG updates.

The above finding might be surprising because PG methods do not appear to take advantage of the unique capabilities of diffusion sampling (e.g., guidance [, ]). Through careful investigative experimentation, however, we find a unique synergy between RL fine-tuning and diffusion-based policies.

> Contribution 3 *(Understanding the mechanism of *DPPO*’s success).* We complement our results with numerous investigative experiments that provide insight into the mechanisms behind DPPO’s strong performance. Compared to other common policy parameterizations, we provide evidence that DPPO engages in *structured exploration* that takes better advantage of the “manifold” of training data, and finds policies that exhibit greater robustness to perturbation.

Through ablations, we further show that our design decisions overcome the speculated limitation of PG methods for fine-tuning Diffusion Policy. Finally, to justify the broad utility of DPPO, we verify its efficacy across both simulated and real environments, and in situations when either ground-truth states or pixels are given to the policy as input.

> Contribution 4 *(Tackling challenging robotic tasks and settings).* We show DPPO is effective in various challenging robotic and control settings, including ones with pixel observations and long-horizon manipulation tasks with sparse reward. We also deploy a policy trained in simulation via DPPO on real hardware in zero-shot, which exhibits a remarkably small sim-to-real gap compared to the baseline.

#### Potential impact beyond robotics.

DPPO is a generic framework that can be potentially applied to fine-tuning diffusion-based models in sequential interactive settings beyond robotics. These include: extending diffusion-based text-to-image generation [, ] to a multi-turn interactive setting with human feedback; drug design/discovery applications [, ] with policy search on the molecular level in feedback with simulators (in the spirit of prior non-diffusion-based drug discovery with RL []); and the adaptation of diffusion-based language modeling [, ] to interactive (e.g. with human feedback []), problem-solving and planning tasks.

Figure: Figure 2: DPPO solves challenging long-horizon manipulation tasks from Furniture-Bench ([]), and allows robust sim-to-real transfer without using any real data (see Section 5.4).
Refer to caption: x2.png

### 1.1 Overview of Approach

Like prior work [, ], our approach unrolls the denoising diffusion process into an MDP in which action likelihoods are explicit; from this, we construct a *two-layer MDP* whose outer layer corresponds to the environment MDP and inner layer to the denoising MDP.
Given the challenges of RL training, we also present a number of design decisions tailored to DPPO that are crucial for enabling DPPO’s performance in challenging robotic settings, discussed in Contributions 2 and 4, including:

- •
We apply *Proximal Policy Optimization* (PPO) [] to the two-layer MDP. We show how to efficiently estimate the advantage function for the PPO update.
- •
We show that for fine-tuning tasks,
it often suffices to fine-tune only the last few steps of the denoising process, or fine-tune the Denoising Diffusion Implicit Model (DDIM) [] instead.
- •
We propose modifications to the diffusion noise schedule to ensure stable training and adequate exploration, whilst also leveraging the natural stochasticity of diffusion models.

Our method is formally detailed in Section 4.
As noted above, we conduct extensive experiments demonstrating both the success of DPPO in Section 5 and potential explanations thereof in Section 6.

## 2 Related Work

#### Policy optimization and its application to robotics.

Policy optimization methods update an explicit representation of an RL policy — typically parameterized by a neural network — by taking gradients through action likelihoods. Following the seminal policy gradient (PG) method [, ], there have been a range of algorithms that further improve training stability and sample efficiency such as DDPG [] and PPO [].
PG methods have been broadly effective in training robot policies [, , , ], largely due to their training stability with high-dimensional continuous action spaces, as well as their favorable scaling with parallelized simulated environments.
Given the challenges of from-scratch exploration in long-horizon tasks, PG has seen great success in fine-tuning a baseline policy trained from demonstrations [, , ]. Our experiments find DPPO performing on-policy PG often achieves stronger final performance in manipulation tasks, especially ones with long horizon and high-dimensional action space, than off-policy Q-learning methods [, , ]. See Section A.1 for extended discussions on PG fine-tuning of robot policies and related methods.

#### Learning and improving diffusion-based policies.

Diffusion-based policies [, , , , , , ] have shown recent success in robotics and decision making applications. Most typically, these policies are trained from human demonstrations through a supervised objective, and enjoy both high training stability and strong performance in modeling complex and multi-modal trajectory distributions.

As demonstration data are often limited and/or suboptimal, there have been many approaches proposed to improve the performance of diffusion-based policies. One popular approach has been to guide the diffusion denoising process using objectives such as reward signal or goal conditioning [, , , , ].
More recent work has explored the possibilities of techniques including Q-learning and weighted regression, either from purely offline estimation [, , ], and/or with online interaction [, , , ]. See Section A.2 for detailed descriptions of these methods.

#### Policy gradient through diffusion models.

RL techniques have been used to fine-tune diffusion models such as ones for text-to-image generation [, , , ]. Black et al. [] treat the denoising process as an MDP and apply PPO updates.
We build upon these earlier findings by embedding the denoising MDP into the environmental MDP of the dynamics in control tasks, forming a two-layer “Diffusion Policy MDP”. Though Psenka et al. [] have already shown how PG can be taken through Diffusion Policy by propagating PG through both MDPs, they conjecture that it is likely to be ineffective due to large action variance caused by the increased effective horizon induced from the denoising steps.
Our results contravene this supposition for diffusion-based policies in the fine-tuning setting.

## 3 Preliminaries

#### Markov Decision Process.

We consider a *Markov Decision Process* (MDP)(^1^11More generally, we can view our environment as a Partially Observed Markov Decision Process (POMDP) where the agent’s actions depend on observations $o$ of the states $s$ (e.g., action from pixels). Our implementation applies in this setting, but we omit additional observations from the formalism to avoid notional clutter.) $\mathcal{M}_{\textsc{Env}}:=(\mathcal{S},\mathcal{A},P_{0},P,R)$ with states $s\in\mathcal{S}$, actions $a\in\mathcal{A}$, initial state distribution $P_{0}$, transition probabilities $P$, and reward $R$.

At each timestep $t$, the agent (e.g., robot) observes the state $s_{t}\in\mathcal{S}$, takes an action $a_{t}\sim\pi(a_{t}\mid s_{t})\in\mathcal{A}$, transitions to the next state $s_{t+1}$ according to $s_{t+1}\sim P(s_{t+1}\mid s_{t},a_{t})$ while receiving the reward $R(s_{t},a_{t})$(^2^22For simplicity, we overload $R(\cdot,\cdot)$ to denote both the random variable reward and its distribution.).
Fixing the MDP $\mathcal{M}_{\textsc{Env}}$, we let $\mathbb{E}^{\pi}$ (resp. $\operatorname{\mathbb{P}}^{\pi}$) denote the expectation (resp. probability distribution) over trajectories
$(s_{0},a_{0},...,s_{T},a_{T})$ with length $T+1$, with initial state distribution $s_{0}\sim P_{0}$ and transition operator $P$. We aim to train a policy to optimize the cumulative reward, discounted by a function $\gamma(\cdot)$:
:

$$ $\displaystyle\mathcal{J}(\pi_{\theta})=\mathbb{E}^{\pi_{\theta},P_{0}}\left[ \sum_{t\geq 0}\gamma(t)R(s_{t},a_{t})\right].$ (3.1) $$

#### Policy optimization.

The *policy gradient method* (e.g., Reinforce []) allows for improving policy performance by approximating the gradient of this objective w.r.t. the policy parameters:

$$ $\displaystyle\nabla_{\theta}\mathcal{J}(\pi_{\theta})=\mathbb{E}^{\pi_{\theta} ,P_{0}}\left[\sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})r_{t}(s _{t},a_{t})\right],\quad r_{t}(s_{t},a_{t}):=\sum_{\tau\geq t}\gamma(\tau)R(s_ {\tau},a_{\tau}),$ (3.2) $$

where $r_{t}$ is the discounted cumulative future reward from time $t$ (more generally, $r_{t}$ can be replaced by a Q-function estimator []), $\gamma$ is the discount factor that depends on the time-step, and $\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})$ denotes the gradient of the logarithm of the *likelihood* of $a_{t}\mid s_{t}$. To reduce the variance of the gradient estimation, a state-value function $\hat{V}^{\pi_{\theta}}(s_{t})$ can be learned to approximate $\mathbb{E}[r_{t}]$. The estimated advantage function $\hat{A}^{\pi_{\theta}}(s_{t},a_{t}):=r_{t}(s_{t},a_{t})-\hat{V}^{\pi_{\theta}}
(s_{t})$ substitutes $r_{t}(s_{t},a_{t})$.

#### Diffusion models.

A denoising diffusion probabilistic model (DDPM) [, , ] represents a continuous-valued data distribution $p(\cdot)=p(x^{0})$ as the reverse denoising process of a forward noising process $q(x^{k}|x^{k-1})$ that iteratively adds Gaussian noise to the data. The reverse process is parameterized by a neural network $\varepsilon_{\theta}(x_{k},k)$, predicting the added noise $\varepsilon$ that converts $x_{0}$ to $x_{k}$ [].
Sampling starts with a random sample $x^{K}\sim\mathcal{N}(0,\mathrm{I})$ and iteratively generates the denoised sample:

$$ $\displaystyle x^{k-1}\sim p_{\theta}(x^{k-1}|x^{k}):=\mathcal{N}(x^{k-1};\mu_{ k}(x^{k},\varepsilon_{\theta}(x^{k},k)),\sigma_{k}^{2}\mathrm{I}).$ (3.3) $$

Above, $\mu_{k}(\cdot)$ is a fixed function, independent of $\theta$, that maps $x^{k}$ and predicted $\varepsilon_{\theta}$ to the next mean, and $\sigma_{k}^{2}$ is a variance term that abides by a fixed schedule from $k=1,\dots,K$. We refer the reader to Chan [] for an in-depth survey.

#### Diffusion models as policies.

*Diffusion Policy* (DP; see Chi et al. []) is a policy $\pi_{\theta}$ parameterized by a DDPM which takes in $s$ as a conditioning argument, and parameterizes $p_{\theta}(a^{k-1}\mid a^{k},s)$ as in (3.3).
DPs can be trained via behavior cloning by
fitting the conditional noise prediction $\varepsilon_{\theta}(a^{k},s,k)$ to predict the added noise. Notice that unlike more standard policy parameterizations such as unimodal Gaussian policies, DPs do not maintain an explicit likelihood of $p_{\theta}(a^{0}\mid s)$. In this work, we adopt the common practice of training DPs to predict an action chunk — a sequence of actions a few time steps (denoted $T_{p}$) into the future — to promote temporal consistency. Policy executes $T_{a}\leq T_{p}$ steps of the predicted chunk before the next prediction. From now on $a\in\mathcal{A}$ denotes the entire executed action chunk. For fair comparison, our diffusion and non-diffusion baselines use the same $T_{a}$ unless noted.

## 4 DPPO : Diffusion Policy Policy Optimization

As noted in Section 1 and Section 2, there has been much attention devoted to fine-tuning Diffusion Policy for improved performance [, ], focusing primarily on off-policy Q-learning [, ] and/or weighted regression. DPPO takes a different approach: differentiate through action likelihood and apply a policy gradient (PG) update like (3.2).

### 4.1 A Two-Layer “Diffusion Policy MDP”

Figure: Figure 3: We treat the denoising process in Diffusion Policy as an MDP, and the whole environment episode can be considered as a chain of such MDPs. Now the entire chain (“Diffusion Policy MDP”, $\mathcal{M}_{\textsc{DP}}$) involves a Gaussian likelihood at each (denoising) step and thus can be optimized with policy gradient. Blue circle denotes the state and red circle denotes the action in $\mathcal{M}_{\textsc{DP}}$.
Refer to caption: x3.png

As observed in Black et al. [] and [], a denoising process can be represented as a multi-step MDP in which policy likelihood of each denoising step can be obtained directly. We extend this formalism by embedding the Diffusion MDP into the environmental MDP, obtaining a larger “Diffusion Policy MDP” denoted $\mathcal{M}_{\textsc{DP}}$, visualized in Fig. 3. Below, we use the notation $\updelta$ to denote a Dirac distribution and $\otimes$ to denote a product distribution.

Recall the environment MDP $\mathcal{M}_{\textsc{Env}}:=(\mathcal{S},\mathcal{A},P_{0},P,R)$ in Section 3. The Diffusion MDP
$\mathcal{M}_{\textsc{DP}}$ uses indices $\bar{t}(t,k)=tK+(K-k-1)$ corresponding to $(t,k)$, which increases in $t$ but (to keep the indexing conventions of diffusion) *decreases* lexicographically with $K-1\geq k\geq 0$. The states, actions and rewards are

$$ $\displaystyle\bar{s}_{\bar{t}(t,k)}=(s_{t},a^{k+1}_{t}),\quad\bar{a}_{\bar{t}( t,k)}=a^{k}_{t},\quad\bar{R}_{\bar{t}(t,k)}(\bar{s}_{\bar{t}(t,k)},\bar{a}_{ \bar{t}(t,k)})=\begin{cases}0&k>0\\ R(s_{t},a_{t}^{0})&k=0\end{cases},$ (4.1) $$

where the bar-action at $\bar{t}(t,k)$ is the action $a_{t}^{k}$ after one denoising step.
Reward is only given at times corresponding to when $a_{t}^{0}$ is taken. The initial state distribution is $\bar{P}^{0}=P_{0}\otimes\mathcal{N}(0,\mathbf{I})$, corresponding to $s_{0}\sim P_{0}$ is the initial distribution from the environmental MDP and $a_{0}^{K}\sim\mathcal{N}(0,\mathbf{I})$ independently. Finally, the transitions are

$$ $\displaystyle\bar{P}(\bar{s}_{\bar{t}+1}\mid\bar{s}_{\bar{t}},\bar{a}_{\bar{t} })=\begin{cases}(s_{t},a^{k}_{t})~{}\sim\updelta_{(s_{t},{a}_{t}^{k})}&\bar{t} =\bar{t}(t,k),k>0\\ (s_{t+1},a^{K}_{t+1})\sim P(s_{t+1}\mid s_{t},a_{t}^{0})\otimes\mathcal{N}(0, \mathbf{I})&\bar{t}=\bar{t}(t,k),k=0\\ \end{cases}.$ (4.2) $$

That is, the transition moves the denoised action $a_{t}^{k}$ at step $\bar{t}(t,k)$ *into the next state* when $k>0$, or otherwise progresses the environment MDP dynamics with $k=0$. The pure noise $a_{t}^{K}$ is considered part of the *environment* when transitioning at $k=0$. In light of (3.3), the policy in $\mathcal{M}_{\textsc{DP}}$ takes the form

$$ $\displaystyle\bar{\pi}_{\theta}(\bar{a}_{\bar{t}(t,k)}\mid\bar{s}_{\bar{t}(t,k )})=\pi_{\theta}(a_{t}^{k}\mid a_{t}^{k+1},s_{t})=\mathcal{N}(a_{t}^{k};\mu(a_ {t}^{k+1},\varepsilon_{\theta}(a_{t}^{k+1},k+1,s_{t})),\sigma_{k+1}^{2}\mathrm {I}).$ (4.3) $$

Fortunately, (4.3) is a *Gaussian likelihood*, which can be evaluated analytically and is amenable to the policy gradient updates (see also [] for an alternative derivation):

$$ $\displaystyle\nabla_{\theta}\bar{}\mathcal{J}(\bar{\pi}_{\theta})=\mathbb{E}^{ \bar{\pi}_{\theta},\bar{P},\bar{P}^{0}}\left[\sum_{\bar{t}\geq 0}\nabla_{ \theta}\log\bar{\pi}_{\theta}(\bar{a}_{\bar{t}}\mid\bar{s}_{\bar{t}})\bar{r}( \bar{s}_{\bar{t}},\bar{a}_{\bar{t}})\right],\quad\bar{r}(\bar{s}_{\bar{t}}, \bar{a}_{\bar{t}}):=\sum_{\tau\geq\bar{t}}\gamma(\tau)\bar{R}(\bar{s}_{\tau}, \bar{a}_{\tau}).$ (4.4) $$

Evaluating the above involves sampling through the denoising process, which is the usual “forward pass” that samples actions in Diffusion Policy; as noted above, the inital state can be sampled from the enviroment via $\bar{P}^{0}=P_{0}\otimes\mathcal{N}(0,\mathbf{I})$, where $P_{0}$ is from the environment MDP.

### 4.2 Instantiating DPPO with Proximal Policy Optimization

We apply Proximal Policy Optimization (PPO) [, , , ], a popular improvement of the vanilla policy gradient update.

###### Definition 4.1 (Generalized PPO, clipping variant) .

Consider a general MDP. Given an advantage estimator $\hat{A}(s,a)$, the PPO update [] is the sample approximation to

$$ $\displaystyle\nabla_{\theta}\ \mathbb{E}^{(s_{t},a_{t})\sim\pi_{\theta_{\text{ old}}}}\min\Big{(}\hat{A}^{\pi_{\theta_{\text{old}}}}(s_{t},a_{t})\frac{\pi_{ \theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})},\ \hat{A }^{\pi_{\theta_{\text{old}}}}(s_{t},a_{t})\operatorname{clip}\big{(}\frac{\pi_ {\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})},1- \varepsilon,1+\varepsilon\big{)}\Big{)},$ (4.5) $$

where $\varepsilon$, the clipping ratio, controls the maximum magnitude of the policy updated from the previous policy.

We instantiate PPO in our diffusion MDP with $(s,a,t)\leftarrow(\bar{s},\bar{a},\bar{t})$. Our advantage estimator takes a specific form that respects the two-level nature of the MDP:
let $\gamma_{\textsc{env}}\in(0,1)$ be the environment discount and $\gamma_{\textsc{denoise}}\in(0,1)$ be the denoising discount. Consider the environment-discounted return:

$$ $\displaystyle\bar{r}(\bar{s}_{\bar{t}},\bar{a}_{\bar{t}}):=\sum_{t^{\prime} \geq t}\gamma_{\textsc{env}}^{t}\bar{r}(\bar{s}_{\bar{t}(t^{\prime},0)},\bar{a }_{\bar{t}(t^{\prime},0)}),\quad\bar{t}=\bar{t}(t,k),$ (4.6) $$

since $\bar{R}(\bar{t})=0$ at $k>0$. This fact also obviates the need of estimating the value at $k>1$ and allows us to use the following denoising-discounted advantage estimator(^3^33In practice, we use Generalized Advantage Estimation (GAE, Schulman et al. []) that better balances variance and bias in estimating the advantage. We present the simpler form here.):

$$ $\hat{A}^{\pi_{\theta_{\text{old}}}}(\bar{s}_{\bar{t}},\bar{a}_{\bar{t}}):= \gamma_{\textsc{denoise}}^{k}\left(\bar{r}(\bar{s}_{\bar{t}},\bar{a}_{\bar{t}} )-\hat{V}^{\bar{\pi}_{\theta_{\text{old}}}}(\bar{s}_{\bar{t}(t,0)})\right)$ (4.7) $$

The denoising-discounting has the effect of downweighting the contribution of noisier steps (larger $k$) to the policy gradient (see study in Section B.1).
Lastly, we choose the value estimator to *only depend* on the “$s$” component of $\bar{s}$:

$$ $\displaystyle\hat{V}^{\bar{\pi}_{\theta_{\text{old}}}}(\bar{s}_{\bar{t}(t,0)}) :=\tilde{V}^{\bar{\pi}_{\theta_{\text{old}}}}(s_{t}),$ (4.8) $$

which we find leads to more efficient and stable training compared to also estimating the value of applying the denoised action $a_{t}^{k=1}$ (part of $\bar{s}_{\bar{t}(t,0)}$)
as shown in Section B.1.

###### Definition 4.1 (Generalized PPO, clipping variant) .

### 4.3 Best Practices for DPPO

#### Fine-tune only the last few denoising steps.

Diffusion Policy often uses up to $K=100$ denoising steps with DDPM to better capture the complex data distribution of expert demonstrations. With DPPO, we can choose to fine-tune only a subset of the denoising steps instead, e.g., the last $K^{\prime}$ steps. Experimental results in Section B.1 shows this speeds up DPPO training and reduces GPU memory usage without sacrificing the final performance. Instead of fine-tuning the pre-trained model weights $\theta$, we make a copy $\theta_{\textsc{ft}}$ — $\theta$ is frozen and used for the early denoising steps, while $\theta_{\textsc{ft}}$ is used for the last $K^{\prime}$ steps and updated with DPPO.

#### Fine-tune DDIM sampling.

Instead of fine-tuning all $K$ or the last few steps of the DDPM, one can also apply Denoising Diffusion Implicit Model (DDIM) [] during fine-tuning, which greatly reduces the number of sampling steps $K^{\textsc{DDIM}}\ll K$, e.g., as few as 5 steps, and thus potentially improves DPPO efficiency as fewer steps are fine-tuned:

$$ $x^{k-1}\sim p_{\theta}^{\textsc{DDIM}}(x^{k-1}|x^{k}):=\mathcal{N}(x^{k-1};\mu ^{\textsc{DDIM}}(x^{k},\varepsilon_{\theta}(x^{k},k)),\eta\sigma_{k}^{2} \mathrm{I}),\quad k=K^{\textsc{DDIM}},...,0.$ (4.9) $$

Although DDIM is typically used as a deterministic sampler by setting $\eta=0$ in (4.9), we can use $\eta>0$ for fine-tuning in order to provide exploration noise and avoid calculating a Gaussian likelihood with a Dirac distribution. In practice, we set $\eta=1$ for training (equivalent to applying DDPM []) and then $\eta=0$ for evaluation. We use DDIM sampling for our pixel-based experiments and long-horizon furniture assembly tasks, where the efficiency improvements are much desired.

#### Diffusion noise scheduling.

We use the cosine schedule for $\sigma_{k}$ introduced in [], which was originally annealed to a small value on the order of $1\text{e}{-4}$ at $k=0$. In DPPO, the values of $\sigma_{k}$ also translate to the exploration noise that is crucial to training efficiency. Empirically, we find that clipping $\sigma_{k}$ to a higher minimum value (denoted $\sigma^{\text{exp}}_{\text{min}}$, e.g., $0.01-0.1$) when sampling actions helps exploration (see study in Section B.1). Additionally we clip $\sigma_{k}$ to be at least $0.1$ (denoted $\sigma^{\text{prob}}_{\text{min}}$) when evaluating the Gaussian likelihood $\log\bar{\pi}_{\theta}(\bar{a}_{\bar{t}}|\bar{s}_{\bar{t}})$, which improves training stability by avoiding large magnitude.

#### Network architecture.

We study both Multi-layer Perceptron (MLP) and UNet [] as the policy heads in Diffusion Policy. MLP offers a simpler setup, and we find that it generally fine-tunes more stably with DPPO. Meanwhile, UNet, only applying convolutions to $a_{t}^{k}$, has the benefit of allowing fine-tuning with $T_{a}<T_{p}$, e.g., $T_{p}=16$ and $T_{a}=8$. We find that DPPO benefits from pre-training with larger $T_{p}$ (better prediction) and fine-tuning with smaller $T_{a}$ (more amenable to policy gradient) (^4^44With fully-connected layers in MLP, empirically we find that fine-tuning with $T_{a}<T_{p}$ leads to training instability.).

## 5 Performance Evaluation of DPPO

We study the performance of DPPO in popular RL and robotics benchmarking environments.
We compare to other RL methods for fine-tuning a Diffusion Policy (Section 5.1), to other demo-augmented RL methods (Section 5.2), to other policy parameterizations with PG updates (Section 5.3), and then in multi-stage manipulation tasks including hardware evaluation (Section 5.4), and conclude with ablations (Section 5.5). While our evaluations focus primarily on fine-tuning, we also present training-from-scratch results in Appendix B.
Wall-clock times are reported and discussed in Appendix C; they are roughly comparable (often faster) than other diffusion-based RL baselines, much faster than other demo-augmented RL baselines, though can be up to $2\times$ slower than other policy parameterizations.
Full choices of training hyperparameters and additional training details are presented in Appendix D.

#### Environments: OpenAI Gym.

We first consider three OpenAI Gym locomotion benchmarks [] common in the RL literature: {Hopper-v2, Walker2D-v2, HalfCheetah-v2}. All policies are pre-trained with the full medium-level datasets from D4RL [] with state input and action chunk size $T_{a}=4$. We use the original dense reward setup in fine-tuning.

#### Environments: Franka Kitchen.

We also consider the Franka-Kitchen environment first introduced in [] involving a Franka arm completing four countertop tasks. We use the three datasets from D4RL []: {Kitchen-Complete-v0, Kitchen-Partial-v0, Kitchen-Mixed-v0}, containing demonstrations of varying levels of completeness. State input is considered and action chunk size $T_{a}=4$ is used for DPPO. We use the original sparse reward setup in fine-tuning.

#### Environments: Robomimic.

Next we consider four simulated robot manipulation tasks from the Robomimic benchmark [], {Lift, Can, Square, Transport}, ordered in increasing difficulty. These tasks are more representative of real-world robotic tasks, and Square and Transport (Fig. 4) are considered very challenging for RL training.
Both state and pixel policy input are considered.
State-based and pixel-based policies are pre-trained with 300 and 100 demonstrations provided by Robomimic, respectively. We mostly consider the noisier Multi-Human (MH) data from Robomimic but also consider the cleaner Proficient-Human (PH) data in some experiments. We consider $T_{a}=4$ for Can, Lift, and Square, and $T_{a}=8$ for Transport.
They are then fine-tuned with sparse reward equal to $1$ upon task completion.

#### Environments: Furniture-Bench & real furniture assembly.

Finally, we demonstrate solving longer-horizon, multi-stage robot manipulation tasks from the Furniture-Bench [] benchmark. We consider three simulated furniture assembly tasks, {One-leg, Lamp, Round-table}, shown in Fig. 4 and described in detail in Section D.9.
We consider two levels of randomness over initial state distribution, Low and Med, defined by the benchmark.
All policies are pre-trained with 50 human demonstrations collected in simulation and $T_{a}=8$.
They are then fine-tuned with sparse (indicator of task stage completion) reward.
We also evaluate the zero-shot sim-to-real performance with One-leg.

Figure: Figure 4: Long-horizon robot manipulations tasks including (left) the bimanual Transport from Robomimic and (right) Furniture-Bench tasks (full rollouts visualized in Fig. A10).
Refer to caption: x4.png

### 5.1 Comparison to diffusion-based RL algorithms

We compare DPPO to an extensive list of RL methods for fine-tuning diffusion-based policies. Baseline names are color-coordinated with their plot colors. Two methods, DRWR, DAWR, are *our own, novel* baselines that are based on reward-weighted regression (RWR, Peters and Schaal []) and advantage-weighted regression (AWR, Peng et al. []). The remaining methods, DIPO [], IDQL [], DQL [], and QSM [], are existing in the literature. Among them, QSM and DIPO are proposed specifically for training-from-scratch, while IDQL and DQL can be more suitable to online fine-tuning. We evaluate on the three OpenAI Gym tasks and the four Robomimic tasks with state input; detailed descriptions of all baselines and training details are in Section D.3.

Overall, DPPO performs consistently, exhibits great training stability, and enjoys strong fine-tuning performance across tasks. In the Gym tasks (Figure 5, top row), IDQL and DIPO exhibit competitive performance, while the other methods often perform worse and train less stably. Baselines also tend to exhibit performance drop at the beginning of fine-tuning, mostly likely due to misestimating Q values of the pre-trained policy. DPPO is the strongest performer in the Robomimic tasks (Figure 5, bottom row), especially in the challenging Transport tasks. IDQL and, surprisingly, DRWR, are strong baselines in {Lift, Can, Square} but underperforms in Transport, while all other baselines fare worse still. We postulate that the other baselines, all performing off-policy updates and propagating gradients from the imperfect Q function to the actor, suffers from even greater training instability in sparse-reward Robomimic tasks given the continuous action space plus large action chunk sizes
(see furtuer studies in Section B.1).

Figure: Figure 5: Comparing to other diffusion-based RL algorithms. Top row: Gym tasks [] averaged over five seeds. Bottow row: Robomimic tasks [], averaged over three seeds, with state observation. DPPO curves are slightly thicker for better visibility.
Refer to caption: x5.png

### 5.2 Comparison to other demo-augmented RL algorithms

We compare DPPO with recently proposed RL methods for training robot policies (not necessarily diffusion-based) leveraging offline data, including RLPD [], Cal-QL [], and IBRL []. These methods add expert data in the replay buffer and performs off-policy updates (IBRL and Cal-QL also pretrain with behavior cloning and offline RL objectives, respectively), which significantly improves sample efficiency vs. DPPO in HalfCheetah-v2 (Fig. 6, top left).

Among the Franka-Kitchen settings (Fig. 6, top right), we find RLPD and IBRL fail to learn well especially with noisier demonstrations from Kitchen-Partial-v0 and Kitchen-Mixed-v0. Cal-QL achieves competitive performance but DPPO still achieves overall the best performance especially with Kitchen-Complete-v0. We note that DPPO, not using any expert data during fine-tuning, can be sensitive to the pre-training performance; we find the incomplete demonstrations in Kitchen-Partial-v0 and Kitchen-Mixed-v0 cause challenge in fully modeling the multi-modality of the data even with diffusion parameterization and prevent DPPO from achieving (near-)perfect fine-tuning performance.

Nonetheless, we believe the expert demonstrations from Robomimic are most reflective of pre-training plus fine-tuning in robotics as all demonstrations complete the task despite the varying quality. Fig. 6 bottom row shows the performance of DPPO and baselines using either cleaner PH or noisier MH data in Can and Square; DPPO exhibits much stronger final performance. RLPD and Cal-QL fail to learn at all and IBRL saturates at lower success levels.
DPPO also runs significantly faster in wall-clock time than the baselines as it leverages sampling from highly parallelized environments(^5^55Off-policy methods (baselines) usually cannot fully leverage parallelized sampling as the policy is updated less often (e.g., 50 updates per 50 samples instead of 1 update per 1 sample) and the performance can be affected.); thus we cap the number of samples at 1e6 for the baselines in Robomimic, also since their performance saturates.

Figure: Figure 6: Comparing to other demo-augmented RL algorithms. Top left: HalfCheetah-v2 task [] averaged over five seeds. Top right: Franka-Kitchen task [] averaged over three seeds. Bottom row: Robomimic tasks [] averaged over three seeds with state observation, using either Proficient-Human (PH) or Multi-Human (MH) data.
Refer to caption: x6.png

### 5.3 Comparison to other policy parameterizations

We compare DPPO with popular RL policy parameterizations: unimodal Gaussian with diagonal covariance [] and Gaussian Mixture Model (GMM []), using either MLPs or Transformers [], and also fine-tuned with the PPO objective. We compare these to DPPO-MLP and DPPO-UNet, which use either MLP or UNet as the network backbone. We evaluate on the four tasks from Robomimic (Lift, Can, Square, Transport) with both state and pixel input. With state input, DPPO pre-trains with 20 denoising steps and then fine-tunes the last 10. With pixel input, DPPO pre-trains with 100 denoising steps and then fine-tunes 5 DDIM steps.

Figure: Figure 7: Comparing to other policy parameterizations in the more challenging Square and Transport tasks from Robomimic, with state (left) or pixel (right) observation. Results are averaged over three seeds.
Refer to caption: x7.png

Fig. 7 display results for the more challenging Square and Transport — we defer the results in Lift and Can to Fig. A8.
With state input, DPPO outperforms Gaussian and GMM policies, with faster convergence to $\sim$100% success rate in Lift and Can, and greater final performance on Square and the challenging Transport, where it reaches $>90\%$. UNet and MLP variants perform similarly, with the latter training somewhat more rapidly. With pixel inputs, we use a Vision-Transformer-based (ViT) image encoder introduced in Hu et al. [] and an MLP head and compare the resulting variants DPPO-ViT-MLP and Gaussian-ViT-MLP (we omit GMM due to poor performance in state-based training). While the two are comparable on Lift and Can,
DPPO trains more quickly and to higher accuracy on Square, and *drastically outperforms* on Transport, whereas Gaussian does not improve from its 0% pre-trained success rate. To our knowledge, DPPO is *the first RL algorithm* to solve Transport from either state or pixel input to high (>50%) success rates.

### 5.4 Evaluation on Furniture-Bench, and sim-to-real transfer

Figure: Figure 8: (Top) DPPO vs. Gaussian-MLP baseline in simulated Furniture-Bench tasks. Results are averaged over three seeds. (Bottom) Sim-to-real transfer results in One-leg.
Refer to caption: x8.png

Here we evaluate DPPO on the long-horizon manipulation tasks from Furniture-Bench []. We compare DPPO to Gaussian-MLP, the overall most effective baseline from Section 5.3.
Fig. 8 (top row) shows the evaluation success rate over fine-tuning iterations. DPPO exhibits strong training stability and improves policy performance in all six settings. Gaussian-MLP collapses to zero success rate in all three tasks with Med randomness (except for one seed in Lamp) and Round-table with Low randomness.

Note that we are only using 50 human demonstrations for pre-training; we expect DPPO can leverage additional human data (better state space coverage) to further improve in Med, which is corroborated by ablation studies in Section B.2 examining the effect of pre-training data on DPPO.

Figure: Figure 9: Qualitative comparison of pre-trained vs. fine-tuned DPPO policies in hardware evaluation. (A) Successful rollout with the pre-trained policy. (B) Failed rollout with the pre-trained policy due to imprecise insertion. (C) Successful rollout with the fine-tuned policy. (D) Successful rollout with the fine-tuned policy exhibiting corrective behavior.
Refer to caption: x9.png

#### Sim-to-real transfer.

We evaluate DPPO and Gaussian policies trained in the simulated One-leg task on physical hardware zero-shot (i.e., no real data fine-tuning / co-training) over 20 trials. Both policies run at 10Hz. Please see additional simulation training and hardware details in Section D.9.
Fig. 8 (bottom row) shows simulated and hardware success rates after pre-training and fine-tuning. Notably, DPPO improves the real-world performance to 80% (16 out of 20 trials). Though the Gaussian policy achieves a high success rate in simulation after fine-tuning (88%), it fails entirely on hardware (0%). Supplemental video suggests it exhibits volatile and jittery behavior. For stronger comparison, we also fine-tune the Gaussian policy with an auxiliary behavior-cloning loss [] such that the fine-tuned policy is encouraged to stay close to the base policy. However, this limits fine-tuning and only leads to 53% success rate in simulation and 50% in reality.

Qualitatively,
we find fine-tuned policies to be more robust and exhibit more corrective behaviors than pre-trained-only policies, especially during the insertion stage of the task; such behaviors are visualized in Fig. 2 and Fig. 9 shows representative rollouts on hardware.
Overall, these results demonstrate the strong sim-to-real capabilities of DPPO; Section 6 provides a conjectural mechanism for why this may be the case.

### 5.5 Summary of ablation findings

We conduct extensive ablation studies in Section B.1. Our main findings include: (1) for challenging tasks, using a value estimator which depends on environment state but is *independent of denoised action* is crucial for performance; we conjecture that this is related to the high stochasticity of Diffusion Policy; (2) there is a sweet spot for clipping the denoising noise level for DPPO exploration, trading off between too little exploration and too much action noise; (3) DPPO is resilient to fine-tuning fewer-than-$K$ denoising steps, yielding improved runtime and comparable performance; (4) DPPO yields improvements over Gaussian-MLP baselines for varying levels of expert demonstration data, and achieves comparable final performance and sample efficiency when training from scratch in Gym environments.

## 6 Understanding the performance of DPPO

We study the factors contributing to DPPO’s improvements in performance over the popular Gaussian and GMM methods introduced in Section 5.3.
Our findings highlight three major contributing factors:

- (1)
DPPO induces structured exploration near the pre-training data manifold [].
- (2)
DPPO updates the action distribution progressively through the multi-step denoising process, which can be flexible and robust to policy collapse.
- (3)
DPPO leads to fine-tuned policies robust to perturbations in dynamics and initial state distribution.

We use the Avoid environment from D3IL benchmark [], where a robot arm needs to reach the other side of the table while avoiding an array of obstacles (Fig. 10, top-left). The action space is the 2D target location of the end-effector.
D3IL provides a set of expert demonstrations that covers different possible paths to the goal line — we consider three subsets of the demonstrations, M1, M2, and M3 in Fig. 10, each with two distinct modes. We choose such relatively simple setups with only two modes in each setting such that Gaussian (with exploration noise)(^6^66Without noise, Gaussian policy is fully deterministic and cannot capture the two modes.) and GMM can fit the expert data distribution reasonably well, allowing fair comparisons in fine-tuning.

We pre-train MLP-based Diffusion, Gaussian, and GMM policies (action chunk size $T_{a}=4$ unless noted) with the demonstrations. For fine-tuning, we assign (sparse) reward when the robot reaches the goal line from the topmost mode. Gaussian and GMM policies are also fine-tuned with the PPO objective.

Figure: Figure 10: (Left) We use the Avoid environment from D3IL benchmark [] to visualize the DPPO’s exploration tendencies. We design the task of always reaching the green goal line from the topmost mode. (Right) Structured exploration. We show sampled trajectories at the *first iteration of fine-tuning* for DPPO, Gaussian, and GMM after pre-training on three sets of expert demonstrations, M1, M2, and M3.
Refer to caption: x10.png

#### Benefit 1: Structured, on-manifold exploration.

Fig. 10 (right) shows the sampled trajectories (with exploration noise) from DPPO, Gaussian, and GMM during the first iteration of fine-tuning. DPPO explores in wide coverage around the expert data manifold, whereas Gaussian generates less structured exploration noise (especially in M2) and GMM exhibits narrower coverage.
Moreover, the combination of diffusion parameterization with the denoising of *action chunks* means that policy stochasticity in DPPO is structured in both action dimension and time horizon. Quantitatively, Fig. A7 in Appendix shows DPPO achieves greater fine-tuning efficiency than Gaussian and GMM in all settings if a sufficient number of denoising steps is fine-tuned, consistent with the findings in Section 5.3 and Section 5.5.

It is possible, however, that the on-manifold exploration we observe with DPPO hinders fine-tuning when aggressive, unstructured exploration is desired. We conjecture this is the case in the Lamp environment with Low randomness, in which DPPO slightly underperforms the Gaussian baseline (Fig. 8). Moreover, we do not observe that DPPO is substantially better (nor is it any worse) than Gaussian in exploration from-scratch (see Appendix B). Thus, we anticipate that this structured, on-manifold exploration confers the greatest benefit when pre-training provides sufficient coverage of relevant success modes. In particular, we believe that this makes DPPO uniquely suited for fine-tuning large Diffusion Policy pre-trained on multiple tasks [], as it may exhibit sufficient mode coverage given training data diversity.

#### Benefit 2: Training stability from multi-step denoising process.

In Fig. 11 (left), we run fine-tuning after pre-training with M2 and *attempt to de-stabilize fine-tuning* by gradually adding noise to the action during the fine-tuning process (see Section D.10 for details). We find that Gaussian and GMM’s performance both collapse, while with DPPO, the performance is robust to the noise if at least four denoising steps are used. This property also allows DPPO to apply significant noise to the sampled actions, simulating an imperfect low-level controller to facilitate sim-to-real transfer in Section 5.4.
In Fig. 11 (right), we also find DPPO enjoys greater training stability when fine-tuning long action chunks, e.g., up to $T_{a}=16$, while Gaussian and GMM can fail to improve at all.

Figure: Figure 11: Training stability. Fine-tuning performance (averaged over five seeds, standard deviation not shown) after pre-training with M2. (Left) Noise is injected into the applied actions after a few training iterations. (Right) The action chunk size $T_{a}$ is varied.
Refer to caption: x11.png

Fig. 12 visualizes how DPPO affects the multi-step denoising process. Over fine-tuning iterations, the action distribution gradually converges through the denoising steps — the iterative refinement is largely preserved, as opposed to, e.g., “collapsing” to the optimal actions at the first fine-tuned denoising step or the final one. We postulate this contributes to the training stability of DPPO.

Figure: Figure 12: Preserving the iterative refinement. The 2D actions from 50 trajectories at the branching point *through fine-tuning* iterations after pre-training with M2. For DPPO, we also visualize the action distribution through the final denoising steps at each fine-tuning iteration.
Refer to caption: x12.png

#### Benefit 3: Robust and generalizable fine-tuned policy.

DPPO also generates final policies robust to perturbations in dynamics and the initial state distribution. In Fig. 13, we again add noise to the actions sampled from the fine-tuned policy (no noise applied during training) and find that DPPO policy exhibits strong robustness to the noise compared to the Gaussian policy. DPPO policy also converges to the (near-)optimal path from a larger distribution of initial states. This finding echoes theoretical guarantees that Diffusion Policy, capable of representing complex multi-modal data distribution, can effectively deconvolve noise from noisy states [], a property used in Chen et al. [] to stabilize long-horizon video generation.

Figure: Figure 13: Policy robustness *after fine-tuning*. Green dot / box indicates the initial state region.
Refer to caption: x13.png

## 7 Conclusion and Future Work

We present *Diffusion Policy Policy Optimization* (DPPO) for fine-tuning a pre-trained Diffusion Policy with the policy gradient method. DPPO leverages the sequential nature of the diffusion denoising process and fine-tunes the entire chain of diffusion MDPs. DPPO exhibits structured online exploration, strong training stability, and robustness and generalization at deployment. We demonstrate the efficiency and effectiveness of DPPO fine-tuning in various RL and robotics benchmarks, as well as strong sim-to-real transfer of a DPPO policy in a long-horizon, multi-stage manipulation task.

We believe DPPO will become an important component in the pre-training-plus-fine-tuning pipeline for training general-purpose real-world robotic policies. To this end, we hope in future work to further showcase the promise of DPPO for simulation-to-real transfer [, , , ] in which we fine-tune a vision-based policy that has been pre-trained on a variety of diverse tasks. We expect this pre-training to provide a large and diverse expert data manifold, of which, as we have shown in Section 6, DPPO is well-suited to take advantage for better exploration during fine-tuning. We are also excited to understand how DPPO can fit together with other decision-making tools such as model-based planning [] and decision-making aided by video prediction []; these tools may help address the main limitation of DPPO — its lower sample efficiency than off-policy methods — and unlocking performing practical RL in physical hardware or generative simulation (e.g., video-based [, ]). Finally, as noted in the introduction, we eagerly anticipate applications of DPPO in domains beyond robotics, where diffusion models have shown promise for combinatorial search and sequence modeling [, ].

## Acknowledgments

We would like to thank Lirui Wang and Terry Suh for helpful discussions in the early stage of the project. The authors were partially supported by the Toyota Research Institute (TRI). This article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.

## Appendix A Extended Related Work

### A.1 RL training of robot policies with offline data

Here, we discuss related work in training robot policies using RL augmented with offline data to help RL better explore online in sparse reward settings.

One simple form is to use offline data to pre-train the policy, typically using behavior cloning, and then fine-tune the policy online. This is the approach that DPPO takes. Often, a regularization loss is applied to constrain the fine-tuned policy to stay close to the base policy, leading to natural fine-tuned behavior and often better learning [, , ]. DPPO does not apply regularization at fine-tuning as we find the on-manifold exploration helps DPPO maintain natural behavior after fine-tuning Section 5.4. Another popular approach is to learn a *residual* policy with RL on top of the frozen base policy [, ]. A closer work to ours is Ankile et al. [], which trains a one-step residual non-diffusion policy with on-policy RL on top of a pre-trained chunked diffusion policy. This approach has the benefit of being fully closed-loop but lacks the structured on-manifold exploration of DPPO. Another hybrid approach is from Hu et al. [], which uses pre-trained and fine-tuned policies to sample online experiences.

Another popular line of work, instead of training a base policy using offline data, directly adds the data in the replay buffer for online, off-policy learning in a single stage [, , ]. One recent approach from Ball et al. [], RLPD, further improves sample efficiency from previous off-policy methods incorporating, e.g., critic ensembling. Luo et al. [] demonstrates RLPD solving real-world manipulation tasks (although generally less challenging than ones solved by DPPO). Other approaches including Cal-QL build on offline RL to learn from offline data and then switch to online RL [, , ]. IBRL from Hu et al. [] pre-trains the base policy and samples offline data in fine-tuning.

### A.2 Diffusion-based RL methods

This section discusses related methods that directly train or improve diffusion-based policies with RL methods. The baselines to which we compare in Section 5.1 are discussed below as well, and are highlighted in their corresponding colors. We also refer the readers to Zhu et al. [] for an extensive survey on diffusion models for RL.

Most previous works have focused on the offline setting with a static dataset. One line of work focuses on state trajectory planning and *guiding* the denoising sampling process such that the sampled actions satisfy some desired objectives. Janner et al. [] applies classifier guidance that generates trajectories with higher predicted rewards. Ajay et al. [] introduces classifier-free guidance that avoids learning the value of noisy states. There is another line of work that uses diffusion models as an action policy (instead of state planner) and generally applies Q-learning. DQL [] introduces Diffusion Q-Learning that learns a state-action critic for the final denoised actions and backpropagates the gradient from the critic through the entire Diffusion Policy (actor) denoising chain, akin to the usual Q-learning.
IDQL [], or Implicit Diffusion Q-learning, proposes learning the critic to select the actions at inference time for either training or evaluation while fitting the actor to all sampled actions. Kang et al. [] instead proposes using the critic to re-weight the sampled actions for updating the actor itself, similar to weighted regression baselines DAWR and DRWR introduced in our work. Goo and Niekum [] similarly extracts the policy in the spirit of AWR []. Chen et al. [] trains the critic using value iteration instead based on samples from the actor. Finally, Jackson et al. [] explores using diffusion guidance to move offline data towards the target trajectory distribution.

We note that methods like DQL and IDQL can also be applied in the online setting. A small amount of work also focuses entirely on the online setting. DIPO [] differs from DQL and related work in that it uses the critic to update the sampled actions (“action gradient”) instead of the actor — the actor is then fitted with updated actions from the replay buffer. QSM, or Q-Score Matching [], suggests that optimizing the likelihood of the entire chain of denoised actions can be inefficient (contrary to our findings in the fine-tuning setting) and instead proposes learning the optimal policy by iteratively aligning the gradient of the actor (i.e., score) with the action gradient of the critic. Rigter et al. [] proposes learning a diffusion dynamic model to generate synthetic trajectories for online training of a non-diffusion RL policy.

## Appendix B Additional experimental results

### B.1 Ablation studies on design decisions in DPPO

#### 1. Choice of advantage estimator.

In Section 4.3 we demonstrate how to efficiently estimate the advantage used in PPO updates by learning $\tilde{V}(s_{t})$ that only depends on the environment state; the advantage used in DPPO is formally

$$ $\displaystyle\hat{A}=\gamma_{\textsc{denoise}}^{k}(\bar{r}(\bar{s}_{\bar{t}}, \bar{a}_{\bar{t}})-\tilde{V}(s_{t})).$ (A1) $$

We now compare this choice with learning the value of the full state $\bar{s}_{\bar{t}(t,0)}$ that includes environment state $s_{t}$ *and* denoised action $a_{t}^{k=1}$. We additionally compare with the state-action Q-function estimator used in Psenka et al. [](^7^77Psenka et al. [] applies off-policy training with double Q-learning (according to its open-source implementation) and policy gradient over the denoising steps. Note that this is a baseline in Psenka et al. [] that is conjectured to be inefficient. We follow the same except for applying on-policy PPO updates.), $\tilde{Q}(s_{t},a_{t}^{k=0})$, that does not directly use the rollout reward $\bar{r}$ in the advantage.

Fig. A1 shows the fine-tuning results in Hopper-v2 and HalfCheetah-v2 from Gym, and Can and Square from Robomimic.
On the simpler Hopper-v2, we observe that the two baselines, both estimating the value of some action, achieves higher reward during fine-tuning than DPPO’s choice.
However, in the more challenging tasks, the environment-state-only advantage used in DPPO consistently leads to the most improved performance.
We believe estimating the accurate value of applying a continuous and high-dimensional action can be challenging, and this is exacerbated by the high stochasticity of diffusion-based policies and the action chunk size. The results here corroborate the findings in Section 5.1 that off-policy Q-learning methods can perform well in Hopper-v2 and Walker2D-v2, but often exhibit training instability in manipulation tasks from Robomimic.

Figure: Figure A1: Choice of advantage estimator. Results are averaged over five seeds in Hopper-v2 and HalfCheetah-v2 and three seeds in Can and Square.
Refer to caption: x14.png

#### Denoising discount factor.

We further examine how $\gamma_{\textsc{denoise}}$ in the DPPO advantage estimator affects fine-tuning. Using a smaller value (i.e., more discount) has the effect of downweighting the contribution of earlier denoising steps in the policy gradient. Fig. A2 shows the fine-tuning results in the same four tasks with varying $\gamma_{\textsc{denoise}}\in[0.5,0.8,0.9,1]$. We find in Hopper-v2 and HalfCheetah-v2 $\gamma_{\textsc{denoise}}=0.8$ leads to better efficiency while smaller $\gamma_{\textsc{denoise}}=0.5$ slows training. The value does not affect training noticeably in Can. In Square the smaller $\gamma_{\textsc{denoise}}=0.5$ works slightly better. Overall in manipulation tasks, DPPO training seems relatively robust to this choice.

Figure: Figure A2: Choice of denoising discount factor. Results are averaged over five seeds in Hopper-v2 and HalfCheetah-v2 and three seeds in Can and Square.
Refer to caption: x15.png

#### 2. Choice of diffusion noise schedule.

As introduced in Section 4.3, we find it helpful to clip the diffusion noise $\sigma_{k}$ to a higher minimum value $\sigma^{\text{exp}}_{\text{min}}$ to ensure sufficient exploration.
In Figure A3, we perform analysis on varying $\sigma^{\text{exp}}_{\text{min}}\in\{.001,.01,.1,.2\}$ (keeping $\sigma^{\text{prob}}_{\text{min}}=.1$ to evaluate likelihoods). Although in Can the choice of $\sigma^{\text{exp}}_{\text{min}}$ does not affect the fine-tuning performance, in Square a higher $\sigma^{\text{exp}}_{\text{min}}=0.1$ is required to prevent the policy from collapsing. We conjecture that this is due to limited exploration causing policy over-optimizing the collected samples that exhibit limited state-action coverage. We also visualize the trajectories at the beginning of fine-tuning in Avoid task from D3IL. With higher $\sigma^{\text{exp}}_{\text{min}}$, the trajectories still remain near the two modes of the pre-training data but exhibit a higher coverage in the state space — we believe this additional coverage leads to better exploration. Anecdotally, we find terminating the denoising process early can also provide exploration noise and lead to comparable results, but it requires a more involved implementation around the denoising MDP.

Figure: Figure A3: Choice of minimum diffusion noise. Results are averaged over three seeds. Note in Left, with higher minimum noise level, the sampled trajectories exhibit wider coverage at the two modes but still maintain the overall structure.
Refer to caption: x16.png

#### 3. Choice of the number of fine-tuned denoising steps.

We examine how the number of fine-tuned denoising steps in DPPO, $K^{\prime}$, affects the fine-tune performance and wall-clock time in Fig. A4. We show the curves of individual runs (three for each $K^{\prime}$) instead of the average as their wall-clock times (X-axis) are not perfectly aligned.
Generally, fine-tuning too few denoising steps (e.g., 3) can lead to subpar asymptotic performance and slower convergence especially in Can. Fine-tuning 10 steps leads to the overall best efficiency. Similar results are also shown in Fig. A7 with Avoid task.
Lastly, we note that the GPU memory usage scales linearly with $K^{\prime}$.

We note that the findings here mostly correlate with those from varying the denoising discount factor, $\gamma_{\textsc{denoise}}$. Discounting the earlier denoising steps in the policy gradient can be considered as a soft version of hard limiting the number of fine-tuned denoising steps. Depending on the amount of fine-tuning needed from the pre-trained action distribution, one can flexibly adjust $\gamma_{\textsc{denoise}}$ and $K^{\prime}$ to achieve the best efficiency.

Figure: Figure A4: Choice of number of fine-tuned denoising steps, $K^{\prime}$. Individual runs are shown. The curves are smoothed using a Savitzky–Golay filter.
Refer to caption: x17.png

### B.2 Effect of expert data

We investigate the effect of the amount of pre-training expert data on fine-tuning performance. In Fig. A5 we compare DPPO and Gaussian in Hopper-v2, Square, and One-leg task from Furniture-Bench, using varying numbers of expert data (episodes) denoted in the figure. Overall, we find DPPO can better leverage the pre-training data and fine-tune to high success rates. Notably, DPPO obtains non-trivial performance (60% success rate) on One-leg from only 10 episode of demonstrations.

Figure: Figure A5: Varying the number of expert demonstrations. The numbers in the legends indicates the number of episodes used in pre-training.
Refer to caption: x18.png

#### Training from scratch.

In Fig. A6 we compare DPPO (10 denoising steps) and Gaussian *trained from scratch* (no pre-training on expert data) in the three OpenAI Gym tasks. As using larger action chunk sizes $T_{a}$ leads to poor from-scratch training shown in Fig. A5, we focus on single-action chunks $T_{a}=1$ (and $T_{p}=1$) as is typical in RL benchmarking. Though we find Gaussian trains faster than DPPO (expected since DPPO solves an MDP with longer effective horizon), DPPO still attains reasonable final performance. However, due to the multi-step (10) denoising sampling, DPPO takes about $6\times$ wall-clock time compared to Gaussian. We hope that future work will explore how to design the training curriculum of denoising steps for the best balance of training performance and wall-clock efficiency.

Figure: Figure A6: No expert data / pre-training with Gym tasks. Results are averaged over five seeds.
Refer to caption: extracted/5983783/figure/mujoco-ta1-curves.png

### B.3 Comparing to other policy parameterizations in Avoid

Figure A7 depicts the performance of various parameterizations of DPPO (with differing numbers of fine-tuned denoising steps, $K^{\prime}$) to Gaussian and GMM baselines. We study the Avoid task from D3IL, after pre-training with the data from M1, M2, M3 as described in Section 6. We find that, for $K^{\prime}\in\{15,20\}$, DPPO attains the highest performance of all methods and trains the quickest in terms of environment steps; on M1, M2, it appears to attain the greatest terminal performance as well. $K^{\prime}=10$ appears slightly better than, but roughly comparable to, the Gaussian baseline, with GMM and $K^{\prime}<10$ performing less strongly.

Figure: Figure A7: Fine-tuning performance (averaged over five seeds, standard deviation not shown) after pre-training with M1, M2, and M3 in Avoid task from D3IL. DPPO ($K=20$), Gaussian, and GMM policies are compared. We also sweep the number of fine-tuned denoising steps $K^{\prime}$ in DPPO.
Refer to caption: extracted/5983783/figure/d3il-curves.png

### B.4 Comparing to other policy parameterizations in the easier tasks from Robomimic

Figure A8 compares the performance of DPPO to Gaussian and GMM baslines, across a variety of architectures, and with state and pixel inputs, in Lift and Can environments in the Robomimic suite. Compared to the Square and Transport (results shown in Section 5), these environments are considered to be “easier”, and this is reflected in the greater performance of DPPO and Gaussian baselines (GMM still exhibits subpar performance). Nonetheless, DPPO still achieves similar or even better sample efficiency compared to Gaussian baseline.

Figure: Figure A8: Comparing to other policy parameterizations in the easier Lift and Can tasks from Robomimic, with state (left) or pixel (right) observation. Results are averaged over three seeds.
Refer to caption: x19.png

### B.5 Comparing to policy gradient using exact likelihood of Diffusion Policy

Here we experiment another novel method (which, to our knowledge, has not been explicitly studied in any previous work) for performing policy gradient with diffusion-based policies. Although diffusion model does not directly model the action likelihood, $p_{\theta}(a_{0}|s)$, there have been ways to *estimate* the value, e.g., by solving the probability flow ODE that implements DDPM []. We refer the readers to Appendix. D in Song et al. [] for a comprehensive exposition. We follow the official open-source code from Song et al.(^8^88[https://github.com/yang-song/score_sde_pytorch](https://github.com/yang-song/score_sde_pytorch)), and implement policy gradient (single-level MDP) that uses the exact action likelihood $\pi_{\theta}(a_{t}|s_{t})$ (3.1).

Fig. A9 shows the comparison between DPPO and diffusion policy gradient using exact likelihood estimate. Exact policy gradient improves the base policy in Hopper-v2 but does not outperform DPPO. It also requires more runtime and GPU memory as it backpropagates through the ODE. In the more challenging Can its success rate drops to zero. Moreover, policy gradient with exact likelihood does not offer the flexibility of fine-tuning fewer-than-$K$ denoising steps or discounting the early denoising steps that DPPO offers, which have shown in Section B.1 to often improve fine-tuning efficiency.

Figure: Figure A9: Comparing to diffusion policy gradient with exact action likelihood. Results are averaged over five seeds in Hopper-v2 and HalfCheetah-v2, and three seeds in Can.
Refer to caption: x20.png

## Appendix C Reporting of Wall-Clock Times

#### Comparing to other diffusion-based RL algorithms Section 5.1 .

Table A1 and Table A2 shows the the wall-clock time used in each OpenAI Gym task and Robomimic task. In Gym tasks, on average DPPO trains 41%, 37%, and 12% faster than DAWR, DIPO, and DQL, respectively, which all require a significant amount of gradient updates per sample to train stably. QSM, DRWR, and IDQL trains 43%, 33%, and 7% faster than DPPO, respectively. Robomimic tasks are more expensive to simulate, especially with Transport task, and thus the wall-clock difference is smaller among the different methods.
All methods use comparable time except for DIPO that uses slightly more on average.

**Table A1: Wall-clock time in seconds for a single training iteration in OpenAI Gym tasks when comparing diffusion-based RL algorithms. Each iteration involves 500 environment timesteps in each of the 40 parallelized environments running on 40 CPU threads and a NVIDIA RTX 2080 GPU (20000 steps total).**
| Method | Task |  |  |
| --- | --- | --- | --- |
| Hopper-v2 | Walker2D-v2 | HalfCheetah-v2 |  |
| DRWR | 11.3 | 12.7 | 10.4 |
| DAWR | 30.4 | 30.7 | 27.1 |
| DIPO | 27.8 | 27.9 | 26.0 |
| IDQL | 16.3 | 16.1 | 15.5 |
| DQL | 20.5 | 20.5 | 17.6 |
| QSM | 9.6 | 9.9 | 9.7 |
| DPPO | 16.6 | 18.3 | 16.8 |

**Table A2: Wall-clock time in seconds for a single training iteration in Robomimic tasks with state input when comparing diffusion-based RL algorithms. Each iteration involves 4 episodes (1200 environment timesteps for Lift and Can, 1600 for Square, and 3200 for Transport) from each of the 50 parallelized environments running on 50 CPU threads and a NVIDIA L40 GPU (60000, 80000, 160000 steps).**
| Method | Task |  |  |  |
| --- | --- | --- | --- | --- |
| Lift | Can | Square | Transport |  |
| DRWR | 32.5 | 39.5 | 59.8 | 346.1 |
| DAWR | 38.6 | 46.0 | 70.5 | 354.3 |
| DIPO | 43.9 | 51.6 | 73.3 | 359.7 |
| IDQL | 33.8 | 41.7 | 63.7 | 349.9 |
| DQL | 36.9 | 44.4 | 68.5 | 353.5 |
| QSM | 31.8 | 44.5 | 68.7 | 322.5 |
| DPPO | 35.2 | 42.0 | 65.6 | 350.3 |

#### Comparing to other policy parameterizations and architecture Section 5.3 and Section 5.4 .

Table A3 and Table A4 shows the wall-clock time used in fine-tuning in each Robomimic task with state or pixel input, respectively. Gaussian and GMM use similar times and Transformer is slightly more expensive than MLP. On average with state input, DPPO-MLP trains 24%, 21%, 24%, and 22% slower than baselines due to the more expensive diffusion sampling. DPPO-UNet requires more time with the extensive use of convolutional and normalization layers and trains on average 49% slower than DPPO-MLP. On average with pixel input, DPPO-ViT-MLP trains 14% slower than Gaussian-ViT-MLP — the difference is smaller than the state input case as the rendering in simulation can be expensive. Table A5 shows the wall-clock time used in Furniture-Bench tasks. DPPO-UNet trains 20% slower than Gaussian-MLP on average.

**Table A3: Wall-clock time in seconds for a single training iteration in Robomimic tasks with state input when comparing policy parameterizations. Each iteration involves 4 episodes (1200 environment timesteps for Lift and Can, 1600 for Square, and 3200 for Transport) from each of the 50 parallelized environments running on 50 CPU threads and a NVIDIA L40 GPU (60000, 80000, 160000 steps).**
| Method | Task |  |  |  |
| --- | --- | --- | --- | --- |
| Lift | Can | Square | Transport |  |
| Gaussian-MLP | 27.7 | 35.7 | 56.2 | 255.6 |
| Gaussian-Transformer | 29.8 | 37.1 | 57.8 | 266.1 |
| GMM-MLP | 28.0 | 36.2 | 55.2 | 254.5 |
| GMM-Transformer | 29.5 | 37.4 | 58.1 | 260.2 |
| DPPO-MLP | 35.6 | 43.3 | 65.0 | 350.5 |
| DPPO-UNet | 83.6 | 92.7 | 130.4 | 431.1 |

**Table A4: Wall-clock time in seconds for a single training iteration in Robomimic tasks with pixel input when comparing policy parameterizations. Each iteration involves 4 episodes (1200 environment timesteps for Lift and Can, 1600 for Square, and 3200 for Transport) from each of the 50 parallelized environments running on 50 CPU threads and a NVIDIA L40 GPU (60000, 80000, 160000 steps).**
| Method | Task |  |  |  |
| --- | --- | --- | --- | --- |
| Lift | Can | Square | Transport |  |
| Gaussian-ViT-MLP | 153.6 | 173.1 | 277.0 | 770.0 |
| DPPO-ViT-MLP | 194.9 | 202.5 | 328.5 | 871.3 |

**Table A5: Wall-clock time in seconds for a single training iteration in Furniture-Bench tasks when comparing policy parameterizations. Each iteration involves 1 episodes (700 environment timesteps for One-leg, and 1000 for Lamp and Round-table) from each of the 1000 parallelized environments running on a NVIDIA L40 GPU (700000, 1000000, 1000000 steps).**
| Method | Task |  |  |
| --- | --- | --- | --- |
| One-leg | Lamp | Round-table |  |
| Gaussian-MLP | 101.8 | 202.8 | 168.7 |
| DPPO-UNet | 148.4 | 258.2 | 188.6 |

## Appendix D Additional Experimental Details

**Table A6: Comparison of the different tasks considered. “Obs dim - State”: dimension of the state observation input. “Obs dim - State”: dimension of the pixel observation input. “Act dim - State”: dimension of the action space. $T$: maximum number of steps in an episode. “Sparse reward ?”: whether sparse reward is used in training instead of dense reward.**
| Task / Dataset | Obs dim - State | Obs dim - Pixel | Act dim | $T$ | Sparse reward ? |  |
| --- | --- | --- | --- | --- | --- | --- |
| Gym | Hopper-v2 | 11 | - | 3 | 1000 | No |
| Walker2D-v2 | 17 | - | 6 | 1000 | No |  |
| HalfCheetah-v2 | 17 | - | 6 | 1000 | No |  |
| Franka-Kitchen | Kitchen-Complete-v0 | 60 | - | 9 | 280 | Yes |
| Kitchen-Partial-v0 | 60 | - | 9 | 280 | Yes |  |
| Kitchen-Mixed-v0 | 60 | - | 9 | 280 | Yes |  |
| Robomimic, state input | Lift | 19 | - | 7 | 300 | Yes |
| Can | 23 | - | 7 | 300 | Yes |  |
| Square | 23 | - | 7 | 400 | Yes |  |
| Transport | 59 | - | 14 | 800 | Yes |  |
| Robomimic, pixel input | Lift | 9 | 96$\times$96 | 7 | 300 | Yes |
| Can | 9 | 96$\times$96 | 7 | 300 | Yes |  |
| Square | 9 | 96$\times$96 | 7 | 400 | Yes |  |
| Transport | 18 | 2$\times$96$\times$96 | 14 | 800 | Yes |  |
| Furniture-Bench | One-leg | 58 | - | 10 | 700 | Yes |
| Lamp | 44 | - | 10 | 1000 | Yes |  |
| Round-table | 44 | - | 10 | 1000 | Yes |  |
| D3IL | M1 | 4 | - | 2 | 100 | Yes |
| M2 | 4 | - | 2 | 100 | Yes |  |
| M3 | 4 | - | 2 | 100 | Yes |  |

### D.1 Details of policy architectures used in all experiments

#### MLP.

For most of the experiments, we use a Multi-layer Perceptron (MLP) with two-layer residual connection as the policy head. For diffusion-based policies, we also use a small MLP encoder for the state input and another small MLP with sinusoidal positional encoding for the denoising timestep input. Their output features are then concatenated before being fed into the MLP head. Diffusion Policy, proposed by Chi et al. [], does not use MLP as the diffusion architecture, but we find it delivers comparable (or even better) pre-training performance compared to UNet.

#### Transformer.

For comparing to other policy parameterizations in Section 5.3, we also consider Transformer as the policy architecture for the Gaussian and GMM baselines. We consider decoder only. No dropout is used. A learned positional embedding for the action chunk is the sequence into the decoder.

#### UNet.

For comparing to other policy parameterizations in Section 5.3, we also consider UNet [] as a possible architecture for DP. We follow the implementation from Chi et al. [] that uses sinusoidal positional encoding for the denoising timestep input, except for using a larger MLP encoder for the observation input in each convolutional block. We find this modification helpful in more challenging tasks.

#### ViT.

For pixel-based experiments in Section 5.3 we use Vision-Transformer(ViT)-based image encoder introduced by Hu et al. [] before an MLP head. Proprioception input is appended to each channel of the image patches. We also follow [] and use a learned spatial embedding for the ViT output to greatly reduce the number of features, which are then fed into the downstream MLP head.

### D.2 Additional details of Gym tasks and training in Section 5.1

#### Pre-training.

The observations and actions are normalized to $[0,1]$ using min/max statistics from the pre-training dataset. For all three tasks the policy is trained for 3000 epochs with batch size 128, learning rate of 1e-3 decayed to 1e-4 with a cosine schedule, and weight decay of 1e-6. Exponential Moving Average (EMA) is applied with a decay rate of 0.995.

#### Fine-tuning.

All methods from Section 5.1 use the same pre-trained policy. Fine-tuning is done using online experiences sampled from 40 parallelized MuJoCo environments []. Reward curves shown in Fig. 5 are evaluated by running fine-tuned policies with $\sigma^{\text{exp}}_{\text{min}}=0.001$ (i.e., without extra noise) for 40 episodes. Each episode terminates if the default conditions are met or the episode reaches 1000 timesteps. Detailed hyperparameters are listed in Table A7.

### D.3 Descriptions of diffusion-based RL algorithm baselines in Section 5.1

#### DRWR :

This is a customized reward-weighted regression (RWR) algorithm [] that fine-tunes a pre-trained DP with a supervised objective with higher weights on actions that lead to higher reward-to-go $r$.

The reward is scaled with $\beta$ and the exponentiated weight is clipped at $w_{\max}$.
The policy is updated with experiences collected with the current policy (no buffer for data from previous iteration) and a replay ratio of $N_{\theta}$. No critic is learned.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\bar{\pi}_{\theta},\varepsilon_{t}}\Big{[} \min(e^{\beta r_{t}},w_{\max})\|\varepsilon_{t}-\varepsilon_{\theta}(a_{t}^{0} ,s_{t},k)\|^{2}\Big{]}.$ (A1) $$

#### DAWR :

This is a customized advantage-weighted regression (AWR) algorithm [] that builds on DRWR but uses TD-bootstrapped [] advantage estimation instead of the higher-variance reward-to-go for better training stability and efficiency. DAWR (and DRWR) can be seen as approximately optimizing (4.4) with a Kullback–Leibler (KL) divergence constraint on the policy [, ].

The advantage is scaled with $\beta$ and the exponentiated weight is clipped at $w_{\max}$.
Unlike DRWR, we follow [] and trains the actor in an off-policy manner: recent experiences are saved in a replay buffer $\mathcal{D}$, and the actor is updated with a replay ratio of $N_{\theta}$.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D},\varepsilon_{t}}\big{[}\min(e^{ \beta\hat{A}_{\phi}(s_{t},a_{t}^{0})},w_{\max})\|\varepsilon_{t}-\varepsilon_{ \theta}(a_{t}^{0},s_{t},k)\|^{2}\big{]}.$ (A2) $$

The critic is updated less frequently (we find diffusion models need many gradient updates to fit the actions) with a replay ratio of $N_{\phi}$.

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|\hat{A}_{\phi}(s_{t},a_{t} ^{0})-A(s_{t},a_{t}^{0})\|^{2}\big{]},$ (A3) $$

where $A$ is calculated using TD($\lambda$), with $\lambda$ as $\lambda_{\text{DAWR}}$ and the discount factor $\gamma_{\textsc{env}}$.

#### DIPO [ 101 ] :

This baseline applies “action gradient” that uses a learned state-action Q function to update the actions saved in the replay buffer, and then has DP fitting on them without weighting.

Similar to DAWR, recent experiences are saved in a replay buffer $\mathcal{D}$. The actions ($k=0$) in the buffer are updated for $M_{\text{DIPO}}$ iterations with learning rate $\alpha_{\text{DIPO}}$.

$$ $a^{m+1,k=0}_{t}=a^{m,k=0}_{t}+\alpha_{\text{DIPO}}\nabla_{\phi}\hat{Q}_{\phi}( s_{t},a^{m,k=0}_{t}),\ m=0,\dotsc,M_{\text{DIPO}}-1.$ (A4) $$

The actor is then updated with a replay ratio of $N_{\theta}$.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}\|\varepsilon_{t}- \varepsilon_{\theta}(a_{t}^{M_{\text{DIPO}},k=0},s_{t},k)\|^{2}\big{]}.$ (A5) $$

The critic is trained to minimize the Bellman residual with a replay ratio of $N_{\phi}$. Double Q-learning is also applied.

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\hat{Q}_{\phi}(s_{t+1},\bar{\pi}_{\theta}(a_{t+1}^{k=0}|s_{t+1}))-\hat{Q}_{ \phi}(s_{t},a_{t}^{m=0,k=0})\|^{2}\big{]}$ (A6) $$

#### IDQL [ 35 ] :

This baseline learns a state-action Q function and state V function to choose among the sampled actions from DP. DP fits on new samples without weighting.

Again recent experiences are saved in a replay buffer $\mathcal{D}$. The state value function is updated to match the expected Q value with an expectile loss, with a replay ratio of $N_{\psi}$.

$$ $\mathcal{L}_{\psi}=\mathbb{E}^{\mathcal{D}}\big{[}|\tau_{\text{IDQL}}-\mathbbm {1}(\hat{Q}_{\phi}(s_{t},a_{t}^{0})<\hat{V}^{2}_{\psi}(s_{t}))|\big{]}.$ (A7) $$

The value function is used to update the Q function with a replay ratio of $N_{\phi}$.

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\hat{V}_{\psi}(s_{t+1})-\hat{Q}_{\phi}(s_{t},a_{t}^{0})\|^{2}\big{]}.$ (A8) $$

The actor fits all sampled experiences without weighting, with a replay ratio of $N_{\theta}$.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}\|\varepsilon_{t}- \varepsilon_{\theta}(a_{t}^{0},s_{t},k)\|^{2}\big{]}.$ (A9) $$

At inference time, $M_{\text{IDQL}}$ actions are sampled from the actor. For training, Boltzmann exploration is applied based on the difference between $Q$ value of the sampled actions and and the $V$ value at the current state. For evaluation, the greedy action under $Q$ is chosen.

#### DQL [ 98 ] :

This baseline learns a state-action Q function and backpropagates the gradient from the critic through the entire actor (with multiple denoising steps), akin to the usual Q-learning.

Again recent experiences are saved in a replay buffer $\mathcal{D}$. The actor is then updated using both a supervised loss and the value loss with a replay ratio of $N_{\theta}$.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}\|\varepsilon_{t}- \varepsilon_{\theta}(a_{t}^{0},s_{t},k)\|^{2}-\alpha_{\text{DQL}}\hat{Q}_{\phi }(s_{t},\bar{\pi}_{\theta}(a_{t}^{0}|s_{t}))\big{]},$ (A10) $$

where $\alpha_{\text{DQL}}$ is a weighting coefficient. The critic is trained to minimize the Bellman residual with a replay ratio of $N_{\phi}$. Double Q-learning is also applied.

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\hat{Q}_{\phi}(s_{t+1},\bar{\pi}_{\theta}(a_{t+1}^{0}|s_{t+1}))-\hat{Q}_{ \phi}(s_{t},a_{t}^{0})\|^{2}\big{]}$ (A11) $$

#### QSM [ 72 ] :

This baselines learns a state-action Q function, and then updates the actor by aligning the score of the diffusion actor with the gradient of the Q function.

Again recent experiences are saved in a replay buffer $\mathcal{D}$. The critic is trained to minimize the Bellman residual with a replay ratio of $N_{\phi}$. Double Q-learning is also applied.

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\hat{Q}_{\phi}(s_{t+1},\bar{\pi}_{\theta}(a_{t+1}^{0}|s_{t+1}))-\hat{Q}_{ \phi}(s_{t},a_{t}^{0})\|^{2}\big{]}.$ (A12) $$

The actor is updated as follows with a replay ratio of $N_{\theta}$.

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}\|\alpha_{\text{QSM}} \nabla_{a}\hat{Q}_{\phi}(s_{t},a_{t})-(-\varepsilon_{\theta}(a_{t}^{0},s_{t},k ))\|^{2}\big{]},$ (A13) $$

where $\alpha_{\text{QSM}}$ scales the gradient. The negative sign before $\varepsilon_{\theta}$ is from taking the gradient of the mean $\mu$ in the denoising process.

### D.4 Descriptions of RL fine-tuning algorithm baselines in Section 5.2

In this subsection, we detail the baselines RLPD, Cal-QL, and IBRL. All policies $\pi_{\theta}$ are parameterized as unimodal Gaussian.

#### RLPD [ 7 ] :

This baseline is based on Soft Actor Critic (SAC, Haarnoja et al. []) — it learns an entropy-regularized state-action Q function, and then updates the actor by maximizing the Q function w.r.t. the action.

A replay buffer $\mathcal{D}$ is initialized with offline data, and online samples are added to $\mathcal{D}$. Each gradient update uses a batch of mixed 50/50 offline and online data.
An ensemble of $N_{\text{critic}}$ critics is used, and at each gradient step two critics are randomly chosen. The critics are trained to minimize the Bellman residual with replay ratio $N_{\phi}$:

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\hat{Q}_{\phi^{\prime}}(s_{t+1},\pi_{\theta}(a_{t+1}|s_{t+1}))-\hat{Q}_{\phi }(s_{t},a_{t})\|^{2}\big{]}.$ (A14) $$

The target critic parameter $\phi^{\prime}$ is updated with delay. The actor minimizes the following loss with a replay ratio of 1:

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}-\hat{Q}_{\phi}(s_{t},a_{t })+\alpha_{\text{ent}}\log\pi_{\theta}(a_{t}|s_{t})\big{]},$ (A15) $$

where $\alpha_{\text{ent}}$ is the entropy coefficient (automatically tuned as in SAC starting at 1).

#### Cal-QL [ 61 ] :

This baseline trains the policy $\mu$ and the action-value function $Q^{\mu}$ in an offline phase and then an online phase. During the offline phase only offline data is sampled for gradient update, while during the online phase mixed 50/50 offline and online data are sampled.
The critic is trained to minimize the following loss (Bellman residual and calibrated Q-learning):

$$ $\displaystyle\mathcal{L}_{\phi}=$ $\displaystyle\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env}}\hat {Q}_{\phi^{\prime}}(s_{t+1},\pi_{\theta}(a_{t+1}|s_{t+1})))-\hat{Q}_{\phi}(s_{ t},a_{t})\|^{2}\big{]}$ (A16) $\displaystyle+\beta_{\text{cql}}(\mathbb{E}^{\mathcal{D}}\big{[}\max(Q_{\phi}( s_{t},a_{t}),V(s_{t}))\big{]}-\mathbb{E}^{\mathcal{D}}\big{[}Q_{\phi}(s_{t},a_ {t})\big{]}),$ $$

where $\beta_{\text{cql}}$ is a weighting coefficient between Bellman residual and calibration Q-learning and $V(s_{t})$ is estimated using Monte-Carlo returns.
The target critic parameter $\phi^{\prime}$ is updated with delay. The actor minimizes the following loss:

$$ $\mathcal{L}_{\theta}=\mathbb{E}^{\mathcal{D}}\big{[}-\hat{Q}_{\phi}(s_{t},a_{t })+\alpha_{\text{ent}}\log\pi_{\theta}(a_{t}|s_{t})\big{]},$ (A17) $$

where $\alpha_{\text{ent}}$ is the entropy coefficient (automatically tuned as in SAC starting at 1).

#### IBRL [ 40 ] :

This baseline first pre-trains a policy $\mu_{\psi}$ using behavior cloning, and for fine-tuning it trains a RL policy $\pi_{\theta}$ initialized as $\mu_{\psi}$.
During fine-tuning recent experiences are saved in a replay buffer $\mathcal{D}$. An ensemble of $N_{\text{critic}}$ critics is used, and at each gradient step two critics are randomly chosen. The critics are trained to minimize the Bellman residual with replay ratio $N_{\phi}$:

$$ $\mathcal{L}_{\phi}=\mathbb{E}^{\mathcal{D}}\big{[}\|(R_{t}+\gamma_{\textsc{env }}\underset{a^{\prime}\in\{a^{IL},a^{RL}\}}{\max}\hat{Q}_{\phi^{\prime}}(s_{t+ 1},a^{\prime})-\hat{Q}_{\phi}(s_{t},a_{t})\|^{2}\big{]}$ (A18) $$

where $a^{IL}=\mu_{\psi}(s_{t+1})$ (no noise) and $a^{RL}\sim\pi_{\theta^{\prime}}(s_{t+1})$, and $\pi_{\theta^{\prime}}$ is the target actor.
The target critic parameter $\phi^{\prime}$ is updated with delay. The actor minimizes the following loss with a replay ratio of 1:

$$ $\mathcal{L}_{\theta}=-\mathbb{E}^{\mathcal{D}}\big{[}\hat{Q}_{\phi}(s_{t},a_{t })\big{]}.$ (A19) $$

The target actor parameter $\theta^{\prime}$ is also updated with delay.

### D.5 Additional details of DPPO implementation in all tasks

Similar to all baselines in Section D.3, we denote $N_{\theta}$ and $N_{\phi}$ the replay ratio for the actor (Diffusion Policy) and the critic (state value function) in DPPO; in practice we always set $N_{\theta}=N_{\phi}$ in DPPO, with the combined loss $\mathcal{L}=\mathcal{L}_{\theta}+\mathcal{L}_{\phi}$. Similar to usual PPO implementations [], the batch updates in an iteration terminate when the KL divergence between $\pi_{\theta}$ and $\pi_{\theta_{\text{old}}}$ reaches 1.

Since the gradient update in DPPO (Eq. 4.4) involves expectation over both environment steps and denoising steps, we use a larger batch size compared to, e.g., PPO training with Gaussian policy parameterization. Roughly we use the batch size from Gaussian training times the number of the fine-tuned denoising steps; in some cases like Robomimic we also observe that a much smaller batch size (close to that of Gaussian training) can be used and significantly improves sample efficiency.

We also find the PPO clipping ratio, $\varepsilon$, can affect the training stability significantly in DPPO (as well as in Gaussian and GMM policies) especially in sparse-reward manipulation tasks. In practice we find that, a good indicator of the amount of clipping leading to optimal training efficiency, is to aim for a clipping fraction (fraction of individual samples being clipped in a batch) of 10% to 20%. For each method in different tasks, we vary $\varepsilon$ in $\{.1,.01,.001\}$ and choose the highest value that satisfies the clipping fraction target. Empirically we also find that, using a higher $\varepsilon$ for earlier denoising steps in DPPO further improves training stability in manipulation tasks. Denote $\varepsilon_{k}$ the clipping value at denoising step $k$, and in practice we set $\varepsilon_{k=(K-1)}=0.1\varepsilon_{k=0}$, and it follows an exponential schedule among intermediate $k$.

### D.6 Additional details of Franka - Kitchen tasks and training in Section 5.2

#### Tasks.

We consider three settings from the D4RL benchmark []:
(1) Kitchen-Complete-v0 containing demonstrations that complete the entire task (four subtasks), (2) Kitchen-Partial-v0 containing some complete demonstrations and many ones completing only subtasks, and (3) Kitchen-Mixed-v0 containing incomplete demonstrations only.

#### Pre-training.

The observations and actions are normalized to $[0,1]$ using min/max statistics from the pre-training dataset. No history observation (proprioception or ground-truth object states) is used. All policies are trained with batch size 128, learning rate 1e-4 decayed to 1e-5 with a cosine schedule, and weight decay 1e-6. DPPO policies are trained with 8000 epochs. For IBRL and Cal-QL we follow the hyperparameters from the original implementations — IBRL proposes using (1) wider MLP layers and (2) dropout during pre-training, which we follow too. We use $T_{a}=4$ for DPPO; we also tried to use the same action chunk size with IBRL, RLPD, and Cal-QL, but we find for all of them $T_{a}=1$ leads to better performance.

#### Fine-tuning.

With DPPO, policies are fine-tuned using online experiences sampled from 40 parallelized MuJoCo environments [], while the baselines use only one environment (matching their original implementations). Episodes terminates when they reach maximum episode lengths (shown in Table A6) or all four subtasks are completed. Detailed hyperparameters are listed in Table A8 — we follow the hyperparameter choices from the original implementations of the baselines.

#### Larger variance with DPPO in Fig. 6 .

In Fig. 6, it is shown that DPPO exhibits a larger variance in normalized score with Kitchen-Partial-v0 than Cal-QL. This is due to DPPO solving either 3/4 or 2/4 subtasks in one seed (low variance within the evaluation episodes in one seed) but high variance over seeds, whereas Cal-QL has higher variance among evaluation episodes in one seed but on average over seeds it shows lower variance. This also highlights a notable property of DPPO: Kitchen-Partial-v0 and Kitchen-Mixed-v0 have trajectories only completing subtasks, thus being highly multi-modal. Diffusion policy can sometimes struggle to learn all the modes from pre-training, and since DPPO directly fine-tunes the pre-trained policy, it can fail to converge to 100% success rate at fine-tuning. Cal-QL instead learns from all offline data during fine-tuning in an off-policy manner, thus less sensitive to pre-training performance. Nonetheless, with offline data completing tasks consistently despite varying quality (Robomimic and Kitchen-Complete-v0, which, we believe, are more realistic in the current paradigm of robot manipulation), DPPO demonstrates much better final performance than Cal-QL and other baselines in Fig. 6.

### D.7 Additional details of Robomimic tasks and training in Section 5.3

#### Tasks.

We consider four tasks from the Robomimic benchmark []:
(1) Lift: lifting a cube from the table, (2) Can: picking up a Coke can and placing it at a target bin, (3) Square: picking up a square nut and place it on a rod, and (4) Transport: two robot arms removing a bin cover, picking and placing a cube, and then transferring a hammer from one container to another one.

#### Pre-training.

Robomimic provides the Multi-Human (MH) dataset with noisy human demonstrations for each task, which we use to pre-train the policies. The observations and actions are normalized to $[0,1]$ using min/max statistics from the pre-training dataset. No history observation (pixel, proprioception, or ground-truth object states) is used. All policies are trained with batch size 128, learning rate 1e-4 decayed to 1e-5 with a cosine schedule, and weight decay 1e-6. Diffusion-based policies are trained with 8000 epochs, while Gaussian and GMM policies are trained with 5000 epochs — we find diffusion models require more gradient updates to fit the data well.

#### Fine-tuning.

Diffusion-based, Gaussian, and GMM pre-trained policies are then fine-tuned using online experiences sampled from 50 parallelized MuJoCo environments []. Success rate curves shown in Fig. 5, Fig. 7, and Fig. A8 are evaluated by running fine-tuned policies with $\sigma^{\text{exp}}_{\text{min}}=0.001$ (i.e., without extra noise) for 50 episodes. Episodes terminates only when they reach maximum episode lengths (shown in Table A6). Detailed hyperparameters are listed in Table A9.

#### Pixel training.

We use the wrist camera view in Lift and Can, the third-person camera view in Square, and the two robot shoulder camera views in Transport. Random-shift data augmentation is applied to the camera images during both pre-training and fine-tuning. Gradient accumulation is used in fine-tuning so that the same batch size (as in state-input training) can fit on the GPU. Detailed hyperparameters are listed in Table A10.

### D.8 Descriptions of policy parameterization baselines in Section 5.3

#### Gaussian.

We consider unimodal Gaussian with diagonal covariance, the most commonly used policy parameterization in RL. The standard deviation for each action dimension, $\sigma_{\text{Gau}}$, is fixed during pre-training; we also tried to learn $\sigma_{\text{Gau}}$ from the dataset but we find the training very unstable. During fine-tuning $\sigma_{\text{Gau}}$ is learned starting from the same fixed value and also clipped between $0.01$ and $0.2$. Additionally we clip the sampled action to be within $3$ standard deviation from the mean. As discusses in Section D.5, we choose the PPO clipping ratio $\varepsilon$ based on the empirical clipping fraction in each task. This setup is also used in the Furniture-Bench experiments. We note that we spend significant amount of efforts tuning the Gaussian baseline, and our results with it are some of the best known ones in RL training for long-horizon manipulation tasks (exceeding our initial expectations), e.g., reaching $\sim$100% success rate in Lamp with Low randomness.

#### GMM.

We also consider Gaussian Mixture Model as the policy parameterization. We denote $M_{\text{GMM}}$ the number of mixtures. The standard deviation for each action dimension in each mixture, $\sigma_{\text{GMM}}$, is also fixed during pre-training. Again during fine-tuning $\sigma_{\text{GMM}}$ is learned starting from the same fixed value and also clipped between $0.01$ and $0.2$.

### D.9 Additional details of Furniture - Bench tasks and training in Section 5.4

#### Tasks.

We consider three tasks from the Furniture-Bench benchmark []: (1) One-leg: assemble one leg of a table by placing the tabletop in the fixture corner, grasping and inserting the table leg, and screwing in the leg, (2) Lamp: place the lamp base in the fixture corner, grasp, insert, and screw in the light bulb, and finally place the lamp shade, (3) Round-table: place a round tabletop in the fixture corner, insert and screw in the table leg, and then insert and screw in the table base. See Fig. A10 for the visualized rollouts in simulation.

#### Pre-training.

The pre-training dataset is collected in the simulated environments using a SpaceMouse(^9^99[https://3dconnexion.com/us/product/spacemouse-wireless/](https://3dconnexion.com/us/product/spacemouse-wireless/)), a 6 DoF input device. The simulator runs at 10Hz. At every timestep, we read off the state of the SpaceMouse as $\delta\mathbf{a}=[\Delta x,\Delta y,\Delta z,\Delta\text{roll},\Delta\text{
pitch},\Delta\text{yaw}]$, which is converted to a quaternion before passed to the environment step and stored as the action alongside the current observation in the trajectory. If $|\Delta\mathbf{a}_{i}|<\varepsilon\ \forall i$ for some small $\varepsilon=0.05$ defining the threshold for a no-op, we do not record any action nor pass it to the environment. Discarding no-ops is important for allowing the policies to learn from demonstrations effectively. When the desired number of demonstrations has been collected (typically 50), we process the actions to convert the delta actions stored from the SpaceMouse into absolute pose actions by applying the delta action to the current EE pose at each timestep.

The observations and actions are normalized to $[-1,1]$ using min/max statistics from the pre-training dataset. No history observation (proprioception or ground-truth object states) is used, i.e., only the current observation is passed to the policy. All policies are trained with batch size 256, learning rate 1e-4 decayed to 1e-5 with a cosine schedule, and weight decay 1e-6. Diffusion-based policies are trained with 8000 epochs, while Gaussian policies are trained with 3000 epochs. Gaussian policies can easily overfit the pre-trained dataset, while diffusion-based policies are more resilient. Gaussian policies also require a very large MLP ($\sim$10 million parameters) to fit the data well.

#### Fine-tuning.

Diffusion-based and Gaussian pre-trained policies are then fine-tuned using online experiences sampled from 1000 parallelized IsaacGym environments [].
Success rate curves shown in Fig. 8 are evaluated by running fine-tuned policies with $\sigma^{\text{exp}}_{\text{min}}=0.001$ (i.e., without extra noise) for 1000 episodes. Episodes terminate only when they reach maximum episode length (shown in Table A6). Detailed hyperparameters are listed in Table A11. We find a smaller amount of exploration noise (we set $\sigma^{\text{exp}}_{\text{min}}$ and $\sigma_{\text{Gau}}$ to be 0.04) is necessary for the pre-trained policy achieving nonzero success rates at the beginning of fine-tuning.

Figure: Figure A10: Representative rollouts from simulated Furniture-Bench tasks.
Refer to caption: x21.png

#### Hardware setup - robot control.

The physical robot used is a Franka Emika Panda arm. The policies output a sequence of desired end-effector poses in the robot base frame to control the robot. These poses are converted into joint position targets through differential inverse kinematics. We calculate the desired end-effector velocity as the difference between the desired and current poses divided by the delta time $dt=1/10$. We then convert this to desired joint velocities using the Jacobian and compute the desired joint positions with a first-order integration over the current joint positions and desired velocity. The resulting joint position targets are passed to a low-level joint impedance controller provided by Polymetis [], running at 1kHz.

#### Hardware setup - state estimation.

To deploy state-based policies on real hardware, we utilize AprilTags [] for part pose estimation. The Furniture-Bench [] task suite provides AprilTags for each part and code for estimating part poses from tag detections. The process involves several steps: (1) detecting tags in the camera frame, (2) mapping tag detections to the robot frame for policy compatibility, (3) utilizing known offsets between tags and object centers in the simulator, and (4) calibrating the camera pose using an AprilTag at a known position relative to the robot base. Despite general accuracy, detections can be noisy, especially during movement or partial occlusion, which the One-leg task features. Since the task requires high precision, we find the following to help make the estimation reliable enough:

- •
Camera coverage: We find detection quality sensitive to distance and angle between the camera and tag. This issue is likely due to the RealSense D435 camera having mediocre image quality and clarity and the relatively small tags. To remedy this, we opt to use 4 cameras roughly evenly spread out around the scene to ensure that at least one camera has a solid view of a tag on all the parts (i.e., as close as possible with a straight-on view). To find the best camera positions, we start with having a camera in each of the cardinal directions around the scene. Then, we adjust the pose of each to get it as close as possible to the objects while still covering the necessary workspace and capturing the base tag for calibration. Moving the robot arm around the scene to avoid the worst occlusion is also helpful.
- •
Lighting: Even with better camera coverage and placement, detection quality depends on having crisp images. We find proper lighting helpful to improve image quality. In particular, the scene should be well and evenly lit around the scene without causing reflections in either the tag or table.
- •
Filtering: Bad detections can sometimes cause the resulting pose estimate to deviate significantly from the true pose, i.e., jumping several centimeters from one frame to the next. This usually only happens on isolated frames, and thus before “accepting” a given detection, we check if the new position and orientation are within 5 cm and 20 degrees of the previously accepted pose. In addition, we apply low-pass filtering on the detection using a simple exponential average (with $\alpha=0.25$) to smooth out the high-frequency noise.
- •
Averaging: The objects have multiple tags that can be detected from multiple cameras. After performing the filtering step, we average all pose estimates for the same object across different tags and cameras, which also helps smooth out noise. This alone, however, does not fully cancel the case when a single detection has a large jump, as this can severely skew the average, still necessitating a filtering step. Having multiple cameras benefits this step, too, as it provides more detections to average over.
- •
Caching part pose in hand: A particularly difficult phase of the task to achieve good detections is when the robot transports the table leg from the initial position to the tabletop for insertion. The main problems are that the movement can blur the images, and the grasping can cause occlusions. Therefore, we found it helpful to assume that once the part was grasped by the robot, it would not move in the grasp until the gripper opened. With this, we can “cache” the pose of the part relative to the end-effector once the object is fully grasped and use this instead of relying on detections during the movement.
- •
Normalization pitfalls and clipping: We generally use min-max normalization of the state observations to ensure observations are in $[-1,1]$. The tabletop part moves very little in the $z$-direction demonstration data, meaning the resulting normalization limits (the minimum and maximum value of the data) can be very close, $x_{\text{max}}-x_{\text{min}}\approx 0$. With these tight limits, the noise in the real-world detection can be amplified greatly as $x_{\text{norm}}=\frac{x-x_{\text{min}}}{x_{\text{max}}-x_{\text{min}}}$. Therefore, ensure that normalization ranges are reasonable. As an extra safeguard, clipping the data to $[-1,1]$ can also help.
- •
Only estimate necessary states: Despite the One-leg task having 5 parts, only 2 are manipulated. Only estimating the pose of those parts can eliminate a lot of noise. In particular, the pose of the 3 legs that are not used and the obstacle (the U-shaped fixture) can be set to an arbitrary value from the dataset.
- •
Visualization for debugging: We use the visualization tool MeshCat(^10^1010[https://github.com/meshcat-dev/meshcat](https://github.com/meshcat-dev/meshcat)) extensively for debugging of state estimation. The tool allows for easy visualizations of poses of all relevant objects in the scene, like the robot end-effector and parts, which makes sanity-checking the implementation far easier than looking at raw numbers.

#### Hardware evaluation.

We perform 20 trials for each method. We adopt a single-blind model selection process: at the beginning of each trial, we first randomize the initial state. Then, we randomly select a method and roll it out, but the experimenter does not observe which model is used. We record the success and failure of each trial and then aggregate statistics for each model after all trials are completed.

#### Domain randomization for sim-to-real transfer.

To facilitate the sim-to-real transfer, we apply additional domain randomization to the simulation training. We record the range of observation noises in hardware without any robot motion and then apply the same amount of noise to state observations in simulation. We find the state estimation in hardware particularly sensitive to the object heights. Also, we apply random noise (zero mean with 0.03 standard deviation) to the sampled action from DPPO to simulate the imperfect low-level controller; we find adding such noise to the Gaussian policy leads to zero task success rate while DPPO is robust to it (also see discussion in Section 6).

#### BC regularization loss used for Gaussian baseline.

Since the fine-tuned Gaussian policy exhibits very jittery behavior and leads to zero success rate in real evaluation, we further experiment with adding a behavior cloning (BC) regularization loss in fine-tuning with the Gaussian baseline. The combined loss follows

$$ $\mathcal{L}_{\theta,\text{+BC}}=\mathcal{L_{\theta}}-\alpha_{\text{BC}}\mathbb {E}^{\pi_{\theta_{\text{old}}}}[\sum_{k=0}^{K-1}\log\pi_{\theta_{\text{pre- trained}}}(a_{t}^{k}|a_{t}^{k+1},s_{t})],$ (A20) $$

where $\pi_{\theta_{\text{pre-trained}}}$ is the frozen BC-only policy. The extra term encourages the newly sampled actions from the fine-tuned policy to remain high-likelihood under the BC-only policy. We set $\alpha_{\text{BC}}=0.1$. However, although this regularization reduces the sim-to-real gap, it also significantly limits fine-tuning, leading to the fine-tuning policy saturating at 53% success rate shown in Fig. 8.

### D.10 Additional details of Avoid task from D3IL and training in Section 6

#### Pre-training.

We split the original dataset from D3IL based on the three settings, M1, M2, and M3; in each setting, observations and actions are normalized to $[0,1]$ using min/max statistics. All policies are trained with batch size 16 (due to the small dataset size), learning rate 1e-4 decayed to 1e-5 with a cosine schedule, and weight decay 1e-6. Diffusion-based policies are trained with about 15000 epochs, while Gaussian and GMM policies are trained with about 10000 epochs; we manually examine the trajectories from different pre-trained checkpoints and pick ones that visually match the expert data the best.

#### Fine-tuning.

Diffusion-based, Gaussian, and GMM pre-trained policies are then fine-tuned using online experiences sampled from 50 parallelized MuJoCo environments []. Reward curves shown in Fig. 11 and Fig. A7 are evaluated by running fine-tuned policies with the same amount of exploration noise used in training for 50 episodes; we choose to use the training (instead of evaluation) setup since Gaussian policies exhibit multi-modality only with training noise. Episodes terminate only when they reach 100 steps.

#### Added action noise during fine-tuning.

In Fig. 11 left, we demonstrate that DPPO exhibits stronger training stability when noise is added to the sampled actions during fine-tuning. The noise starts at the 5th iteration. It is sampled from a uniform distribution with the lower limit ramping up to 0.1 and the upper limit ramping up to 0.2 linearly in 5 iterations. The limits are kept the same from the 10th iteration to the end of fine-tuning.

### D.11 Listed training hyperparameters

**Table A7: Fine-tuning hyperparameters for OpenAI Gym and Robomimic tasks when comparing diffusion-based RL methods. We list hyperparameters shared by all methods first, and then method-specific ones.**
|  |  | Task(s) |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Parameter | Gym | Lift, Can | Square | Transport |
| Common | $\gamma_{\textsc{env}}$ | 0.99 | 0.999 | 0.999 | 0.999 |
| $\sigma^{\text{exp}}_{\text{min}}$ | 0.1 | 0.1 | 0.1 | 0.08 |  |
| $\sigma^{\text{prob}}_{\text{min}}$ | 0.1 |  |  |  |  |
| $T_{p}$ | 4 | 4 | 4 | 8 |  |
| $T_{a}$ | 4 | 4 | 4 | 8 |  |
| $K$ | 20 |  |  |  |  |
| Actor learning rate | 1e-4 for DPPO and 1e-5 for others (tuned from 1e-4 to 1e-5) |  |  |  |  |
| Critic learning rate (if applies) | 1e-3 |  |  |  |  |
| Actor MLP dims | [512, 512, 512] | [512, 512, 512] | [1024, 1024, 1024] | [1024, 1024, 1024] |  |
|  | Critic MLP dims (if applies) | [256, 256, 256] |  |  |  |
| DRWR | $\beta$ | 10 |  |  |  |
| $w_{\max}$ | 100 |  |  |  |  |
| $N_{\theta}$ | 16 |  |  |  |  |
| Batch size | 1000 |  |  |  |  |
| DAWR | $\beta$ | 10 |  |  |  |
| $w_{\max}$ | 100 |  |  |  |  |
| $\lambda_{\text{DAWR}}$ | 0.95 |  |  |  |  |
| $N_{\theta}$ | 64 |  |  |  |  |
| $N_{\phi}$ | 16 |  |  |  |  |
| Buffer size | 200000 | 120000 | 120000 | 120000 |  |
| Batch size | 1000 |  |  |  |  |
| DIPO | $\alpha_{\text{DIPO}}$ | 1e-4 |  |  |  |
| $M_{\text{DIPO}}$ | 10 |  |  |  |  |
| $N_{\theta}$ | 64 |  |  |  |  |
| Buffer size | 1000000 |  |  |  |  |
| Batch size | 1000 |  |  |  |  |
| IDQL | $M_{\text{IDQL}}$ | 20 | 10 | 10 | 10 |
| $N_{\theta}$ | 128 |  |  |  |  |
| $N_{\phi}$ | 128 |  |  |  |  |
| Buffer size | 1000000 | 250000 | 250000 | 250000 |  |
| Batch size | 1000 |  |  |  |  |
| DQL | $\alpha_{\text{DQL}}$ | 1 |  |  |  |
| $N_{\theta}$ | 16 |  |  |  |  |
| $N_{\phi}$ | 16 |  |  |  |  |
| Buffer eize | 1000000 |  |  |  |  |
| Batch size | 1000 |  |  |  |  |
| QSM | $\alpha_{\text{QSM}}$ | 10 |  |  |  |
| $N_{\theta}$ | 16 |  |  |  |  |
| $N_{\phi}$ | 16 |  |  |  |  |
| Buffer size | 1000000 | 250000 | 250000 | 250000 |  |
| Batch size | 1000 |  |  |  |  |
| DPPO | $\gamma_{\textsc{denoise}}$ | 0.99 |  |  |  |
| GAE $\lambda$ | 0.95 |  |  |  |  |
| $N_{\theta}$ | 5 | 10 | 10 | 10 |  |
| $N_{\phi}$ | 5 | 10 | 10 | 10 |  |
| $\varepsilon$ | 0.01 |  |  |  |  |
| Batch size | 50000 | 7500 | 10000 | 10000 |  |
| $K^{\prime}$ | 10 |  |  |  |  |

**Table A8: Fine-tuning hyperparameters for HalfCheetah-v2, Franka-Kitchen, Can, and Square (PH or MH datasets) when comparing demo-augmented RL methods. We list hyperparameters shared by all methods first, and then method-specific ones.**
|  |  | Task(s) |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Parameter | HalfCheetah-v2 | Kitchen-Complete-v0 | Kitchen-Partial-v0 | Kitchen-Mixed-v0 |
| Common | $\gamma_{\textsc{env}}$ | 0.99 |  |  |  |
| RLPD | $T_{p}$ | 1 |  |  |  |
| $T_{a}$ | 1 |  |  |  |  |
| $N_{\phi}$ | 2 3 | 10310 | 10 |  |  |
|  | $N_{\text{critic}}$ | 10 | 5 | 5 | 5 |
|  | Batch size | 256 |  |  |  |
| Cal-QL | $T_{p}$ | 1 |  |  |  |
| $T_{a}$ | 1 |  |  |  |  |
|  | $\beta_{\text{cql}}$ | 5 |  |  |  |
|  | Batch size | 256 |  |  |  |
| IBRL | $T_{p}$ | 1 |  |  |  |
| $T_{a}$ | 1 |  |  |  |  |
| $N_{\phi}$ | 5 |  |  |  |  |
|  | $N_{\text{critic}}$ | 5 |  |  |  |
|  | Batch size | 256 |  |  |  |
| DPPO | $T_{p}$ | 1 | 4 | 4 | 4 |
| $T_{a}$ | 1 | 4 | 4 | 4 |  |
| $\sigma^{\text{exp}}_{\text{min}}$ | 0.1 |  |  |  |  |
| $\sigma^{\text{prob}}_{\text{min}}$ | 0.1 |  |  |  |  |
| $\gamma_{\textsc{denoise}}$ | 0.99 |  |  |  |  |
| GAE $\lambda$ | 0.95 |  |  |  |  |
| $N_{\theta}$ | 5 | 10 | 10 | 10 |  |
| $N_{\phi}$ | 5 | 10 | 10 | 10 |  |
| $\varepsilon$ | 0.01 |  |  |  |  |
|  | Batch size | 10000 | 5600 | 5600 | 5600 |
|  | $K$ | 20 |  |  |  |
|  | $K^{\prime}$ | 10 |  |  |  |

**Table A9: Fine-tuning hyperparameters for Robomimic tasks with state input when comparing policy parameterizations. We list hyperparameters shared by all methods first, and then method-specific ones. Since the different policy parameterizations use different neural network architecture, we list the total model size here instead of the details such as MLP dimensions.**
|  |  | Task |  |  |
| --- | --- | --- | --- | --- |
| Method | Parameter | Lift, Can | Square | Transport |
| Common | $\gamma_{\textsc{env}}$ | 0.999 |  |  |
| $T_{a}$ | 4 | 4 | 8 |  |
| Actor learning rate | 1e-4 | 1e-5 | 1e-5 (decayed to 1e-6) |  |
| Critic learning rate | 1e-3 |  |  |  |
| GAE $\lambda$ | 0.95 |  |  |  |
| $N_{\theta}$ | 10 | 10 | 8 |  |
| $N_{\phi}$ | 10 | 10 | 8 |  |
| $\varepsilon$ | 0.01 (annealed in DPPO) |  |  |  |
| Gaussian, Common | $\sigma_{\text{Gau}}$ | 0.1 | 0.1 | 0.08 |
| Batch size | 7500 | 10000 | 10000 |  |
| Gaussian-MLP | Model size | 552K | 2.15M | 1.93M |
| Gaussian-Transformer | Model size | 675K | 1.86M | 1.87M |
| GMM, Common | $M_{\text{GMM}}$ | 5 |  |  |
| $\sigma_{\text{GMM}}$ | 0.1 | 0.1 | 0.08 |  |
| Batch size | 7500 | 10000 | 10000 |  |
| GMM-MLP | Model size | 1.15M | 4.40M | 4.90M |
| GMM-Transformer | Model size | 680K | 1.87M | 1.89M |
| DPPO, Common | $\gamma_{\textsc{denoise}}$ | 0.99 |  |  |
| $\sigma^{\text{exp}}_{\text{min}}$ | 0.1 | 0.1 | 0.08 |  |
| $\sigma^{\text{prob}}_{\text{min}}$ | 0.1 | 0.1 | 0.1 |  |
| $K$ | 20 |  |  |  |
| $K^{\prime}$ | 10 |  |  |  |
|  | Batch size | 75000 | 100000 | 100000 |
| DPPO-MLP | Model size | 576K | 2.31M | 2.43M |
| DPPO-UNet | Model size | 652K | 1.62M | 1.68M |

**Table A10: Fine-tuning hyperparameters for Robomimic tasks with pixel input when comparing policy parameterizations. We list hyperparameters shared by all methods first, and then method-specific ones. Since the different policy parameterizations use different neural network architecture, we list the total model size here instead of the details such as MLP dimensions.**
|  |  | Task |  |  |
| --- | --- | --- | --- | --- |
| Method | Parameter | Lift, Can | Square | Transport |
| Common | $\gamma_{\textsc{env}}$ | 0.999 |  |  |
| $T_{a}$ | 4 | 4 | 8 |  |
| Actor learning rate | 1e-4 | 1e-5 | 1e-5 (decayed to 1e-6) |  |
| Critic learning rate | 1e-3 |  |  |  |
| GAE $\lambda$ | 0.95 |  |  |  |
| $N_{\theta}$ | 10 | 10 | 8 |  |
| $N_{\phi}$ | 10 | 10 | 8 |  |
| $\varepsilon$ | 0.01 (annealed in DPPO) |  |  |  |
| Gaussian-ViT-MLP | Model size | 1.03M | 1.03M | 1.93M |
| $\sigma_{\text{Gau}}$ | 0.1 | 0.1 | 0.08 |  |
|  | Batch size | 7500 | 10000 | 10000 |
| DPPO-ViT-MLP | Model size | 1.06M | 1.06M | 2.05M |
| $\gamma_{\textsc{denoise}}$ | 0.9 |  |  |  |
| $\sigma^{\text{exp}}_{\text{min}}$ | 0.1 | 0.1 | 0.08 |  |
| $\sigma^{\text{prob}}_{\text{min}}$ | 0.10 |  |  |  |
| $K$ | 100 |  |  |  |
| $K^{\prime}$ | 5 (DDIM) |  |  |  |
|  | Batch size | 37500 | 50000 | 50000 |

**Table A11: Fine-tuning hyperparameters for Furniture-Bench tasks when comparing policy parameterizations. We list hyperparameters shared by all methods first, and then method-specific ones.**
|  |  | Task |  |  |
| --- | --- | --- | --- | --- |
| Method | Parameter | One-leg | Lamp | Round-table |
| Common | $\gamma_{\textsc{env}}$ | 0.999 |  |  |
| $T_{a}$ | 8 |  |  |  |
| Actor learning rate | 1e-5 (decayed to 1e-6) |  |  |  |
| Critic learning rate | 1e-3 |  |  |  |
| GAE $\lambda$ | 0.95 |  |  |  |
| $N_{\theta}$ | 5 |  |  |  |
| $N_{\phi}$ | 5 |  |  |  |
| $\varepsilon$ | 0.001 |  |  |  |
| Gaussian-MLP | Model size | 10.64M | 10.62M | 10.62M |
| $\sigma_{\text{Gau}}$ | 0.04 |  |  |  |
|  | Batch size | 8800 |  |  |
| DPPO-UNet | Model size | 6.86M | 6.81M | 6.81M |
| $\gamma_{\textsc{denoise}}$ | 0.9 |  |  |  |
| $\sigma^{\text{exp}}_{\text{min}}$ | 0.04 |  |  |  |
| $\sigma^{\text{prob}}_{\text{min}}$ | 0.1 |  |  |  |
| $K$ | 100 |  |  |  |
| $K^{\prime}$ | 5 (DDIM) |  |  |  |
|  | Batch size | 44000 |  |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: - What intuition, if any, do the authors have on why DPPO does better on long-horizon sparse-reward tasks? Is there something special about finetuning with PPO as opposed to other finetuning strategies that helps? 

- In Figure 6, why does DPPO-UNet get outperformed by DPPO-MLP on state-based tasks? And why is DPPO-UNet not present on the pixel-based versions of the task? 

- On lines 264-265 the authors state *"We propose clipping the diffusion noise schedule at a larger-than-
standard noise level to encourage exploration and training stability."*. If more noise encourages exploration, would it not be more beneficial to use the variance-exploding line of diffusion models such as EDM instead of the variance-preserving ones? 

- In Figure 7 we see positive sim2real transfer from pretraining only, but negative transfer after fine-tuning. Do the authors have some intuition as to why this is the case?
soundness: 4
strengths: - The authors explore the possibility of fine-tuning diffusion-based policies with on-policy RL and demonstrate some surprising results. Specifically, the authors find improved performance over baselines on long-horizon and sparse reward tasks. The authors also demonstrate how their method can leverage environment parallelization to train faster than off-policy RL methods. Given the abundance of GPU-accelerated simulators these days, I believe this to be a major strength as off-policy RL finetuning methods cannot leverage parallelization in the same way, leaving these high-throughput simulators as a relatively untapped resource until now. 

- The authors provide extensive comparisons to a variety of other RL finetuning methods, including two of their own novel baselines. The ablations are also thorough. 

- The paper is well written and the quality of the presentation is high.
summary: The authors propose a framework for finetuning a pretrained diffusion policy using on-policy RL, specifically PPO, and show that their method improves performance on long-horizon sparse-reward tasks and works on real hardware.
weaknesses: - Much of section 6, including the first few paragraphs up to and including Figure 8 are redundant and unnecessary. At this point there are a number of diffusion policy papers applied to various tasks as well as the original diffusion policy paper itself that show that diffusion models consistently outperform GMM and Gaussian policies at capturing the modes of a multi-modal distribution, and so this result is no longer surprising or novel. I'd have rather seen this space dedicated to showing more experimental results that were instead put in the appendix, such as the comparisons in Figure 12 to methods that finetune on additional expert data, or perhaps a more in-depth analysis on why DPPO does better on long-horizon sparse-reward tasks. "How does this method compare to just collecting and training (via BC or whatever else) on additional expert data?" was a question that came to mind while I read the results, which is an important one to discuss and one I imagine other readers will have. 

- While the technical contribution is fairly good, the novelty of this approach is relatively low. The proposed method follows the more general and somewhat repetitive trend of "take diffusion policy and try it on task X and potentially finetune with method Y" style papers in the robot learning community. 

Despite the somewhat low novelty, I believe that the impressive results and the fact that this paper is one of the first, if not the first, diffusion policy finetuning methods capable of leveraging high-throughput simulators is an important contribution to the robotics and ML communities.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * In the original diffusion policy paper, the performance in the transport is nearly 100% success rate, which conflicts with the results in the paper.
* How about the results compared with the baselines that introduce a residual policy?
* How about the results compared with the baselines that use RLPD or Cal-QL for the pre-trained diffusion policies?
soundness: 2
strengths: * This paper conducts rich experiments and analysis. They compare DPPO with other fine-tuning RL methods for diffusion policies and other fine-tuning RL policies. They also conduct several ablation studies and visualization to analyze the effectiveness of DPPO.
* This paper is well-written and well-structured. They conclude the best practice for DPPO and the benefits of such framework.
summary: This paper proposes a framework (DPPO) to enable fine-tuning pre-trained diffusion-based policies through policy gradient methods. They compare DPPO with rich baselines in common benchmarks. They conclude best practices for DPPO and conduct ablation studies to understand the effectiveness of DPPO.
weaknesses: * The novelty of the paper is limited. Previous works have proposed the idea that considers a denoising process as multi-step MDP. This paper simply integrate this MDP into the environmental MDP. The tricks to adapt PPO for better fine-tuning are also seen in the previous works like GAE except fine-tuning the last few steps.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weaknesses
soundness: 3
strengths: (1) The paper introduces a novel dual-layer MDP framework and the use of PPO to fine-tune Diffusion-Policy, enhancing its performance in various tasks.
(2) The paper proposes several best practices for DPPO, such as fine-tuning only the last few denoising steps and using DDIM sampling to replace some denoising steps, improving efficiency and performance.
(3) DPPO shows a strong performance in all benchmark tests, particularly in tasks involving pixel-based observations and long-horizon rollouts.
(4)  DPPO achieves highly efficient zero-shot transfer from simulation to real-world tasks in robotics, demonstrating a minimal sim-to-real performance gap.
summary: The papers introduces Diffusion Policy Policy Optimization (DPPO), an algorithmic framework designed to fine-tune diffusion-based policies using policy gradient optimization (PPO) from reinforcement learning (RL). DPPO enables structured exploration, enhances training stability, and demonstrates the robustness and generalization during deployment. It improves policy performance across various benchmarks, including tasks with pixel-based observations and long-horizon rollouts, which have traditionally been challenging for previous RL methods.
weaknesses: (1) Figure 10 shows that the pre-trained Diffusion Policy can model multimodality in the behavior policy. However, after fine-tuning, the Diffusion Policy gradually converges into a deterministic policy. In this case, does it lose the advantage of expressing multimodal action distributions that Diffusion Policy is known for?

(2) In the experiments, DPPO is only compared against other diffusion-based methods, without any comparison to existing state-of-the-art Offline2Online or Sim2Real algorithms, such as Uni-O4 [1] and O3F [2]. Clearly, there is a significant optimal gap in environments like HalfCheetah.

(3) The motivation for fine-tuning models is typically to reduce the number of online interactions and improve sample efficiency. However, DPPO does not seem to significantly enhance sample efficiency, as its convergence speed is similar to other online reinforcement learning methods.

(4) The v-D4RL [3] dataset could have been considered in the experiments for broader evaluation.

(5) Compared to QSM [4], DPPO does not introduce particularly novel concepts but rather provides engineering or empirical adjustments, such as fine-tuning certain denoising steps and replacing value estimation with advantage estimation.

[1] Kun Lei, et al. Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
[2] Seunghyun Lee, et al. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
[3] Cong Lu, et al. Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
[4] Michael Psenka, et al. Learning a Diffusion Model Policy from Rewards via Q-Score Matching

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. Could you change the title "Diffusion Policy Policy Optimization" to provide more contexts to readers? Current title is too short and abstract and your work is not as general as it indicates. 
2. I would  suggest deleting/relocating some contents in early chapters as your paper is already over-sized and some of these contents are already known.
4. in line 249, the definition of $\gamma_{\text{ENV}}^t$ seems to be incorrect? I thinks it should be $\gamma_{\text{ENV}}^{t^\prime-t}$? or are there any further treatments of this discount factor?
4. there lacks a pseudo-code for the whole algorithm, please provide one. For example, how is your V function trained?
5. for fine-tuning control policy in RL, most existing algorithms finish in less than 1e6 steps. But the results presented throughout this paper is conducted in 1e6~1e7 steps. Could you truncate the figures to 5e5 such that we could better compare the performance of various methods?

6. Could you provide more comparisons of DPPO against established offline-to-online methods like [1] and [2]?

>[1] Wang, Shenzhi, et al. "Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
>
>[2] Zhang, Haichao, Wei Xu, and Haonan Yu. "Policy Expansion for Bridging Offline-to-Online Reinforcement Learning." The Eleventh International Conference on Learning Representations.

7. Could you provide more explanations about the worse performance of DAWR, DRWR and Cal-QL? In appendix, the result of the performance of Cal-QL on  Can and Square are almost zero, which is surprising to me. Could you elaborate more on this? BTW, what is the performance of Cal-QL when only offline training is performed?

8. for DRWR, in the weight function, could you minus a V function from the estimated return such that its’ more aligned toward DPPO?
9. in line 988-994, you mentioned speedup and less GPU memory usage is obtained by fine-tuning only the last few K denoising steps. But the method is to  use a trainable $\theta$ for the last K steps and a frozen $\theta_{\text{FT}}$ for previous steps. How does this method improve the training speed and reduce GPU memory?

10. Sect. 6 of structural exploration is of much importance and novelty. Could you try ablating on this phenomenon for DP? for example, by tuning the denoising process such that DP itself performs less or more structural exploration and see if the performance is affected?

There are many interesting, fruitful discoveries and experiments in this work. But I find none of them is studied exhaustively or presented clearly.
soundness: 2
strengths: This paper contains tons of discoveries related to fine-tuning diffusion policy via PPO. Specifically, numerous aspects like denoising steps to be fine-tuned, network architecture, GAE variants  and effects of expert data are all studied with experimental results reported. As fine-tuning diffusion policy for real-world robot is a very important field, these experiments are welcome and contain significant value  for the community. Last but not least, the authors also provide an empirical explanation for why their method obtains better results than others (i.e. structural exploration).

$~$

## Originality 
Utilizing RL method to finetune diffusion model is in great need. And the fruitful experimental results presented in this work  are of great value and originality.

## Quality

Many experiments and ablation studies are conducted and the best practices are summarized from them.

## Clarity

Unfortunately, this paper is not clearly written. In the main papers (1-10 pages), authors’ writing lacks a central logic  and the results are presented without  a focus. I have great trouble understanding the contribution and experimental results during reading. 

## Significance
The large amount of experiments conducted is of great value to the community, especially considering there are also real-world experiments.
summary: This paper studies how to use PPO to finetune the diffusion-model based policy in an offline-to-online setting. By integrating denoising steps of diffusion model into task MDP, authors could utilize PPO to finetune the diffusion model for improved performance. Extensive efforts are put to find the best practice in fine-tuning diffusion policy, in terms of hyperparameters, network architecture and algorithmic variants. Finally,  the efficacy and the performance of the proposed framework  is demonstrated on various experiments in both simulated and real-world scenarios.
weaknesses: Unfortunately, I personally think the presentation of this paper is very poor. With the extensive and fruitful experiments results, the authors fail to present them in a concise and focused manner in the main pages and make me quite confused during reading and comprehending this paper.

Firstly, there are three components that are actively mentioned and studied: the diffusion policy, the PPO method and the best practice for combining them together. 

But based on my comprehension, the diffusion policy and PPO used in this work are of no novel changes & contributions.

Combining these 2 together and successfully applying on real-world robotic system, is with great value and potential hidden evils. But  the authors fail to present this to me  in a clear and concise manner: for example, there are  experiments comparing diffusion policy (DP) to gaussian parameterized policy, but the advantage of DP to vanilla parameterization is already well-established and not the contribution of this paper, in my opinion. Since this paper is already over-sized, I suggest the authors to relocate some contents into appendix and focus on your main contribution in the main pages



To summarize, I hope the organization and presentation of this paper could be improved such that the problem formulation, method design and experimental results could be tightly chained and focused on  your main contribution.

Meta-review: This paper introduces a technique for fine-tuning diffusion policies with a policy gradient method. Previously, policy gradient updates of diffusion policies were conjectured to have training instability. With the new technique proposed, the authors show that the resulting algorithm called DPPO can fine-tune to superior policies both in simulations and real robots. Extensive comparison is provided. This paper has the potential to be adopted by many papers in the future and as such I recommend this paper for a spotlight presentation.

Minor comments:

> using the policy gradient (PG) method

Change to “Using a policy gradient method”

> As the first glance

Change to “At first glance”

Decision: Accept (Poster)