One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation

## Abstract

Abstract Diffusion models, praised for their success in generative tasks, are increasingly being applied to robotics, demonstrating exceptional performance in behavior cloning. However, their slow generation process stemming from iterative denoising steps poses a challenge for real-time applications in resource-constrained robotics setups and dynamically changing environments.
In this paper, we introduce the One-Step Diffusion Policy (OneDP), a novel approach that distills knowledge from pre-trained diffusion policies into a single-step action generator, significantly accelerating response times for robotic control tasks. We ensure the distilled generator closely aligns with the original policy distribution by minimizing the Kullback-Leibler (KL) divergence along the diffusion chain, requiring only 2 % percent 2 2\% 2 % - 10 % percent 10 10\% 10 % additional pre-training cost for convergence. We evaluated OneDP on 6 challenging simulation tasks as well as 4 self-designed real-world tasks using the Franka robot. The results demonstrate that OneDP not only achieves state-of-the-art success rates but also delivers an order-of-magnitude improvement in inference speed, boosting action prediction frequency from 1.5 Hz to 62 Hz, establishing its potential for dynamic and computationally constrained robotic applications. We share the project page here https://research.nvidia.com/labs/dir/onedp/ .

## 1 Introduction

Diffusion models (Sohl-Dickstein et al., ; Ho et al., ) have emerged as a leading approach to generative AI, achieving remarkable success in diverse applications such as text-to-image generation (Saharia et al., ; Ramesh et al., ; Rombach et al., ), video generation (Ho et al., ; OpenAI, ), and online/offline reinforcement learning (RL) (Wang et al., ; Chen et al., ; Hansen-Estruch et al., ; Psenka et al., ). Recently, Chi et al. (); Team et al. (); Reuss et al. (); Ze et al. (); Ke et al. (); Prasad et al. () demonstrated impressive results of diffusion models in imitation learning for robot control. In particular, Chi et al. () introduces the diffusion policy and achieves a state-of-the-art imitation learning performance on a variety of robotics simulation and real-world tasks.

However, because of the necessity of traversing the reverse diffusion chain, the slow generation process of diffusion models presents significant limitations for their application in robotic tasks. This process involves multiple iterations to pass through the same denoising network, potentially thousands of times (Song et al., ; Wang et al., ). Such a long inference time restricts the practicality of using the diffusion policy (Chi et al., ), which by default runs at $1.49$ Hz, in scenarios where quick response and low computational demands are essential. While classical tasks like block stacking or part assembly may accommodate slower inference rates, more dynamic activities involving human interference or changing environments require quicker control responses (Prasad et al., ). In this paper, we aim to significantly reduce inference time through diffusion distillation and achieve responsive robot control.

Considerable research has focused on streamlining the reverse diffusion process for image generation, aiming to complete the task in fewer steps.
A prominent approach interprets diffusion models using stochastic differential equations (SDE) or ordinary differential equations (ODE) and employs advanced numerical solvers for SDE/ODE to speed up the process (Song et al., ; Liu et al., ; Karras et al., ; Lu et al., ). Another avenue explores distilling diffusion models into generators that require only one or a few steps through Kullback-Leibler (KL) optimization or adversarial training (Salimans & Ho, ; Song et al., ; Luo et al., ; Yin et al., ). However, accelerating diffusion policies for robotic control has been largely underexplored. Consistency Policy (Prasad et al., ) (CP) employs the consistency trajectory model (CTM) (Kim et al., ) to adapt the pre-trained diffusion policy into a few-step CTM action generator. Despite this, several iterations for sampling are still required to maintain good empirical performance.

In this paper, we introduce the One-Step Diffusion Policy (OneDP), which distills knowledge from pre-trained diffusion policies into a one-step diffusion-based action generator, thus maximizing inference efficiency through a single neural network feedforward operation. We demonstrate superior results over baselines in Figure 1.
Inspired by the success of SDS (Poole et al., ) and VSD (Wang et al., ) in text-to-3D generation, we propose a policy-matching distillation method for robotic control. The training of OneDP consists of three key components: a one-step action generator, a generator score network, and a pre-trained diffusion-policy score network. To align the generator distribution with the pre-trained policy distribution, we minimize the KL divergence over diffused actions produced by the generator, with the gradient of the KL expressed as a score difference loss.
By initializing the action generator and the generator score network with the identical pre-trained model, our method not only preserves or enhances the performance of the original model, but also requires only $2\%$-$10\%$ additional pre-training cost for the distillation to converge. We compare our method with CP and demonstrate that it outperforms CP with a higher success rate across tasks, leveraging a single-step action generator and achieving 20$\times$ faster convergence. A detailed comparison with this approach is provided in Sections 4 and 3.

We evaluate our method in both simulated and real-world environments. In simulated experiments, we test OneDP on the six most challenging tasks of the Robomimic benchmark (Mandlekar et al., ). For real-world experiments, we design four tasks with increasing difficulty and deploy OneDP on a Franka robot arm. In both settings, OneDP demonstrated state-of-the-art success rates with single-step generation, performing $42\times$ faster in inference.

## 2 One-Step Diffusion Policy

### 2.1 Preliminaries

Diffusion models are powerful generative models applied across various domains (Ho et al., ; Sohl-Dickstein et al., ; Song et al., ). They function by defining a forward diffusion process that gradually corrupts the data distribution into a known noise distribution. Given a data distribution $p({\bm{x}})$, the forward process adds Gaussian noise to samples, ${\bm{x}}^{0}\sim p({\bm{x}})$, with each step defined as ${\bm{x}}^{k}=\alpha_{k}{\bm{x}}^{0}+\sigma_{k}{\bm{\epsilon}}_{k}$, where ${\bm{\epsilon}}_{k}\sim{\mathcal{N}}(\bm{0},{\bm{I}})$. The parameters $\alpha_{k}$ and $\sigma_{k}$ are manually designed and vary according to different noise scheduling strategies.

A probabilistic model $p_{\theta}({\bm{x}}^{k-1}|{\bm{x}}^{k})$ is then trained to reverse this diffusion process, enabling data generation from pure noise. DDPM (Ho et al., ) uses discrete-time scheduling with a noise-prediction model $\epsilon_{\theta}$ to parameterize $p_{\theta}$, while EDM (Karras et al., ) employs continuous-time diffusion with ${\bm{x}}^{0}$-prediction. We use epsilon prediction $\epsilon_{\theta}$ in our derivation. The diffusion model is trained using the denoising score matching loss (Ho et al., ; Song et al., ).

Once trained, we can estimate the unknown score $s({\bm{x}}^{k})$ at a diffused sample ${\bm{x}}^{k}$ as:

$$ $s({\bm{x}}^{k})=-\frac{\epsilon^{*}({\bm{x}}^{k},k)}{\sigma_{k}}\approx-\frac{ \epsilon_{\theta}({\bm{x}}^{k},k)}{\sigma_{k}},$ (1) $$

where $\epsilon^{*}({\bm{x}}^{k},k)$ is the true noise added at time $k$ and we denote $s_{\theta}({\bm{x}}^{k})=-\frac{\epsilon_{\theta}({\bm{x}}^{k},k)}{\sigma_{k}}$. With a score estimate, clean data ${\bm{x}}^{0}$ can be sampled by reversing the diffusion chain (Song et al., ). This requires multiple iterations through the estimated score network, making it inherently slow.

Wang et al. (); Chi et al. () extend diffusion models as expressive and powerful policies for offline RL and robotics. In robotics, a set of past observation images, ${\mathbf{O}}$, is used as input to the policy. An action chunk, ${\mathbf{A}}$, which consists of a sequence of consecutive actions, forms the output of the policy. Diffusion policy is represented as a conditional diffusion-based action prediction model,

$$ $\pi_{\theta}({\mathbf{A}}^{0}|{\mathbf{O}}):=\int\cdots\int{\mathcal{N}}({ \mathbf{A}}^{K};\bm{0},{\bm{I}})\prod_{k=K}^{k=1}p_{\theta}({\mathbf{A}}^{k-1} |{\mathbf{A}}^{k},{\mathbf{O}})d{\mathbf{A}}^{K}\cdots d{\mathbf{A}}^{1},$ (2) $$

The explicit form of $\pi_{\theta}({\mathbf{A}}^{0}|{\mathbf{O}})$ is often impractical due to the complexity of integrating actions from ${\mathbf{A}}^{K}$ to ${\mathbf{A}}^{1}$. However, we can obtain action chunk samples from it by iterative denoising. More details are provided in Appendix D

### 2.2 One-Step Diffusion Policy

Figure: Figure 2: Diffusion Distillation Pipeline. a) Our one-step action generator processes image-based visual observations alongside a random noise input to deliver single-step action predictions. b) We implement KL-based distillation across the entire forward diffusion chain. Direct computation of the KL divergence is often impractical; however, we can effectively utilize the gradient of the KL, formulated into a score-difference loss. The pre-trained score network $\pi_{\phi}$ remains fixed while the action generator $G_{\theta}$ and the generator score network $\pi_{\psi}$ are trained.
Refer to caption: x6.png

Action sampling through the vanilla diffusion policies is notoriously slow due to the need of tens to hundreds of iterative inference steps. The latency issue is critical for computationally sensitive robotic tasks or tasks that require high control frequency.
Although employing advanced ODE solvers (Song et al., ; Karras et al., ) could help speed up the sampling procedure, empirically at least ten iterative steps are required to ensure reasonable performance. Here, we introduce a training-based diffusion policy distillation method, which distills the knowledge of a pre-trained diffusion policy into a single-step action generator, enabling fast action sampling.

We propose a one-step implicit action generator $G_{\theta}$, from which actions can be easily obtained as follows,

$$ ${\bm{z}}\sim{\mathcal{N}}(\bm{0},{\bm{I}}),{\mathbf{A}}_{\theta}=G_{\theta}({ \bm{z}},{\mathbf{O}}).$ (3) $$

We define the action distribution generated by $G_{\theta}$ as $p_{G_{\theta}}$. Assuming the existence of a pre-trained diffusion policy $\pi_{\phi}({\mathbf{A}}|{\mathbf{O}})$ defined by Equation 2 and parameterized by $\epsilon_{\phi}$, its corresponding action distribution is denoted as $p_{\pi_{\phi}}$. Drawing inspiration from the success of SDS (Poole et al., ) and VSD (Wang et al., ) in text-to-3D applications, we propose using the following reverse KL divergence to align the distributions $p_{G_{\theta}}$ and $p_{\pi_{\phi}}$,

$$ ${\mathcal{D}}_{KL}(p_{G_{\theta}}||p_{\pi_{\phi}})=\mathbb{E}_{{\bm{z}}\sim{ \mathcal{N}}(\bm{0},{\bm{I}}),{\mathbf{A}}_{\theta}=G_{\theta}({\bm{z}},{ \mathbf{O}})}\left[\log p_{G_{\theta}}({\mathbf{A}}_{\theta}|{\mathbf{O}})- \log p_{\pi_{\phi}}({\mathbf{A}}_{\theta}|{\mathbf{O}})\right].$ $$

It is generally intractable to estimate this loss by directly computing the probability densities, since $p_{G_{\theta}}$ is an implicit distribution and $p_{\pi_{\phi}}$ involves integrals that are impractical (Equation 2). However, we only need the gradient with respect to $\theta$ to train our generator by gradient descent:

$$ $\nabla_{\theta}{\mathcal{D}}_{KL}(p_{G_{\theta}}||p_{\pi_{\phi}})=\mathbb{E}_{ \scriptstyle\begin{subarray}{c}{\bm{z}}\sim{\mathcal{N}}(\bm{0},{\bm{I}}),\\ {\mathbf{A}}_{\theta}=G_{\theta}({\bm{z}},{\mathbf{O}})\end{subarray}}\left[( \nabla_{{\mathbf{A}}_{\theta}}\log p_{G_{\theta}}({\mathbf{A}}_{\theta}|{ \mathbf{O}})-\nabla_{{\mathbf{A}}_{\theta}}\log p_{\pi_{\phi}}({\mathbf{A}}_{ \theta}|{\mathbf{O}}))\nabla_{\theta}{\mathbf{A}}_{\theta}\right].$ (4) $$

Here $s_{p_{G_{\theta}}}({\mathbf{A}}_{\theta})=\nabla_{{\mathbf{A}}_{\theta}}\log p
_{G_{\theta}}({\mathbf{A}}_{\theta}|{\mathbf{O}})$ and $s_{p_{\pi_{\phi}}}({\mathbf{A}}_{\theta})=\nabla_{{\mathbf{A}}_{\theta}}\log p
_{\pi_{\phi}}({\mathbf{A}}_{\theta}|{\mathbf{O}})$ are the scores of the $p_{G_{\theta}}$ and $p_{\pi_{\phi}}$ respectively. Computing this gradient still presents two significant challenges: First, the scores tend to diverge for samples from $p_{G_{\theta}}$ that have a low probability in $p_{\pi_{\phi}}$, especially when $p_{\pi_{\phi}}$ may approach zero. Second, the primary tool for estimating these scores, the diffusion models, only provides scores for the diffused distribution.

Inspired by Diffusion-GAN (Wang et al., ), which proposed to optimize statistical divergence, such as the Jensen–Shannon divergence (JSD), throughout diffused data samples, we propose to similarly optimize the KL divergence outlined in Equation 4 across diffused action samples as described below:

$$ $\nabla_{\theta}\mathbb{E}_{k\sim{\mathcal{U}}}[{\mathcal{D}}_{KL}(p_{G_{\theta },k}||p_{\pi_{\phi},k})]=\mathbb{E}_{\scriptstyle\begin{subarray}{c}{\bm{z}} \sim{\mathcal{N}}(\bm{0},{\bm{I}}),k\sim{\mathcal{U}}\\ {\mathbf{A}}_{\theta}=G_{\theta}({\bm{z}},{\mathbf{O}})\\ {\mathbf{A}}_{\theta}^{k}\sim q({\mathbf{A}}_{\theta}^{k}|{\mathbf{A}}_{\theta },k)\end{subarray}}\left[w(k)(s_{p_{G_{\theta}}}({\mathbf{A}}_{\theta}^{k})-s_ {p_{\pi_{\phi}}}({\mathbf{A}}_{\theta}^{k}))\nabla_{\theta}{\mathbf{A}}_{ \theta}^{k}\right].$ (5) $$

where $w(k)$ is a reweighting function, $q$ is the forward diffusion process and $s_{p_{\pi_{\phi}}}({\mathbf{A}}_{\theta}^{k})$ could be obtained through Equation 1 with $\epsilon_{\phi}$.
In order to estimate the score of the generator distribution, $s_{p_{G_{\theta}}}$, we introduce an auxiliary diffusion network $\pi_{\psi}({\mathbf{A}}|{\mathbf{O}})$, parameterized by $\epsilon_{\psi}$.
We follow the typical way of training diffusion policies, which optimizes $\psi$ by treating $p_{G_{\theta}}$ as the target action distribution (Wang et al., ),

$$ $\min_{\psi}\mathbb{E}_{{\bm{x}}^{k}\sim q({\bm{x}}^{k}|{\bm{x}}^{0}),{\bm{x}}^ {0}=\text{stop-grad}(G_{\theta}({\bm{z}})),{\bm{z}}\sim{\mathcal{N}}(\bm{0},{ \bm{I}}),k\sim{\mathcal{U}}}[\lambda(k)\cdot||\epsilon_{\psi}({\bm{x}}^{k},k)- {\bm{\epsilon}}_{k}||^{2}].$ (6) $$

Then we can obtain $s_{p_{\pi_{\psi}}}({\mathbf{A}}_{\theta}^{k})$ by applying $\epsilon_{\psi}$ to Equation 1. We approximate $s_{p_{G_{\theta}}}({\mathbf{A}}_{\theta}^{k})$ in Equation 5 with $s_{p_{\pi_{\psi}}}({\mathbf{A}}_{\theta}^{k})$. We iteratively update the generator parameters $\theta$ by Equation 5, and the generator score network parameter $\psi$ by Equation 6. The parameter of the prertrained diffusion policy $\phi$ is fixed throughout the training. During inference, we directly perform one-step sampling with Equation 3. We name our algorithm OneDP-S, where S denotes the stochastic policy.

When we apply a deterministic action generator by omitting random noise ${\bm{z}}$, such that ${\mathbf{A}}_{\theta}=G_{\theta}({\mathbf{O}})$, the distribution $p_{G_{\theta}}$ becomes a Dirac delta function centered at $G_{\theta}({\mathbf{O}})$, that is, $p_{G_{\theta}}=\delta_{G_{\theta}({\mathbf{O}})}({\mathbf{A}})$. Consequently, $s_{p_{G_{\theta}}}({\mathbf{A}}_{\theta}^{k})$ can be explicitly solved as follows:

$$ $s_{p_{G_{\theta}}}({\mathbf{A}}_{\theta}^{k})=\nabla_{{\mathbf{A}}_{\theta}^{k }}\log p_{\theta}({\mathbf{A}}_{\theta}^{k})=\nabla_{{\mathbf{A}}_{\theta}^{k} }\log p_{\theta}({\mathbf{A}}_{\theta}^{k}|{\mathbf{A}}_{\theta})=-\frac{{\bm{ \epsilon}}_{k}}{\sigma_{k}};{\mathbf{A}}_{\theta}^{k}=\alpha_{k}{\mathbf{A}}_{ \theta}+\sigma_{k}{\bm{\epsilon}}_{k},{\bm{\epsilon}}_{k}\sim{\mathcal{N}}(\bm {0},{\bm{I}}).$ (7) $$

By incorporating Equation 7 into Equation 5, we can have a simplified loss function without the need of introducing the generator score network:

$$ $\nabla_{\theta}\mathbb{E}_{k\sim{\mathcal{U}}}[{\mathcal{D}}_{KL}(p_{G_{\theta },k}||p_{\pi_{\phi},k})]=\mathbb{E}_{\scriptstyle\begin{subarray}{c}{\bm{z}} \sim{\mathcal{N}}(\bm{0},{\bm{I}}),k\sim{\mathcal{U}}\\ {\mathbf{A}}_{\theta}=G_{\theta}({\bm{z}},{\mathbf{O}})\\ {\mathbf{A}}_{\theta}^{k}\sim q({\mathbf{A}}_{\theta}^{k}|{\mathbf{A}}_{\theta },k)\end{subarray}}\left[\frac{w(k)}{\sigma_{k}}(\epsilon_{\phi}({\mathbf{A}}_ {\theta}^{k},k))-\epsilon_{k})\nabla_{\theta}{\mathbf{A}}_{\theta}^{k}\right].$ (8) $$

We name this deterministic diffusion policy distillation OneDP-D. We illutrate our training pipeline in Figure 2, and summarize our algorithm training in Algorithm 1.

Policy Discussion. A stochastic policy, which encompasses deterministic policies, is more versatile and better suited to scenarios requiring exploration, potentially leading to better convergence at a global optimum (Haarnoja et al., ). In our case, OneDP-D simplifies the training process, though it may exhibit slightly weaker empirical performance. We offer a comprehensive comparison between OneDP-S and OneDP-D in Section 3.

Distillation Discussion. We discuss the benefits of optimizing the expectational reverse KL divergence. First, reverse KL divergence typically induces mode-seeking behavior, which has been shown to improve empirical performance in offline RL (Chen et al., ). Therefore, we anticipate that reverse KL-based distillation offers similar advantages for robotic tasks. Second, as demonstrated by Wang et al. (), optimizing JSD, a combination of KLs, between diffused action samples provides stronger performance when dealing with distributions with misaligned supports. This aligns with our approach of performing KL optimization over the diffused distribution.

Figure: Algorithm 1 OneDP Training

### 2.3 Implementation Details

Diffusion Policy. Following Chi et al. (), we construct a diffusion policy using a 1D temporal convolutional neural network (CNN) (Janner et al., ) based U-Net and a standard ResNet18 (without pre-training) (He et al., ) as the vision encoder. We implement the diffusion policy with two noise scheduling methods: DDPM (Ho et al., ) and EDM (Karras et al., ). We use $\epsilon$ noise prediction for discrete-time (100 steps) diffusion and $x^{0}$ prediction for continuous-time diffusion, respectively. The EDM scheduling is essential for Consistency Policy (Prasad et al., ) due to the use of CTM (Kim et al., ). For DDPM, we set $\lambda(k)=1$ and use the original SDE and DDIM (Song et al., ) sampling.
For EDM, we use the default $\lambda(k)=\frac{\sigma_{k}^{2}+\sigma_{d}^{2}}{(\sigma_{k}\sigma_{d})^{2}}$ with $\sigma_{d}=0.5$.
We use the second-order EDM sampler, which requires two neural network forwards per discretized step in the ODE.

Distillation. We warm-start both the stochastic and deterministic action generator $G_{\theta}$, and the generator score network, $\epsilon_{\psi}$, by duplicating the neural-network structure and weights from the pre-trained diffusion policy, aligning with strategies from Luo et al. (); Yin et al. (); Xu et al. (). Following DreamFusion (Poole et al., ), we set $w(k)=\sigma_{k}^{2}$. In the discrete-time domain, distillation occurs over [2, 95] diffusion timesteps to avoid edge cases. In continuous-time, we employ the same log-normal noise scheduling as EDM (Karras et al., ) used during distillation. The generators operate at a learning rate of $1\times 10^{-6}$, while the generator score network is accelerated to a learning rate of $2\times 10^{-5}$. Vision encoders are also actively trained during the distillation process.

## 3 Experiments

We evaluate OneDP on a wide variety of tasks in both simulated and real environments. In the following sections, we first report the evaluation results in simulation across six tasks that include different complexity levels. Then we demonstrate the results in the real environment by deploying OneDP in the real world with a Franka robot arm for object pick-and-place tasks and a coffee-machine manipulation task. We compare our method with the pre-trained backbone Diffusion Policy (Chi et al., ) (DP) and related distillation baseline Consistency Policy (Prasad et al., ) (CP). We also report the ablation study results in Appendix C to present more detailed analyses on our method and discuss the effect of different design choices.

### 3.1 Simulation experiments

Figure: Figure 3: Simulation tasks. We evaluate our method against baselines on the single-robot tasks: PushT, Square, and ToolHang, as well as a dual-robot task Transport. Task difficulty increases from left to right.
Refer to caption: x7.png

Datasets. Robomimic. Proposed in (Mandlekar et al., ), Robomimic is a large-scale benchmark for robotic manipulation tasks. The original benchmark consists of five tasks: Lift, Can, Square, Transport, and Tool Hang. We find that the the performance of state-of-the-art methods was already saturated on two easy tasks Lift and Can, and therefore only conduct the evaluation on the harder tasks Square, Transport and Tool Hang. For each of these tasks, the benchmark provides two variants of human demonstrations: proficient human (PH) demonstrations and mixed proficient/non-proficient human (MH) demonstrations. PushT. Adapted from IBC (Florence et al., ), Chi et al. () introduced the PushT task, which involves pushing a T-shaped block into a fixed target using a circular end-effector. A dataset of 200 expert demonstrations is provided with RGB image observations.

Experiment Setup. We pretrain the DP model for 1000 epochs on each benchmark under both DDPM (Ho et al., ) and EDM (Karras et al., ) noise scheduling. Note EDM noise scheduling is a requirement for CP (Prasad et al., ) to satisfy diffusion boundary conditions. Subsequently, we train OneDP for 20 epochs and the baseline CP for 450 epochs until convergence. During evaluation, we observe significant variance in evaluating success rates with different environment initializations. We present average success rates across 5 training seeds and 100 different initial
conditions (500 in total). We report the peak success rate for each method during training, corresponding to the peak points of the curves in Figure 4.
The metric for most tasks is the success rate, except for PushT, which is evaluated using the coverage of the target area.

Table 1 presents the results of OneDP compared with DP under the default DDPM setting. For DP, we report the average success rate using DDPM sampling with 100 timesteps, as well as the accelerated DDIM sampling with 1 and 10 timesteps. Notably, DP fails to generate reasonable actions with single-step generation, yielding a 0% success rate for all tasks. DP with 10 steps under DDIM slightly outperforms DP under DDPM. However, OneDP demonstrates the highest average success rate with single-step generation across the six tasks, with the stochastic variant OneDP-S surpassing the deterministic OneDP-D. This superior performance of OneDP-S aligns with our discussion in Section 2.2, suggesting that stochastic policies generally perform better in complex environments. Interestingly, OneDP-S even slightly outperforms the pre-trained DP, which is not unprecedented, as shown in cases of image distillation (Zhou et al., ) and offline RL (Chen et al., ). We attribute this to the fact that iterative sampling may introduce subtle cumulative errors during the denoising process, whereas single-step sampling avoids this issue by jumping directly from the end to the start of the reverse diffusion chain.

In Table 2, we report a similar comparison under the EDM setting, including CP. We report DP under the same 1 and 10 DDIM steps, and 100 DDPM steps, which correspond to 1, 19, and 35 number of function evaluations (NFE) in EDM due to second-order ODE sampling. OneDP-S outperforms the baseline CP with single-step and its default best setting of 3-step chain generation. Under EDM, OneDP-S matches the average success rate of the pre-trained DP, while OneDP-D performs slightly worse. We also observe that CP converges much more slowly compared to OneDP, as shown in Figure 4. This slower convergence is likely because CP, based on CTM, does not involve the auxiliary discriminator training that is used to enhance distillation performance in CTM.

Figure: Figure 4: Convergence Comparison. We show our method OneDP converges $20\times$ faster than the baseline method Consistency Policy (CP) under EDM setting.
Refer to caption: extracted/5959465/figs/4-experiments/plot_pusht_image_ph_zoomed.png

### 3.2 Real world experiments

We design four tasks to evaluate the real-world performance of OneDP, including three common tasks where the robot picks and places objects at designated locations, referred to as pnp, and one challenging task where the robot learns to manipulate a coffee machine, called coffee. Figure 5 shows the experimental setup, with the first row illustrating the pnp tasks and the second row depicting the coffee task. We introduce the data collection process and the evaluation setup in the following section and provide more details in Appendix A.

pnp Tasks. This task requires the robot to pick an object from the table and put it in a box. We design three variants of this task: pnp-milk, pnp-anything and pnp-milk-move. In pnp-milk, the object is always the same milk box. In pnp-anything, we expand the target to 11 different objects as shown in Figure 8. For pnp-milk-move, we involve human interference to create a dynamic environment. Whenever the robot gripper attempts to grasp the milk box, we move it away, following the trajectory as shown in Figure 9. We collect 100 demonstrations each for the pnp-milk and pnp-anything tasks. Separate models are trained for both tasks, with the pnp-anything model utilizing all 200 demonstrations. The pnp-milk-move task is evaluated using the checkpoint from the pnp-anything model.

Coffee Task. This task requires the robot to operate a coffee machine. It involves the following steps: (1) picking up the coffee pod, (2) placing the coffee pod in the pod holder on the coffee machine, and (3) closing the lid of the coffee machine. This task is more challenging since it involves more steps and requires the robot to insert the pod in the holder accurately. We collect 100 human demonstrations for this task. We train one specific model for this task.

Evaluation. We evaluate the success rate and task completion time from 20 predetermined initial positions for the pnp-milk, pnp-anything, and coffee tasks, as well as 10 motion trajectories for the pnp-milk-move task. The left side of Figure 7 shows the setup of the robot, destination box, and coffee machine, with 20 fixed initialization points. Figure 9 shows the 10 trajectories for evaluating pnp-milk-move. Details of the evaluation are provided in Appendix A.
For DP, we follow Chi et al. () to use DDIM (10 steps) to accelerate the real-world experiment.

We compare OneDP against the DP backbone in real-world experiments, focusing on three key aspects: success rate, responsiveness, and time efficiency. Table 3 demonstrates that OneDP consistently outperforms DP across all tasks, with the most significant improvement seen in pnp-milk-move. This task demands rapid adaptation to dynamic environmental changes, particularly due to sudden human interference. The wall-clock time for action generation is reported in Table 5. The slow action generation of DP hinders its ability to track the moving milk box effectively, often losing control when the box moves out of its visual range, as it is still predicting actions based on outdated information. In contrast, OneDP generates actions quickly, allowing it to instantly follow the box’s movement, achieving a 100% success rate in this dynamic task. OneDP-S slightly outperforms OneDP-D, aligning with the observations from the simulation experiments.

Additionally, we measure the task completion time for successful evaluation rollouts across all algorithms. As shown in Table 4, OneDP completes tasks faster than DP. Both OneDP-S and OneDP-D exhibit similarly-rapid task completion times. The quick action prediction of OneDP reduces hesitation during robot arm movements, particularly when the arm camera’s viewpoint changes abruptly. This leads to significant improvements in task completion speed. In Figure 7, we present a heatmap for illustrating the task completion times; lighter colors indicate faster completion times, while dark red demonstrates failure cases. Overall, OneDP completes tasks more efficiently across most locations. Although all three algorithms encounter failures in some corner cases for the coffee task, OneDP-S shows fewer failures.

Figure: Figure 5: Real-World Experiment Illustration. In the first row, we display the setup for the pick-and-place experiments, featuring three tasks: pnp-milk, pnp-anything, and pnp-milk-move. In total, pnp-anything handles around 10 random objects as shown in Figure 8. The second row illustrates the procedure for the more challenging coffee task, where the Franka arm is tasked with locating the coffee cup, precisely positioning it in the machine’s cup holder, inserting it, and finally closing the machine’s lid.
Refer to caption: x8.png

**Table 5: Real-world inference speeds. We report the wall clock times for each policy in real-world scenarios. The action generation process consists of two parts: observation encoding (OE) and action prediction by each method. All measurements were taken using a local NVIDIA V100 GPU, with the same neural network size for each method. The policy frequencies, shown in Figure 1, are based on the values from this table.**
|  | OE | DDPM (100 steps) | DDIM (10 steps) | OneDP (1 step) |
| --- | --- | --- | --- | --- |
| Time (ms) | 9 | 660 | 66 | 7 |
| NFE | 1 | 100 | 10 | 1 |

## 4 Related Work

Diffusion Models. Diffusion models have emerged as a powerful framework for modeling complex data distributions and have achieved groundbreaking performance across various tasks involving generative modeling (Ho et al., ; Karras et al., ).
They operate by transforming data into Gaussian noise through a diffusion process and subsequently learning to reverse this process via iterative denoising.
Diffusion models have been successfully applied to a wide range of domains, including image, video, and audio generation Saharia et al. (); Ramesh et al. (); Balaji et al. (); Chen et al. (); Ho et al. (); Popov et al. (); Kong et al. (), reinforcement learning (Janner et al., ; Wang et al., ; Psenka et al., ) and robotics (Ajay et al., ; Urain et al., ; Chi et al., ).

Diffusion Policies. Diffusion models have shown promising results as policy representations for control tasks.
Janner et al. () introduced a trajectory-level diffusion model that predicts all timesteps of a plan simultaneously by denoising two-dimensional arrays of state and action pairs.
Wang et al. () proposed Diffusion Q-learning, which leverages a conditional diffusion model to represent
the policy in offline reinforcement learning. An action-space diffusion model is trained to generate actions conditioned on the states.
Similarly, Chi et al. () used a conditional diffusion model in the robot action space to represent the visuomotor policy and demonstrated a significant performance boost in imitation learning for various robotics tasks. Ze et al. () further incorporated the power of a compact 3D visual representations to improve diffusion policies in robotics.

Diffusion Distillations. Although diffusion models are powerful, their iterative denoising process makes them inherently slow in generation, which poses challenges for time-sensitive applications like robotics and real-time control. Motivated by the need to accelerate diffusion models, diffusion distillation has become an active research topic in image generation. Diffusion distillation aims to train a student model that can generate samples with fewer denoising steps by distilling knowledge from a pre-trained teacher model (Salimans & Ho, ; Luhman & Luhman, ; Zheng et al., ; Song et al., ; Kim et al., ). Salimans & Ho () proposed a method to distill a teacher model into a new model that takes half the number of sampling steps, which can be further reduced by progressively applying this procedure. Song et al. () introduced consistency models that enable fewer step sampling by enforcing self-consistency of the ODE trajectories. CTM (Kim et al., ) improved consistency models and provided the flexibility to trade-off quality and speed.
(Luo et al., ; Yin et al., ) leverage the success of stochastic distillation sampling (Poole et al., ) in text-to-3D and proposes KL-based score distillation for image generation. Beyond KL, Zhou et al. () proposes the SiD distillation technique derived from Fisher Divergence.
However, leveraging diffusion distillation to accelerate diffusion policies for robotics remains an underexplored and pressing challenge, particularly for real-time control applications. Consistency Policy (Prasad et al., ) explored applying CTM to reduce the number of denoising steps and accelerate inference of the diffusion policies. It simplifies the original CTM training by ignoring the adversarial auxiliary loss.
While this approach achieves a considerable speed-up, it leads to performance degradation compared to pre-trained models, and its complex training process and slow convergence present challenges for robotics applications.
In contrast, OneDP employs expectational reverse KL optimization to distill a powerful one-step action generator, achieving comparable or higher success rates than the original diffusion policy, while converging 20$\times$ faster.

## 5 Conclusion

In this paper, we introduced the One-Step Diffusion Policy (OneDP) through advanced diffusion distillation techniques. We enhanced the slow, iterative action prediction process of Diffusion Policy by reducing it to a single-step process, dramatically decreasing action inference time and enabling the robot to respond quickly to environmental changes. Through extensive simulation and real-world experiments, we demonstrate that OneDP not only achieves a slightly higher success rate, but also responds quickly and effectively to environmental interference. The rapid action prediction further allows the robot to complete tasks more efficiently.

However, this work has some limitations. In the experiments, we did not test OneDP on long-horizon real-world tasks. Furthermore, in the real-world experiments, we limited the robot’s operation frequency to 20 Hz for controlling stability, which underutilized OneDP ’s full potential. Additionally, the KL-based distillation method may not be the optimal choice for distribution matching, and introducing a discriminator term could potentially improve distillation performance.

## Appendix A Real-World Experiment Setup

Figure: Figure 6: Real-world Experiment Setup
Refer to caption: extracted/5959465/figs/appendix/real_setup.jpg

#### Robot Setup.

The physical robot setup consists of a Franka Panda robot arm, a front-view Intel RealSense D415 RGB-D camera, and a wrist-mounted Intel RealSense D435 RGB-D camera. The RGB image resolution was set to 120x160. The depth image is not used in our experiments.

#### Teleoperation.

Demonstration data for the real robot tasks was collected using a phone-based teleoperation system (Mandlekar et al., ; ).

#### Data Collection.

We collect 100 demonstrations for each task separately: pnp-milk, pnp-anything, and coffee. In pnp-milk, the target object is always the milk box, and the task involves picking up the milk box from various random locations and placing it into a designated target box at a fixed location. For pnp-anything, we extend the set of target objects to 11 different items, as shown in Figure 8, with the target box location randomized vertically. In the coffee task, the coffee cup is randomly placed, and the robot is required to pick it up, insert it into the coffee machine, and close the lid.

The area and location for each task are illustrated in the left column of Figure 7. During data collection, target objects are randomly positioned within the blue area; the grid is used for evaluation, as described in the next section.
For the pnp tasks, the blue area is a rectangle measuring 23 cm in height and 20 cm in width, while the target box is a square with a side length of 13 cm. In the coffee task, the blue area is slightly smaller, measuring 18 cm in height and 20 cm in width.

**Table 6: Real-world experiment demonstrations. In total we collect 300 demonstrations, with 100 demonstrations for each task.**
|  | pnp-milk | pnp-anything | coffee |
| --- | --- | --- | --- |
| Demos | 100 | 100 | 100 |

#### Evaluation.

To ensure a fair comparison between OneDP and all baseline methods, we standardize the evaluation process. For the pnp-milk, pnp-anything, and coffee tasks, we evaluate each method according to the grid order shown in Figure 7. The target object is placed at the center of the grid to ensure consistent initial conditions across evaluations. For task pnp-anything, the picked object also follows the order shown in Figure 8.
For the dynamic environment task pnp-milk-move, we introduce human interference during the evaluation. Whenever the robot gripper attempts to grasp the target milk box, we manually move it away along the trajectory depicted in Figure 9. Although we aim to maintain consistent conditions during each evaluation, the exact nature of human interference cannot be guaranteed. Some trajectories involve a single instance of interference, while others may involve two consecutive human movements.

The original DDPM sampling in Diffusion Policy is too slow for real-world experiments. To speed up the evaluation, we follow (Chi et al., ) and use DDIM with 10 steps. For OneDP, we use single-step generation. In real-world experiments, we do not select intermediate checkpoints but use the final checkpoint after training for each method.

We record both the success rates and completion times, reporting their mean values. For pnp-milk-move, evaluations are conducted over 10 trajectories, while for the other tasks, results are obtained from 20 grid points. In Figure 7, we present a heatmap to visualize task completion times, where lighter colors represent faster completions and dark red indicates failure cases. Overall, OneDP completes tasks more efficiently across most locations. While all three algorithms experience failures in certain corner cases for the coffee task, OneDP-S demonstrates fewer failures.

Figure: Figure 7: Real-World Comparison Illustration. We present the time taken by each algorithm to complete tasks from a specific starting point in colors. A color map on the right side ranges from white to red indicating the time in seconds. Dark red signifies that the algorithm failed at that location. The three rows represent tasks pnp-milk, pnp-anything, coffee. Details of the evaluation of pnp-anything can be found in Figure 8.
Refer to caption: x9.png

Figure: Figure 8: Evaluation setup for pnp-anything.
Refer to caption: x10.png

Figure: Figure 9: Evaluation trajectories for pnp-milk-move. The box is always on the left-hand side of the tested blue area.
Refer to caption: x11.png

## Appendix B Training Details

We follow the CNN-based neural network architecture and observation encoder design from Chi et al. (). For simulation experiments, we use a 256-million-parameter version for DDPM and a 67-million-parameter version for EDM, as the smaller EDM network performs slightly better. In real-world experiments, we also use the 67-million-parameter version. Additionally, we adopt the action chunking idea from Chi et al. () and Zhao et al. (), using 16 actions per chunk for prediction, and utilize two observations for vision encoding.

We first train DP for 1000 epochs in both simulation and real-world experiments with a default learning rate of 1e-4 and weight decay of 1e-6. We then perform distillation using the pre-trained checkpoints, distilling for 20 epochs in simulation and 100 epochs in real-world experiments.

For distillation, we warm-start both the stochastic and deterministic action generators, $G_{\theta}$, and the generator score network, $\epsilon_{\psi}$, by duplicating the network structure and weights from the pre-trained diffusion-policy checkpoints. Since the generator network is initialized from a denoising network, a timestep input is required, as this was part of the original input. We fix the timestep at 65 for discrete diffusion and choose $\sigma=2.5$ for continuous EDM diffusion. The generator learning rate is set to 1e-6. We find these hyperparameters to be stable without causing significant performance variation. We provide an ablation study that focuses primarily on the generator score network’s learning rate and optimizer settings in Appendix C. We provide the hyperparameter details in Table 7.

**Table 7: Hyperparameters**
| Hyperparameters | Values |
| --- | --- |
| generator learning rate | lr=1e-6 |
| generator score network learning rate | lr=2e-5 |
| generator optimizer | Adam([0.0, 0.999]) |
| generator score network optimizer | Adam([0.0, 0.999]) |
| action chunk size | n=16 |
| number of observations | n=2 |
| discrete diffusion init timestep | $t_{\text{init}}$=65 |
| discrete diffusion distillation $t$ range | [2, 95] |
| continuous diffusion init sigma | $\sigma=2.5$ |

## Appendix C Ablation Study

As shown in the first panel of Figure 10, we explore a range of learning rates for the generator score network in the grid [1e-6, 1e-5, 2e-5, 3e-5, 4e-5] and find 2e-5 to be optimal in most cases. A higher learning rate for the score network compared to the generator ensures that the score network keeps pace with the generator’s distribution updates during training. In the second panel, we search for the best optimizer settings, finding that setting $\beta_{1}$ to 0 for both the generator and the generator score network optimizers is effective. This approach, commonly used in GANs, allows the two networks to evolve together more quickly.

Figure: Figure 10: Ablation studies on the learning rate of the generator score network and optimizer settings.
Refer to caption: extracted/5959465/figs/appendix/ablation_lr.png

## Appendix D Detailed Preliminaries

Diffusion models are robust generative models utilized across various domains (Ho et al., ; Sohl-Dickstein et al., ; Song et al., ). They operate by establishing a forward diffusion process that incrementally transforms the data distribution into a known noise distribution, such as standard Gaussian noise. A probabilistic model is then trained to methodically reverse this diffusion process, enabling the generation of data samples from pure noise.

Suppose the data distribution is $p({\bm{x}})$. The forward diffusion process is conducted by gradually adding Gaussian noise to samples ${\bm{x}}^{0}\sim p({\bm{x}})$ as follows,

$$ ${\bm{x}}^{k}=\alpha_{k}{\bm{x}}^{0}+\sigma_{k}{\bm{\epsilon}}_{k},{\bm{ \epsilon}}_{k}\sim{\mathcal{N}}(\bm{0},{\bm{I}});\quad q({\bm{x}}^{k}|{\bm{x}} ^{0}):={\mathcal{N}}(\alpha_{k}{\bm{x}}^{0},\sigma_{k}^{2}{\bm{I}})$ $$

where $\alpha_{k}$ and $\sigma_{k}$ are parameters manually designed to vary according to different noise scheduling strategies. DDPM (Ho et al., ) is a discrete-time diffusion model with $k\in\{1,\dots,K\}$. It can be easily extended to continuous-time diffusion from the score-based generative model perspective (Song et al., ; Karras et al., ) with $k\in[0,1]$. With sufficient amount of noise added, ${\bm{x}}^{K}\simeq{\mathcal{N}}(\bm{0},{\bm{I}})$.
Ho et al. () propose to reverse the diffusion process and iteratively reconstruct the original sample ${\bm{x}}^{0}$ by training a neural network $\epsilon_{\theta}({\bm{x}}^{k},k)$ to predict the noise ${\bm{\epsilon}}_{k}$ added at each forward diffusion step (epsilon prediction).
With reparameterization ${\bm{\epsilon}}_{k}=({\bm{x}}^{k}-\alpha_{k}{\bm{x}}^{0})/\sigma_{k}$, the diffusion model could also be formulated as a ${\bm{x}}^{0}$-prediction process $x_{\theta}({\bm{x}}^{k},k)$ (Karras et al., ; Xiao et al., ). We use epsilon prediction $\epsilon_{\theta}$ in our derivation. The diffusion model is trained with the denoising score matching loss (Ho et al., ),

$$ $\min_{\theta}\mathbb{E}_{{\bm{x}}^{k}\sim q({\bm{x}}^{k}|{\bm{x}}^{0}),{\bm{x} }^{0}\sim p({\bm{x}}),k\sim{\mathcal{U}}}[\lambda(k)\cdot||\epsilon_{\theta}({ \bm{x}}^{k},k)-{\bm{\epsilon}}_{k}||^{2}]$ $$

where ${\mathcal{U}}$ is a uniform distribution over the $k$ space, and $\lambda(k)$ is a noise-ratio re-weighting function. With a trained diffusion model, we could sample ${\bm{x}}^{0}$ by reversing the diffusion chain, which involves discretizing the ODE (Song et al., ) as follows:

$$ $d{\bm{x}}^{k}=\left[f(k){\bm{x}}^{k}-\frac{1}{2}g^{2}(k)\nabla_{{\bm{x}}_{k}} \log q({\bm{x}}^{k})\right]dk$ (9) $$

where $f(k)=\frac{d\log\alpha_{k}}{dk}$ and $g^{2}(k)=\frac{d\sigma_{k}^{2}}{dk}-2\frac{d\log\alpha_{k}}{dk}\sigma_{k}^{2}$. The unknown score $\nabla_{{\bm{x}}_{k}}\log q({\bm{x}}^{k})$ could be estimated as follows:

$$ $s({\bm{x}}^{k})=\nabla_{{\bm{x}}_{k}}\log q({\bm{x}}^{k})=-\frac{\epsilon^{*}( {\bm{x}}^{k},k)}{\sigma_{k}}\approx-\frac{\epsilon_{\theta}({\bm{x}}^{k},k)}{ \sigma_{k}},$ $$

where $\epsilon^{*}({\bm{x}}^{k},k)$ is the true noise added at time $k$, and we let $s_{\theta}({\bm{x}}^{k})=-\frac{\epsilon_{\theta}({\bm{x}}^{k},k)}{\sigma_{k}}$.

Wang et al. (); Chi et al. () extend diffusion models as expressive and powerful policies for offline RL and robotics. In robotics, a set of past observation images ${\mathbf{O}}$ is used as input to the policy. An action chunk ${\mathbf{A}}$, which consists of a sequence of consecutive actions, forms the output of the policy. ResNet (He et al., ) based vision encoders are commonly utilized to encode multiple camera observation images into observation features. Diffusion policy is represented as a conditional diffusion-based action prediction model,

$$ $\pi_{\theta}({\mathbf{A}}^{0}_{t}|{\mathbf{O}}_{t}):=\int\cdots\int{\mathcal{N }}({\mathbf{A}}^{K}_{t};\bm{0},{\bm{I}})\prod_{k=K}^{k=1}p_{\theta}({\mathbf{A }}^{k-1}_{t}|{\mathbf{A}}^{k}_{t},{\mathbf{O}}_{t})d{\mathbf{A}}_{t}^{K}\cdots d {\mathbf{A}}_{t}^{1},$ $$

where ${\mathbf{O}}_{t}$ contains the current and a few previous vision observation features at timestep $t$, and $p_{\theta}$ could be represented by $\epsilon_{\theta}$ as shown in DDPM (Ho et al., ). The explicit form of $\pi_{\theta}({\mathbf{A}}^{0}_{t}|{\mathbf{O}}_{t})$ is often impractical due to the complexity of integrating actions from ${\mathbf{A}}_{t}^{K}$ to ${\mathbf{A}}_{t}^{1}$. However, we can obtain an action chunk prediction ${\mathbf{A}}_{t}^{0}$ by iteratively solving Equation 9 from $K$ to 0 0.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Section 2.2 Line 203, why can’t you obtain the score for Equation 1? Why do you have to learn an auxiliary diffusion network? This feels unnecessary and computationally burdensome, if you are training three separate identical diffusion models (since the generator and action generator are initialized from the pretrained DP). 
2. What is the input and output of the generator? The generator is defined as an implicit policy and a one-step policy. Since the generator is initialized from Diffusion Policy parameters, is the time embedding (e.g. noise timestep 0, … 100) simply set always to 1? Or 100? Is the output just the predicted noise? 
3. Why not learn an explicit policy and use the SDS loss?
4. How would distilling Diffusion Policy into a smaller DP model perform? For instance, reduce the number of parameters for DP and train it to match the pretrained DP outputs. DDIM with 10 steps should be faster due to the smaller size of the smaller DP model. 
5. What is a deterministic action generator? Is z simply set to 0? 
6. Is Diffusion Policy natively 1Hz? Even large VLAs like OpenVLA or Octo can run at 5Hz… Where exactly does this number come from?
soundness: 3
strengths: The paper is relatively straightforward. The contribution presented in this paper is limited, because the tasks presented don’t necessarily require such dynamic control and fast inference. However, the results seem reasonable, and the distillation approach is different from Consistency Policy.

OneDP consistently outperforms Consistency Policy. It matches DDPM / EDM, for the most part.
summary: How can we distill diffusion policies into fast, one-step action generators? The authors propose One-Step Diffusion Policy (OneDP), a distillation-based approach to improve inference time of diffusion policies for robotics tasks from 1.5Hz to 62Hz. Diffusion policies have shown impressive results in imitation, and this work improves imitation learning by allowing for more responsive control due to fast inference. In order to distill Diffusion Policy, the authors opt for an SDS-based approach rather than one based on Consistency Models. The authors learn a generator (initialized from the pretrained imitation policy), which is updated via an SDS loss. For stochastic One-DP, an addition network is learned to approximate the score.
weaknesses: 1. The contribution is limited. Although faster inference is a worthy contribution for imitation learning, none of the tasks, neither sim nor real, necessitate faster inference. If existing imitation learning algorithms can solve these tasks with their existing control frequency, what necessitates faster inference? What would be convincing is if there were tasks that genuinely required responsive, closed-loop control.
2, Figure 2 is unclear. In part a, it is unclear what the action generator is optimizing exactly. Part b seems to make more sense, although it is not immediately clear from the figure what the authors are trying to convey. The paper could benefit from a clearer diagram on both ends.
3, The input and output of the generator and action generator are unclear. 
4. In the introduction, it is unclear the difference between the generator and action generator. Which one is actually being used as the policy?
5. Typo Line 212: “prertrained” should be “The parameters of the pretrained diffusion policy are fixed”

The authors should address the questions and weaknesses.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Questions

1) The clarity of the Method section could be enhanced by using consistent terminology for the auxiliary diffusion network in Lines 204 and 222. Additionally, consider removing some unnecessary subscripts in equations such as (5) and (8). Relocate expressions like $A_{\theta} = G_{\theta}(z, O), \quad z \sim N(0, I)$ to a "where..." explanation following the equation if needed.

2) About  “OneDP-S even slightly outperforms the pre-trained DP”.

    2.1 In Lines 345-347, the authors claim that “OneDP-S even slightly outperforms the pre-trained DP” and provide an explanation. However, this slight improvement may be due to stochastic effects caused by limited training seeds, as indicated by results in Square-mh, Square-ph, and Transport-ph in Table 1. Will this empirical ranking hold if the environment initialization state is expanded (both in the numbers and range)? 
    
    2.2 Additionally, the authors' explanation requires further discussion. If the improvement stems from the pre-trained diffusion model (teacher) accumulating more errors, this suggests the teacher is not overfitted to the training set. Then, how could the student policy outperform the teacher? Including results on the training scenarios may offer further insights.

3) There is a lack of empirical results on inference efficiency. The authors should include comprehensive comparisons of inference efficiency between OneDP, the original diffusion model (DDPM), and other distillation methods like DDIM, CP, and Progressive Distillation (Salimans & H, 2022), using simulation experiments (if real-world ones are expensive).
soundness: 3
strengths: Strengths


1) High inference time is a major limitation of diffusion policies. The proposed OneDP demonstrates less inference time through distillation.

2) OneDP introduces a weighted score estimation method from Diffusion-GAN to address the intractable KL divergence during the diffusion process.

3) The authors integrate the proposed distillation algorithm with two scheduling methods for discrete and continuous denoising time spaces.
summary: Summary:

This paper proposes a One-step Diffusion Policy (OneDP) that distills a pretrained diffusion policy into a single-step action generator by minimizing the KL divergence along the reverse chain. 
Experiments on the robomimic benchmark and real-world tasks show that OneDP reduces inference time while maintaining state-of-the-art success rates.
weaknesses: Weaknesses

1) The contribution of this method is unclear, as score distillation sampling is well-explored in prior text-to-3D research. The paper does not discuss the differences between VSD and OneDP, nor does it consider the temporal control characteristic in OneDP.

2) Training an auxiliary diffusion network could increase computational costs and optimization complexity. Could the author provide further details on the computational costs and optimization complexity introduced by the generator score network?

3) Distilling to a deterministic policy may diminish the stochastic advantages of diffusion policies.

4) The claim that the student policy slightly outperforms the teacher policy is interesting. However, it requires further discussion and empirical validation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. A reverse KL is employed to align the distributions of the action generator and pre-trained DP, which is known to induce mode-collapse problems. How does the reverse KL affect the multi-modal property of DP? 
2. I am confused about the fourth bar chart in Figure 1. Since OneDP requires first pre-training a DP, it seems confusing that the training cost of OneDP can be lower than DP itself; in other words, only considering the distillation procedure in OneDP can be an unfair comparison with DP in terms of training cost. Can you explain how you are measuring and comparing the training costs between OneDP and DP?
soundness: 3
strengths: 1. This work tackles a popular yet significant problem concerning accelerating inference in diffusion policy.
2. The authors conduct extensive experiments on visuomotor tasks in both simulation and real world.
3. The idea is straight-forward and the paper is well-written and easy to follow.
summary: This work proposes an inference-efficient diffusion policy by distilling a pre-trained DP into an action generator with one-step inference. It greatly improves the practicality of DP methods in real-world tasks, especially for responsive tasks.
weaknesses: 1. Consistency Policy (CP) [1] in the real world is missing in the experiment part, which can be a representative baseline considering accelerating DP. An additional experiment can be added to compare OneDP with CP in terms of inference speed and success rate.
2. More experiments on responsive or dynamic tasks can be added in the experiment part since potential application on responsive or dynamic tasks is a contribution of this work. A simple example can be randomly resetting the object pose sometime during the task execution.

[1] Prasad A, Lin K, Wu J, et al. Consistency policy: Accelerated visuomotor policies via consistency distillation[J]. arXiv preprint arXiv:2405.07503, 2024.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What are the limitations of OneDP?  (like ability on modeling mutimodality, capacity, etc.)
- Can the authors elaborates on why stochastic policies perform better in complex environments compared to deterministic ones? The stochastic policies seem to outperform deterministic ones in all the tasks.
soundness: 3
strengths: - The proposed approach is straightforward and easy to follow.
- The method is evaluated on a wide range of tasks in both simulation and real-world.
- The paper is well written and easy to follow.
summary: The paper proposed One-Step Diffusion Policy (OneDP) to reduce inference time through diffusion distillation. The method outperforms diffusion policy in both simulated and real-world environments.
weaknesses: - The method of distillation follows the methodologies in image generation. The novelty is limited.
- Only one baseline (DDIM in Tab. 1 and CP in Tab. 2) is compared. There are many methods speeding up diffusion models that can be compared.

Meta-review: The paper proposes a One-Step Diffusion Policy (OneDP), a method that distills a pretrained diffusion policy (DP) into a fast, one-step action generator, significantly improving the inference time of diffusion policies. OneDP achieves inference speeds of 62Hz, making it suitable for real-world robotics tasks. The method uses score-based distillation and introduces an auxiliary diffusion network for training. The experimental results show that OneDP improves inference speed while maintaining competitive performance in various tasks, including both simulation and real-world settings.

Reasons to accept
- The paper is well-written and easy to follow.
- OneDP successfully accelerates diffusion policy inference, addressing a significant limitation of traditional diffusion models in real-time applications.
- The method is evaluated on a variety of tasks, both simulated and real-world, showcasing its practical potential.
- The method shows promising results, outperforming previous methods like Consistency Policy (CP) and DDIM in some cases.

Reasons to reject
- The distillation method is not particularly novel, as similar approaches have been used in other domains like image generation.
- The tasks evaluated do not convincingly require such fast inference speeds. The paper does not clearly explain why faster inference is necessary, especially when existing methods can already handle the tasks at lower frequencies.
- The experiments compare OneDP to only a few methods, and additional comparisons to other distillation techniques (e.g., Progressive Distillation) would strengthen the evaluation.
- The introduction of auxiliary diffusion networks and the need for additional networks in training raise questions about computational costs and optimization complexity, which are not fully addressed.
- (Minor) The revised paper violates the page limit (10 pages).

While this paper studies a promising research direction, i.e., accelerating the inference time of Diffusion Policy for controlling robots in real-world scenarios, and presents an interesting approach, the paper fails to convince the reviewers that the proposed method applies to high-dynamic tasks requiring higher inference speed. Consequently, I recommend rejecting the paper.

Decision: Reject