.

Diffusion Policy: Visuomotor Policy
Learning via Action Diffusion
Cheng Chi∗ 1 , Zhenjia Xu∗ 1 , Siyuan Feng2 , Eric Cousineau2 , Yilun Du3 , Benjamin Burchfiel2 ,Russ Tedrake 2,3 , Shuran Song1,4

Abstract
This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot’s visuomotor
policy as a conditional denoising diffusion process. We benchmark Diffusion Policy across 15 different tasks from 4
different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning
methods with an average improvement of 46.9%. Diffusion Policy learns the gradient of the action-distribution scorefunction and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevindynamics steps. We find that the diffusion formulation yields powerful advantages when used for robot policies, including
gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting
impressive training stability. To fully unlock the potential of diffusion models for visuomotor policy learning on physicalrobots, this paper presents a set of key technical contributions including the incorporation of receding horizon control,
visual conditioning, and the time-series diffusion transformer. We hope this work will help motivate a new generation ofpolicy learning techniques that are able to leverage the powerful generative modeling capabilities of diffusion models.
Code, data, and training details is available diffusion-policy.cs.columbia.edu
Keywords
Imitation learning, visuomotor policy, manipulation

0

ActionRepresentation

Scalar (Regression)

Implicit Policy

EnergyGradient Field

1

Diffusion Policy

Explicit Policy

Mixture of Gaussians

iter

Categorical(a) Explicit Policy(b) Implicit Policy(c) Diffusion Policy

Figure 1. Policy Representations. a) Explicit policy with different types of action representations. b) Implicit policy learns anenergy function conditioned on both action and observation and optimizes for actions that minimize the energy landscape c)
Diffusion policy refines noise into actions via a learned gradient field. This formulation provides stable training, allows the learned
policy to accurately model multimodal action distributions, and accommodates high-dimensional action sequences.

1

Introduction

Policy learning from demonstration, in its simplest form, canbe formulated as the supervised regression task of learning to
map observations to actions. In practice, however, the uniquenature of predicting robot actions — such as the existenceof multimodal distributions, sequential correlation, and therequirement of high precision — makes this task distinct andchallenging compared to other supervised learning problems.
Prior work attempts to address this challenge by
exploring different action representations (Fig 1 a) – using
mixtures of Gaussians Mandlekar et al. (2021), categoricalrepresentations of quantized actions Shafiullah et al. (2022),
or by switching the the policy representation (Fig 1 b) – fromexplicit to implicit to better capture multi-modal distributions
Florence et al. (2021); Wu et al. (2020).
In this work, we seek to address this challenge byintroducing a new form of robot visuomotor policy thatgenerates behavior via a “conditional denoising diffusion
process Ho et al. (2020) on robot action space”, Diffusion
Policy. In this formulation, instead of directly outputting
an action, the policy infers the action-score gradient,conditioned on visual observations, for K denoising
iterations (Fig. 1 c). This formulation allows robot policies
to inherit several key properties from diffusion models –significantly improving performance.
• Expressing multimodal action distributions. Bylearning the gradient of the action score function
Song and Ermon (2019) and performing Stochastic
Langevin Dynamics sampling on this gradient field,
Diffusion policy can express arbitrary normalizable
distributions Neal et al. (2011), which includes multimodal action distributions, a well-known challenge
for policy learning.

2• High-dimensional output space. As demonstrated bytheir impressive image generation results, diffusion
models have shown excellent scalability to highdimension output spaces. This property allows thepolicy to jointly infer a sequence of future actions
instead of single-step actions, which is critical forencouraging temporal action consistency and avoiding
myopic planning.

providing strong evidence of the effectiveness of Diffusion
Policy. We also provide detailed analysis to carefully
examine the characteristics of the proposed algorithm andthe impacts of the key design decisions.
This work is an extended version of the conference paper
Chi et al. (2023). We expand the content of this paper in thefollowing ways:• Include additional discussions on the connections
between diffusion policy and control theory, as wellas its implication to representation learning Sec. 4.5.

• Stable training. Training energy-based policies often
requires negative sampling to estimate an intractablenormalization constant, which is known to cause
training instability Du et al. (2020); Florence et al.
(2021). Diffusion Policy bypasses this requirement
by learning the gradient of the energy function andthereby achieves stable training while maintaining
distributional expressivity.

• Include additional ablation studies on alternative
network architecture design and different pretraining
and finetuning paradigms, Sec. 5.4.
• Extend the real-world experimental results with threebimanual manipulation tasks including Egg Beater,
Mat Unrolling, and Shirt Folding in Sec. 7.

Our primary contribution is to bring the above
advantages to the field of robotics and demonstrate theireffectiveness on complex real-world robot manipulation
tasks. To successfully employ diffusion models forvisuomotor policy learning, we present the following
technical contributions that enhance the performance ofDiffusion Policy and unlock its full potential on physicalrobots:

The code, data, and training details are publicly available
for reproducing our results diffusion-policy.cs.
columbia.edu.

• Closed-loop action sequences. We combine thepolicy’s capability to predict high-dimensional action
sequences with receding-horizon control to achieverobust execution. This design allows the policy tocontinuously re-plan its action in a closed-loop manner
while maintaining temporal action consistency –achieving a balance between long-horizon planningand responsiveness.

We formulate visuomotor robot policies as Denoising
Diffusion Probabilistic Models (DDPMs) Ho et al. (2020).
Crucially, Diffusion policies are able to express complex
multimodal action distributions and possess stable training
behavior – requiring little task-specific hyperparametertuning. The following sections describe DDPMs in moredetail and explain how they may be adapted to represent
visuomotor policies.

• Visual conditioning. We introduce a visionconditioned diffusion policy, where the visualobservations are treated as conditioning instead of apart of the joint data distribution. In this formulation,
the policy extracts the visual representation once
regardless of the denoising iterations, which
drastically reduces the computation and enables
real-time action inference.

2.1• Time-series diffusion transformer. We propose
a new transformer-based diffusion network thatminimizes the over-smoothing effects of typical
CNN-based models and achieves state-of-the-art
performance on tasks that require high-frequency
action changes and velocity control.
We systematically evaluate Diffusion Policy across 15
tasks from 4 different benchmarks Florence et al. (2021);
Gupta et al. (2019); Mandlekar et al. (2021); Shafiullah
et al. (2022) under the behavior cloning formulation.
The evaluation includes both simulated and real-world
environments, 2DoF to 6DoF actions, single- and multi-task
benchmarks, and fully- and under-actuated systems, withrigid and fluid objects, using demonstration data collectedby single and multiple users.
Empirically, we find consistent performance boost acrossall benchmarks with an average improvement of 46.9%,2

Diffusion Policy Formulation

Denoising Diffusion Probabilistic Models

DDPMs are a class of generative model where the output
generation is modeled as a denoising process, often called
Stochastic Langevin Dynamics Welling and Teh (2011).
Starting from xK sampled from Gaussian noise, the DDPM
performs K iterations of denoising to produce a series
of intermediate actions with decreasing levels of noise,
xk , xk−1 ...x0 , until a desired noise-free output x0 is formed.
The process follows the equationxk−1 = α(xk − γεθ (xk , k) + N 0, σ 2 I ),(1)
where εθ is the noise prediction network with parameters θthat will be optimized through learning and N 0, σ 2 I is
Gaussian noise added at each iteration.
The above equation 1 may also be interpreted as a single
noisy gradient descent step:x′ = x − γ∇E(x),(2)

where the noise prediction network εθ (x, k) effectively
predicts the gradient field ∇E(x), and γ is the learning rate.
The choice of α, γ, σ as functions of iteration step k, also
called noise schedule, can be interpreted as learning ratescheduling in gradient decent process. An α slightly smaller
than 1 has been shown to improve stability Ho et al. (2020).
Details about noise schedule will be discussed in Sec. 3.3.

Diffusion Policy

3

Input: Image Observation Sequenceot-2 ot-1 ot∇E(At)b

Conv1D

×K

Conv1Da⋅x+b
Conv1Dx: Action Emb
Conv1D
conditioning

FiLM

At

a) Diffusion Policy General Formulationb) CNN-based

A Emb

b

Cross AttentionaAction Emb

Obs Emb

Output: Action Sequence

at+4

A0t

Cross Attention

k

A3t

k

AKt

At+4

Conv1Da⋅x+bx: Action Emb

Linearat at+1 at+2 at+3aObservation Ot

Diffusion Policy

Action Sequence At
Prediction Horizon Tp∇E(At)…

…

ot+4
Observation Ot

Ot

Ot+4
LinearRobot
Pose

Observation Ot

×K

Action Emb

Atc) Transformer-based

Figure 2. Diffusion Policy Overview a) General formulation. At time step t , the policy takes the latest To steps of observation
data Ot as input and outputs Ta steps of actions At . b) In the CNN-based Diffusion Policy, FiLM (Feature-wise Linear Modulation)
Perez et al. (2018) conditioning of the observation feature Ot is applied to every convolution layer, channel-wise. Starting from AtK
drawn from Gaussian noise, the output of noise-prediction network εθ is subtracted, repeating K times to get At0 , the denoised
action sequence. c) In the Transformer-based Vaswani et al. (2017) Diffusion Policy, the embedding of observation Ot is passed
into a multi-head cross-attention layer of each transformer decoder block. Each action embedding is constrained to only attend toitself and previous action embeddings (causal attention) using the attention mask illustrated.

2.2

DDPM Training

The training process starts by randomly drawing unmodified
examples, x0 , from the dataset. For each sample, we
randomly select a denoising iteration k and then sample arandom noise ε k with appropriate variance for iteration k.
The noise prediction network is asked to predict the noisefrom the data sample with noise added.
L = MSE(ε k , εθ (x0 + ε k , k))(3)

As shown in Ho et al. (2020), minimizing the loss function
in Eq 3 also minimizes the variational lower bound of the
KL-divergence between the data distribution p(x0 ) and the
distribution of samples drawn from the DDPM q(x0 ) using
Eq 1.

action sequence prediction when the prediction horizon isgreater than the action horizon. Specifically, warm starting isdone by initializing the first few actions with the unexecuted
actions from the previous inference step.
Visual observation conditioning: We use a DDPM toapproximate the conditional distribution p(At |Ot ) instead of
the joint distribution p(At , Ot ) used in Janner et al. (2022a)
for planning. This formulation allows the model to predict
actions conditioned on observations without the cost ofinferring future states, speeding up the diffusion process andimproving the accuracy of generated actions. To capture theconditional distribution p(At |Ot ), we modify Eq 1 to:Atk−1 = α(Atk − γεθ (Ot , Atk , k) + N 0, σ 2 I )(4)The training loss is modified from Eq 3 to:

2.3

Diffusion for Visuomotor Policy Learning

While DDPMs are typically used for image generation (xis an image), we use a DDPM to learn robot visuomotor
policies. This requires two major modifications in theformulation: 1. changing the output x to represent robot
actions. 2. making the denoising processes conditioned on
input observation Ot . The following paragraphs discuss eachof the modifications, and Fig. 2 shows an overview.
Closed-loop action-sequence prediction: An effective
action formulation should encourage temporal consistency
and smoothness in long-horizon planning while allowing
prompt reactions to unexpected observations. To accomplish
this goal, we commit to the action-sequence prediction
produced by a diffusion model for a fixed duration before
replanning. Concretely, at time step t the policy takes thelatest To steps of observation data Ot as input and predicts
Tp steps of actions, of which Ta steps of actions are executedon the robot without re-planning. Here, we define To as
the observation horizon, Tp as the action prediction horizon
and Ta as the action execution horizon. This encourages
temporal action consistency while remaining responsive.
More details about the effects of Ta are discussed in Sec. 4.3.
Our formulation also allows receding horizon control Mayne
and Michalska (1988) to futher improve action smoothness
by warm-starting the next inference setup with previous

L = MSE(ε k , εθ (Ot , At0 + ε k , k))(5)

The exclusion of observation features Ot from the outputof the denoising process significantly improves inference
speed and better accommodates real-time control. It also
helps to make end-to-end training of the vision encoder
feasible. Details about the visual encoder are described inSec. 3.2.

3

Key Design Decisions

In this section, we describe key design decisions for
Diffusion Policy as well as its concrete implementation ofεθ with neural network architectures.

3.1

Network Architecture Options

The first design decision is the choice of neural networkarchitectures for εθ . In this work, we examine two common
network architecture types, convolutional neural networks(CNNs) Ronneberger et al. (2015) and Transformers
Vaswani et al. (2017), and compare their performanceand training characteristics. Note that the choice of noise
prediction network εθ is independent of visual encoders,which will be described in Sec. 3.2.

4

CNN-based Diffusion Policy We adopt the 1D temporal
CNN from Janner et al. (2022b) with a few modifications:First, we only model the conditional distribution p(At |Ot )by conditioning the action generation process on observation
features Ot with Feature-wise Linear Modulation (FiLM)
Perez et al. (2018) as well as denoising iteration k, shownin Fig 2 (b). Second, we only predict the action trajectory
instead of the concatenated observation action trajectory.
Third, we removed inpainting-based goal state conditioning
due to incompatibility with our framework utilizing areceding prediction horizon. However, goal conditioning is
still possible with the same FiLM conditioning method used
for observations.
In practice, we found the CNN-based backbone to work
well on most tasks out of the box without the need for much
hyperparameter tuning. However, it performs poorly when
the desired action sequence changes quickly and sharply
through time (such as velocity command action space), likely
due to the inductive bias of temporal convolutions to preferlow-frequency signals Tancik et al. (2020).
Time-series diffusion transformer To reduce the oversmoothing effect in CNN models Tancik et al. (2020), we
introduce a novel transformer-based DDPM which adopts
the transformer architecture from minGPT Shafiullah et al.
(2022) for action prediction. Actions with noise Atk arepassed in as input tokens for the transformer decoder
blocks, with the sinusoidal embedding for diffusion iteration
k prepended as the first token. The observation Ot istransformed into observation embedding sequence by ashared MLP, which is then passed into the transformer
decoder stack as input features. The “gradient” εθ (Ot , At k , k)is predicted by each corresponding output token of thedecoder stack.
In our state-based experiments, most of the bestperforming policies are achieved with the transformer
backbone, especially when the task complexity and rate ofaction change are high. However, we found the transformer
to be more sensitive to hyperparameters. The difficulty oftransformer training Liu et al. (2020) is not unique toDiffusion Policy and could potentially be resolved in the
future with improved transformer training techniques orincreased data scale.
Recommendations. In general, we recommend startingwith the CNN-based diffusion policy implementation as thefirst attempt at a new task. If performance is low due to taskcomplexity or high-rate action changes, then the Time-series
Diffusion Transformer formulation can be used to potentially
improve performance at the cost of additional tuning.

3.2

Visual Encoder

The visual encoder maps the raw image sequence intoa latent embedding Ot and is trained end-to-end withthe diffusion policy. Different camera views use separate
encoders, and images in each timestep are encoded
independently and then concatenated to form Ot . We used astandard ResNet-18 (without pretraining) as the encoder withthe following modifications: 1) Replace the global average
pooling with a spatial softmax pooling to maintain spatial
information for consistency with Mandlekar et al. (2021)
baselines . 2) Replace BatchNorm with GroupNorm Wu andHe (2018) for stable training. This is important when thenormalization layer is used in conjunction with Exponential
Moving Average He et al. (2020) (commonly used in
DDPMs), as the standard implementation of Exponential
Moving Average interferes with BatchNorm’s calculation ofmoving averages and standard deviations.

3.3

Noise Schedule

The noise schedule, defined by σ , α, γ and the additive
Gaussian Noise ε k as functions of k, has been actively
studied Ho et al. (2020); Nichol and Dhariwal (2021).
The underlying noise schedule controls the extent towhich diffusion policy captures high and low-frequency
characteristics of action signals. In our control tasks, we
empirically found that the Square Cosine Schedule proposed
in iDDPM Nichol and Dhariwal (2021) works best for ourtasks.

3.4

Accelerating Inference for Real-timeControl

We use the diffusion process as the policy for robots; hence,
it is critical to have a fast inference speed for closed-loop
real-time control. The Denoising Diffusion Implicit Models(DDIM) approach Song et al. (2021) decouples the numberof denoising iterations in training and inference, thereby
allowing the algorithm to use fewer iterations for inference
to speed up the process. In our real-world experiments,using DDIM with 100 training iterations and 10 inference
iterations enables 0.1s inference latency on a Nvidia 3080
GPU.

4

Intriguing Properties of Diffusion Policy

In this section, we provide some insights and intuitions about
diffusion policy and its advantages over other forms of policy
representations.

4.1

Model Multi-Modal Action Distributions

The challenge of modeling multi-modal distribution inhuman demonstrations has been widely discussed inbehavior cloning literature Florence et al. (2021); Shafiullah
et al. (2022); Mandlekar et al. (2021). Diffusion Policy’s
ability to express multimodal distributions naturally andprecisely is one of its key advantages.
Intuitively, multi-modality in action generation for
diffusion policy arises from two sources – an underlying
stochastic sampling procedure and a stochastic initialization.
In Stochastic Langevin Dynamics, an initial sample AtK is
drawn from standard Gaussian at the beginning of eachsampling process, which helps specify different possible
convergence basins for the final action prediction At0 . Thisaction is then further stochastically optimized, with added
Gaussian perturbations across a large number of iterations,which enables individual action samples to converge andmove between different multi-modal action basins. Fig. 3,
shows an example of the Diffusion Policy’s multimodal
behavior in a planar pushing task (Push T, introduced below)
without explicit demonstration for the tested scenario.

Diffusion Policy

LSTM-GMM

BET

IBC

Figure 3. Multimodal behavior. At the given state, theend-effector (blue) can either go left or right to push the block.
Diffusion Policy learns both modes and commits to only one
mode within each rollout. In contrast, both LSTM-GMM
Mandlekar et al. (2021) and IBC Florence et al. (2021) arebiased toward one mode, while BET Shafiullah et al. (2022) fails
to commit to a single mode due to its lack of temporal action
consistency. Actions generated by rolling out 40 steps for the
best-performing checkpoint.

4.2

Synergy with Position Control

We find that Diffusion Policy with a position-control
action space consistently outperforms Diffusion Policy withvelocity control, as shown in Fig 4. This surprising result
stands in contrast to the majority of recent behavior cloning
work that generally relies on velocity control Mandlekar
et al. (2021); Shafiullah et al. (2022); Zhang et al. (2018);
Florence et al. (2019); Mandlekar et al. (2020b,a). We
speculate that there are two primary reasons for thisdiscrepancy: First, action multimodality is more pronounced
in position-control mode than it is when using velocity
control. Because Diffusion Policy better expresses action
multimodality than existing approaches, we speculate that
it is inherently less affected by this drawback than existing
methods. Furthermore, position control suffers less thanvelocity control from compounding error effects and is thus
more suitable for action-sequence prediction (as discussed in
the following section). As a result, Diffusion Policy is bothless affected by the primary drawbacks of position control
and is better able to exploit position control’s advantages.

6IPEXMZI7YGGIWW6EXI'LERKI



:IPSGMX]ZW4SWMXMSREP'SRXVSP
0781+11&)8
(MJJYWMSR4SPMG]'(MJJYWMSR4SPMG]8


7UYEVI

/MXGLIRT

Figure 4. Velocity v.s. Position Control. The performancedifference when switching from velocity to position control.
While both BCRNN and BET performance decrease, Diffusion
Policy is able to leverage the advantage of position and improve
its performance.

4.3

Benefits of Action-Sequence Prediction

Sequence prediction is often avoided in most policy learning
methods due to the difficulties in effectively sampling fromhigh-dimensional output spaces. For example, IBC would
struggle in effectively sampling high-dimensional action
space with a non-smooth energy landscape. Similarly, BCRNN and BET would have difficulty specifying the numberof modes that exist in the action distribution (needed for
GMM or k-means steps).
In contrast, DDPM scales well with output dimensions
without sacrificing the expressiveness of the model, as
demonstrated in many image generation applications.
Leveraging this capability, Diffusion Policy represents actionin the form of a high-dimensional action sequence, which
naturally addresses the following issues:• Temporal action consistency: Take Fig 3 as an example.
To push the T block into the target from the bottom, thepolicy can go around the T block from either left or right.
However, suppose each action in the sequence is predicted
as independent multimodal distributions (as done in BCRNN and BET). In that case, consecutive actions could be
drawn from different modes, resulting in jittery actions thatalternate between the two valid trajectories.
• Robustness to idle actions: Idle actions occur whena demonstration is paused and results in sequences ofidentical positional actions or near-zero velocity actions.
It is common during teleoperation and is sometimes
required for tasks like liquid pouring. However, singlestep policies can easily overfit to this pausing behavior.
For example, BC-RNN and IBC often get stuck in realworld experiments when the idle actions are not explicitly
removed from training.

6IPEXMZI4IVJ'LERKI

Diffusion Policy

5







%GXMSR,SVM^SR

0EXIRG]6SFYWXRIWW               
%GXMSR,SVM^SRWXITW
0EXIRG]WXITW
4YWL8
7UYEVI

Figure 5. Diffusion Policy Ablation Study. Change(difference) in success rate relative to the maximum for eachtask is shown on the Y-axis. Left: trade-off between temporalconsistency and responsiveness when selecting the action
horizon. Right: Diffusion Policy with position control is robust
against latency. Latency is defined as the number of steps
between the last frame of observations to the first action thatcan be executed.

4.4

Training Stability

While IBC possess similar advantages as diffusion polices(such as representing action multimodality) in theory, it
is difficult to reliably achieve diffusion policy level of
performance in practice, due to IBC’s inherent training
instability Ta et al. (2022). Fig 6 shows training error
spikes and unstable evaluation performance throughout thetraining process, making hyperparameter turning critical andcheckpoint selection difficult. As a result, Florence et al.
(2021) evaluate every checkpoint and report results for the
best-performing checkpoint. In a real-world setting, thisworkflow necessitates the evaluation of many policies on
hardware to select a final policy. Here, we discuss why
Diffusion Policy appears significantly more stable to train.

6

An implicit policy represents the action distribution using
an Energy-Based Model (EBM):
e−Eθ (o,a)
Z(o, θ )pθ (a|o) =(6)

where Z(o, θ ) is an intractable normalization constant (withrespect to a).
To train the EBM for implicit policy, an InfoNCE-styleloss function is used, which equates to the negative loglikelihood of Eq 6:
Lin f oNCE = − log(e−Eθ (o,a))
jNneg −E (o,ee−Eθ (o,a) + ∑ j=1
e θ a)

Now imagine we obtain demonstrations (rollouts) from a
linear feedback policy: at = −Kst . This policy could beobtained, for instance, by solving a linear optimal control
problem like the Linear Quadratic Regulator. Imitating thispolicy does not need the modeling power of diffusion, but
as a sanity check, we can see that Diffusion Policy does theright thing.
In particular, when the prediction horizon is one time
step, Tp = 1, it can be seen that the optimal denoiser which
minimizes
L = MSE(ε k , εθ (st , −Kst + ε k , k))(7)
is given byεθ (s, a, k) = a + Ks.

N

neg
where a set of negative samples {ea j } j=1
are used to estimate
the intractable normalization constant Z(o, θ ). In practice,the inaccuracy of negative sampling is known to cause
training instability for EBMs Du et al. (2020); Ta et al.
(2022).
Diffusion Policy and DDPMs sidestep the issue of
estimating Z(a, θ ) altogether by modeling the scorefunction Song and Ermon (2019) of the same action
distribution in Eq 6:∇a log p(a|o) = −∇a Eθ (a, o) − ∇a log Z(o, θ ) ≈ −εθ (a, o)
|
{z
}
=0

(8)
where the noise-prediction network εθ (a, o) is approximating the negative of the score function ∇a log p(a|o) Liu
et al. (2022), which is independent of the normalization
constant Z(o, θ ). As a result, neither the inference (Eq 4)
nor training (Eq 5) process of Diffusion Policy involves
evaluating Z(o, θ ), thus making Diffusion Policy training
more stable.







)TSGL

7MQ4YWL87XEXI
7YGGIWW6EXI8VEMR%GXMSR4VIH17)

6IEP4YWL8-QK



5

We systematically evaluate Diffusion Policy on 15 tasks
from 4 benchmarks Florence et al. (2021); Gupta et al.
(2019); Mandlekar et al. (2021); Shafiullah et al. (2022).
This evaluation suite includes both simulated and realenvironments, single and multiple task benchmarks, fully
actuated and under-actuated systems, and rigid andfluid objects. We found Diffusion Policy to consistently
outperform the prior state-of-the-art on all of the tested
benchmarks, with an average success-rate improvement of
46.9%. In the following sections, we provide an overview of
each task, our evaluation methodology on that task, and our
key takeaways.






(MJJYWMSR4SPMG]
)TSGL-&'

Connections to Control Theory

Diffusion Policy has a simple limiting behavior when thetasks are very simple; this potentially allows us to bring
to bear some rigorous understanding from control theory.
Consider the case where we have a linear dynamical system,in standard state-space form, that we wish to control:st+1 = Ast + Bat + wt ,Furthermore, at inference time, the DDIM sampling will
converge to the global minima at a = −Ks.
Action-sequence prediction (Tp > 1) follows naturally. In
order to predict at+t ′ as a function of st , the optimal denoiser′
will produce at+t ′ = −K(A − BK)t st ; all terms involving wt
are zero in expectation. This shows that in order to perfectlypredict future actions that depend on the state, the learnermust implicitly learn about the dynamics, represented hereby (A − BK). But interestingly, it need only learn the taskrelevant dynamics Subramanian and Mahajan (2019); Zhang
et al. (2020); in the linear dynamics example here, if K has
a column that is all zeros, then leftmost multiplication by K
means that the associated state dynamics are not relevant to
the task and need not be learned.
This simple example shows that action-sequence prediction has interesting consequences if we consider behavior
cloning as a tool for representation learning. If either theplant or the policy is nonlinear, then the same lessons hold,but the notion of task-relevance can become much more rich,and even Gaussian noise can lead to rich multimodal state(and therefore action) distributions.



Figure 6. Training Stability. Left: IBC fails to infer training
actions with increasing accuracy despite smoothly decreasing
training loss for energy function. Right: IBC’s evaluation successrate oscillates, making checkpoint selection difficult (evaluatedusing policy rollouts in simulation).

4.5

(9)

wt ∼ N (0, Σw ).

5.1

Evaluation

Simulation Environments and datasets

Robomimic Mandlekar et al. (2021) is a large-scalerobotic manipulation benchmark designed to study imitation
learning and offline RL. The benchmark consists of 5 tasks
with a proficient human (PH) teleoperated demonstration
dataset for each and mixed proficient/non-proficient human(MH) demonstration datasets for 4 of the tasks (9 variants
in total). For each variant, we report results for both state-Diffusion Policy

7

Lift
ph

Can
mh

ph

mh

ph

Squaremh

Transportphmh

ToolHang Push-T
phph

LSTM-GMM 1.00/0.96 1.00/0.93 1.00/0.91 1.00/0.81 0.95/0.73 0.86/0.59 0.76/0.47 0.62/0.20 0.67/0.31 0.67/0.61
IBC 0.79/0.41 0.15/0.02 0.00/0.00 0.01/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.90/0.84
BET 1.00/0.96 1.00/0.99 1.00/0.89 1.00/0.90 0.76/0.52 0.68/0.43 0.38/0.14 0.21/0.06 0.58/0.20 0.79/0.70
DiffusionPolicy-C 1.00/0.98 1.00/0.97 1.00/0.96 1.00/0.96 1.00/0.93 0.97/0.82 0.94/0.82 0.68/0.46 0.50/0.30 0.95/0.91
DiffusionPolicy-T 1.00/1.00 1.00/1.00 1.00/1.00 1.00/0.94 1.00/0.89 0.95/0.81 1.00/0.84 0.62/0.35 1.00/0.87 0.95/0.79
Table 1. Behavior Cloning Benchmark (State Policy) We present success rates with different checkpoint selection methods inthe format of (max performance) / (average of last 10 checkpoints), with each averaged across 3 training seeds and 50 different
environment initial conditions (150 in total). LSTM-GMM corresponds to BC-RNN in RoboMimicMandlekar et al. (2021), which we
reproduced and obtained slightly better results than the original paper. Our results show that Diffusion Policy significantly improves
state-of-the-art performance across the board.

Lift
ph

Can
mh

ph

mh

ph

Squaremh

Transportphmh

LSTM-GMM 1.00/0.96 1.00/0.95 1.00/0.88 0.98/0.90 0.82/0.59 0.64/0.38 0.88/0.62 0.44/0.24
IBC 0.94/0.73 0.39/0.05 0.08/0.01 0.00/0.00 0.03/0.00 0.00/0.00 0.00/0.00 0.00/0.00
DiffusionPolicy-C 1.00/1.00 1.00/1.00 1.00/0.97 1.00/0.96 0.98/0.92 0.98/0.84 1.00/0.93 0.89/0.69
DiffusionPolicy-T 1.00/1.00 1.00/0.99 1.00/0.98 1.00/0.98 1.00/0.90 0.94/0.80 0.98/0.81 0.73/0.50

ToolHang Push-Tphph0.68/0.49
0.00/0.00
0.95/0.73
0.76/0.47

0.69/0.54
0.75/0.64
0.91/0.84
0.78/0.66

Table 2. Behavior Cloning Benchmark (Visual Policy) Performance are reported in the same format as in Tab 1. LSTM-GMM
numbers were reproduced to get a complete evaluation in addition to the best checkpoint performance reported. Diffusion Policy
shows consistent performance improvement, especially for complex tasks like Transport and ToolHang.

Task# Rob # Obj ActD #PH #MH Steps Img? HiPrec
Simulation Benchmark

Lift
Can
SquareTransport
ToolHang

1
1
1
2
1

1
1
1
3
2

7
7
7
14
7

200
200
200
200
200

300
300
300
300
0

400
400
400
700
700

Yes
Yes
Yes
Yes
Yes

No
No
Yes
No
Yes

Push-T

1

1

2

200

0

300

Yes

Yes

BlockPush

1

2

2

0

0

350

No

No

Kitchen

1

7

9

656

0

280

No

No

Yes
Yes
Yes
Yes

Yes
No
No
No

Realworld Benchmark
Push-T
6DoF Pour
Peri Spread
Mug Flip

1
1
1
1

1
liquidliquid
1

2
6
6
7

136
90
90
250

0
0
0
0

600
600
600
600

Table 3. Tasks Summary. # Rob: number of robots, #Obj:
number of objects, ActD: action dimension, PH:
proficient-human demonstration, MH: multi-humandemonstration, Steps: max number of rollout steps, HiPrec:
whether the task has a high precision requirement. BlockPush
uses 1000 episodes of scripted demonstrations.

and image-based observations. Properties for each task aresummarized in Tab 3.
Push-T adapted from IBC Florence et al. (2021), requires
pushing a T-shaped block (gray) to a fixed target (red)with a circular end-effector (blue). Variation is added byrandom initial conditions for T block and end-effector. The

task requires exploiting complex and contact-rich object
dynamics to push the T block precisely, using point contacts.
There are two variants: one with RGB image observations
and another with 9 2D keypoints obtained from the groundtruth pose of the T block, both with proprioception for endeffector location.
Multimodal Block Pushing adapted from BET Shafiullah
et al. (2022), this task tests the policy’s ability to model
multimodal action distributions by pushing two blocks
into two squares in any order. The demonstration data is
generated by a scripted oracle with access to groundtruth
state info. This oracle randomly selects an initial block
to push and moves it to a randomly selected square. The
remaining block is then pushed into the remaining square.
This task contains long-horizon multimodality that can not
be modeled by a single function mapping from observationto action.
Franka Kitchen is a popular environment for evaluating
the ability of IL and Offline-RL methods to learn multiple
long-horizon tasks. Proposed in Relay Policy Learning
Gupta et al. (2019), the Franka Kitchen environment
contains 7 objects for interaction and comes with ahuman demonstration dataset of 656 demonstrations, eachcompleting 4 tasks in arbitrary order. The goal is to
execute as many demonstrated tasks as possible, regardlessof order, showcasing both short-horizon and long-horizon
multimodality.

8

LSTM-GMM
IBC
BET
DiffusionPolicy-C
DiffusionPolicy-T

BlockPush
p1
p2

p1

Kitchen
p2
p3

p4

0.03
0.01
0.96
0.36
0.99

1.00
0.99
0.99
1.00
1.00

0.90
0.87
0.93
1.00
0.99

0.34
0.24
0.44
0.99
0.96

0.01
0.00
0.71
0.11
0.94

0.74
0.61
0.71
1.00
0.99

Table 4. Multi-Stage Tasks (State Observation). For
PushBlock, px is the frequency of pushing x blocks into thetargets. For Kitchen, px is the frequency of interacting with x ormore objects (e.g. bottom burner). Diffusion Policy performs
better, especially for difficult metrics such as p2 for Block
Pushing and p4 for Kitchen, as demonstrated by our results.

5.2

Evaluation Methodology

We present the best-performing for each baseline
method on each benchmark from all possible sources –our reproduced result (LSTM-GMM) or original number
reported in the paper (BET, IBC). We report results from theaverage of the last 10 checkpoints (saved every 50 epochs)
across 3 training seeds and 50 environment initializations* (an average of 1500 experiments in total). The metric formost tasks is success rate, except for the Push-T task, whichuses target area coverage. In addition, we report the averageof best-performing checkpoints for robomimic and PushT tasks to be consistent with the evaluation methodologyof their respective original papers Mandlekar et al. (2021);
Florence et al. (2021). All state-based tasks are trained for4500 epochs, and image-based tasks for 3000 epochs. Eachmethod is evaluated with its best-performing action space:position control for Diffusion Policy and velocity control
for baselines (the effect of action space will be discussed
in detail in Sec 5.3). The results from these simulation
benchmarks are summarized in Table 1 and Table 2.

5.3

Key Findings

Diffusion Policy outperforms alternative methods on all tasks
and variants, with both state and vision observations, inour simulation benchmark study (Tabs 1, 2 and 4) with anaverage improvement of 46.9%. The following paragraphs
summarize the key takeaways.
Diffusion Policy can express short-horizon multimodality. We define short-horizon action multimodality as
multiple ways of achieving the same immediate goal, which
is prevalent in human demonstration data Mandlekar et al.
(2021). In Fig 3, we present a case study of this type of shorthorizon multimodality in the Push-T task. Diffusion Policy
learns to approach the contact point equally likely from leftor right, while LSTM-GMM Mandlekar et al. (2021) and
IBC Florence et al. (2021) exhibit bias toward one side and
BET Shafiullah et al. (2022) cannot commit to one mode.

Diffusion Policy can express long-horizon multimodality. Long-horizon multimodality is the completion of different sub-goals in inconsistent order. For example, the order ofpushing a particular block in the Block Push task or the orderof interacting with 7 possible objects in the Kitchen task arearbitrary. We find that Diffusion Policy copes well with thistype of multimodality; it outperforms baselines on both tasks
by a large margin: 32% improvement on Block Push’s p2
metric and 213% improvement on Kitchen’s p4 metric.
Diffusion Policy can better leverage position control.
Our ablation study (Fig. 4) shows that selecting position
control as the diffusion-policy action space significantly
outperformed velocity control. The baseline methods weevaluate, however, work best with velocity control (and thisis reflected in the literature where most existing work reports
using velocity-control action spaces Mandlekar et al. (2021);
Shafiullah et al. (2022); Zhang et al. (2018); Florence et al.
(2019); Mandlekar et al. (2020b,a)).
The tradeoff in action horizon. As discussed in Sec
4.3, having an action horizon greater than 1 helps thepolicy predict consistent actions and compensate for idle
portions of the demonstration, but too long a horizon reduces
performance due to slow reaction time. Our experimentconfirms this trade-off (Fig. 5 left) and found the action
horizon of 8 steps to be optimal for most tasks that we tested.
Robustness against latency. Diffusion Policy employs
receding horizon position control to predict a sequenceof actions into the future. This design helps address thelatency gap caused by image processing, policy inference,and network delay. Our ablation study with simulated
latency showed Diffusion Policy is able to maintain peak
performance with latency up to 4 steps (Fig 5). We also findthat velocity control is more affected by latency than position
control, likely due to compounding error effects.

5.4

Ablation Study

We explore alternative vision encoder design decisions
on the simulated robomimic square task. Specifically, weevaluated 3 different architectures: ResNet-18, ResNet-34
He et al. (2016) and ViT-B/16 Dosovitskiy et al. (2020).
For each architecture, we evaluated 3 different training
strategies: training end-to-end from scratch, using frozen
pre-trained vision encoder, and finetuning pre-trained vision
encoders (with 10x lower learning rate with respect to thepolicy network). We use ImageNet-21k Ridnik et al. (2021)
pretraining for ResNet and CLIP Radford et al. (2021)
pretraining for ViT-B/16. The quantitative comparison on
square task with proficient-human (PH) dataset is shown in
Tab. 5.
We found training ViT from scratch to be challenging(with only 22% success rate), likely due to the limited
amount data. We also found training with frozen pretrained
vision encoder to yield poor performance, which indicatesthat diffusion policy prefers different vision representation
than what is offered in popular pretraining methods.
However, we found finetuning the pretrained vision encoder∗ Due to a bug in our evaluation code, only 22 environment initializations

are used for robomimic tasks. This does not change our conclusion since allbaseline methods are evaluated in the same way.

Diffusion Policy

9Archicture &Prertain Datset

From
ScatchPretrained
frozen finetuning

Resnet18 (in21)
Resnet34 (in21)
ViT-base (clip)

0.94
0.92
0.22

0.58
0.40
0.70

0.92
0.94
0.98

Table 5. Vision Encoder Comparison All models are trainedon the robomimic square (ph) task using CNN-based diffusion
policy. Each model is trained for 500 epochs and evaluatedevery 50 epochs under 50 different environment initialconditions.

②①

Human IBC LSTM-GMM
Diffusion Policy
Demo pos vel pos
vel T-E2E ImgNet R3M E2E
IoU
0.84 0.14 0.19 0.24
Succ% 1.00 0.00 0.00 0.20
Dur. 20.3 56.3 41.6 47.3

0.25
0.10
51.7

0.53
0.65
57.5

0.24
0.15
55.8

0.66 0.80
0.80 0.95
31.7 22.9

Table 6. Realworld Push-T Experiment. a) Hardware setup.
b) Illustration of the task. The robot needs to ⃝1 precisely push
the T-shaped block into the target region, and ⃝2 move theend-effector to the end-zone. c) The ground truth end stateused to calculate IoU metrics used in this table. Table: Successis defined by the end-state IoU greater than the minimum IoU in
the demonstration dataset. Average episode duration presented
in seconds. T-E2E stands for end-to-end trained
Transformer-based Diffusion Policy

with a small learning rate (10x smaller vs diffusion
policy network) gives the best performance overall. Thisis especially true for the CLIP-trained ViT-B/16, whichreaches 98% success rate with only 50 epochs of training.
Overall, the best performance across different architectures
is not large, despite their significant theoretical capacity gap.
We anticipate that their performance gap could be more
pronounced on a complex task.

6

Realworld Evaluation

We evaluated Diffusion Policy in the realworld performanceon 4 tasks across 2 hardware setups – with training data fromdifferent demonstrators for each setup. On the realworld
Push-T task, we perform ablations examining Diffusion
Policy on 2 architecture options and 3 visual encoder options;
we also benchmarked against 2 baseline methods withboth position-control and velocity-control action spaces.
On all tasks, Diffusion Policy variants with both CNN
backbones and end-to-end-trained visual encoders yielded
the best performance. More details about the task setup andparameters may be found in supplemental materials.

6.1

Realworld Push-T Task

Real-world Push-T is significantly harder than the simulated
version due to 3 modifications: 1). The real-world Push-T
task is multi-stage. It requires the robot to ⃝1 push the T
block into the target and then ⃝2 move its end-effector into
a designated end-zone to avoid occlusion. 2). The policy
needs to make fine adjustments to make sure the T is fullyin the goal region before heading to the end-zone, creating
additional short-term multimodality. 3). The IoU metric ismeasured at the last step instead of taking the maximum
over all steps. We threshold success rate by the minimum
achieved IoU metric from the human demonstration dataset.
Our UR5-based experiment setup is shown in Tab. 6 figure.
Diffusion Policy predicts robot commands at 10 Hz and thesecommands then linearly interpolated to 125 Hz for robot
execution.
Result Analysis. Diffusion Policy performed close tohuman level with 95% success rate and 0.8 v.s. 0.84
average IoU, compared with the 0% and 20% success rate
of best-performing IBC and LSTM-GMM variants. Fig 7
qualitatively illustrates the behavior for each method startingfrom the same initial condition. We observed that poor
performance during the transition between stages is themost common failure case for the baseline method due tohigh multimodality during those sections and an ambiguous
decision boundary. LSTM-GMM got stuck near the T blockin 8 out of 20 evaluations (3rd row), while IBC prematurely
left the T block in 6 out of 20 evaluations (4th row).
We did not follow the common practice of removing idle
actions from training data due to task requirements, which
also contributed to LSTM and IBC’s tendency to overfit on
small actions and get stuck in this task. The results are best
appreciated with videos in supplemental materials.
End-to-end v.s. pre-trained vision encoders We tested
Diffusion Policy with frozen pre-trained vision encoders(ImageNet Deng et al. (2009) and R3MNair et al. (2022)), as
seen in Tab. 6. Diffusion Policy with R3M achieves an 80%
success rate but predicts jittery actions and is more likely
to get stuck compared to the end-to-end trained version.
Diffusion Policy with ImageNet showed less promising
results with abrupt actions and poor performance. We foundthat end-to-end training is still the most effective way toincorporate visual observation into Diffusion Policy, and ourbest-performing models were all end-to-end trained.
Robustness against perturbation Diffusion Policy’s
robustness against visual and physical perturbations wasevaluated in a separate episode from experiments in Tab 6.
As shown in Fig 8, three types of perturbations are applied.
1) The front camera was blocked for 3 secs by a waving hand(left column), but the diffusion policy, despite exhibitingsome jitter, remained on-course and pushed the T block
into position. 2) We shifted the T block while Diffusion
Policy was making fine adjustments to the T block’s position.
Diffusion policy immediately re-planned to push from the
opposite direction, negating the impact of perturbation. 3)
We moved the T block while the robot was en route to theend-zone after the first stage’s completion. The Diffusion
Policy immediately changed course to adjust the T blockback to its target and then continued to the end-zone.

10
Time

Avg of End States

Diffusion
Policy(End2End)A
Diffusion
Policy(R3M)

B

LSTM-GMM(End2End)C

IBC(End2End)D
Figure 7. Realworld Push-T Comparisons. Columns 1-4 show action trajectories based on key events. The last column shows
averaged images of the end state. A: Diffusion policy (End2End) achieves more accurate and consistent end states. B: Diffusion
Policy (R3M) gets stuck initially but later recovers and finishes the task. C: LSTM-GMM fails to reach the end zone while adjusting
the T block, blocking the eval camera view. D: IBC prematurely ends the pushing stage.
VisualOcclusion

Perturbation during
Pushing Stage

Perturbation during
Finishing Stage

1
2

Figure 8. Robustness Test for Diffusion Policy. Left: Awaving hand in front of the camera for 3 seconds causes slightjitter, but the predicted actions still function as expected.
Middle: Diffusion Policy immediately corrects shifted block
position to the goal state during the pushing stage. Right: Policy
immediately aborts heading to the end zone, returning the block
to goal state upon detecting block shift. This novel behavior wasnever demonstrated. Please check the videos in the
supplementary material.

6.2

Mug Flipping Task

The mug flipping task is designed to test Diffusion Policy’s
ability to handle complex 3D rotations while operating close
to the hardware’s kinematic limits. The goal is to reorient a
randomly placed mug to have ⃝1 the lip facing down ⃝2 thehandle pointing left, as shown in Fig. 9. Depending on the
mug’s initial pose, the demonstrator might directly place themug in desired orientation, or may use additional push of thehandle to rotation the mug. As a result, the demonstration
dataset is highly multi-modal: grasp vs push, different typesof grasps (forehand vs backhand) or local grasp adjustments(rotation around mug’s principle axis), and are particularly
challenging for baseline approaches to capture.
Result Analysis. Diffusion policy is able to complete
this task with 90% success rate over 20 trials. The richness

Succ %

Human

LSTM-GMM

Diffusion Policy

1.0

0.0

0.9

Figure 9. 6DoF Mug Flipping Task. The robot needs to ⃝1
Pickup a randomly placed mug and place it lip down (marked
orange). ⃝2 Rotate the mug such that its handle is pointing left.

of captured behaviors is best appreciated with the video.
Although never demonstrated, the policy is also able tosequence multiple pushes for handle alignment or regrasps
for dropped mug when necessary. For comparison, we also
train a LSTM-GMM policy trained with a subset of the samedata. For 20 in-distribution initial conditions, the LSTMGMM policy never aligns properly with respect to the mug,and fails to grasp in all trials.

6.3

Sauce Pouring and Spreading

The sauce pouring and spreading tasks are designed to test
Diffusion Policy’s ability to work with non-rigid objects, 6
Dof action spaces, and periodic actions in real-world setups.
Our Franka Panda setup and tasks are shown in Fig 10. The
goal for the 6DoF pouring task is to pour one full ladle ofsauce onto the center of the pizza dough, with performancemeasured by IoU between the poured sauce mask and aDiffusion Policy

11
Spreading Goal

Pouring Goal

1

4

3
2

1

3

2

Pour
IoU Succ

the ladle was successfully lifted, the sauce was poured
off-centered. LSTM-GMM failed to self-terminate in alltrials. We suspect LSTM-GMM’s hidden state failed tocapture sufficiently long history to distinguish between theladle dipping and the lifting phases of the task. For sauce
spreading, LSTM-GMM always lifts the spoon right after
the start, and failed to make contact with the sauce in all 20
experiments.

7

SpreadCoverage Succ %

Human

0.79

1.00

0.79

1.00

LSTM-GMM
Diffusion Policy

0.06
0.74

0.00
0.79

0.27
0.77

0.00
1.00

Figure 10. Realworld Sauce Manipulation. [Left] 6DoF
pouring Task. The robot needs to ⃝1 dip the ladle to scoop
sauce from the bowl, ⃝2 approach the center of the pizza
dough, ⃝3 pour sauce, and ⃝4 lift the ladle to finish the task.
[Right] Periodic spreading Task The robot needs to ⃝1
approach the center of the sauce with a grasped spoon, ⃝2
spread the sauce to cover pizza in a spiral pattern, and ⃝3 liftthe spoon to finish the task.

nominal circle at the center of the pizza dough (illustrated
by the green circle in Fig 10). The goal for the periodic
spreading task is to spread sauce on pizza dough, withperformance measured by sauce coverage. Variations acrossevaluation episodes come from random locations for the
dough and the sauce bowl. The success rate is computed bythresholding with minimum human performance. Results are
best viewed in supplemental videos. Both tasks were trainedwith the same Push-T hyperparameters, and successfulpolicies were achieved on the first attempt.
The sauce pouring task requires the robot to remain
stationary for a period of time to fill the ladle with viscoustomato sauce. The resulting idle actions are known to bechallenging for behavior cloning algorithms and therefore
are often avoided or filtered out. Fine adjustments during
pouring are necessary during sauce pouring to ensure
coverage and to achieve the desired shape.
The demonstrated sauce-spreading strategy is inspired by
the human chef technique, which requires both a longhorizon cyclic pattern to maximize coverage and shorthorizon feedback for even distribution (since the tomato
sauce used often drips out in lumps with unpredictable
sizes). Periodic motions are known to be difficult to learn
and therefore are often addressed by specialized action
representations Yang et al. (2022). Both tasks require thepolicy to self-terminate by lifting the ladle/spoon.
Result Analysis. Diffusion policy achieves close-tohuman performance on both tasks, with coverage 0.74 vs
0.79 on pouring and 0.77 vs 0.79 on spreading. Diffusion
policy reacted gracefully to external perturbations suchas moving the pizza dough by hand during pouring andspreading. Results are best appreciated with videos in the
supplemental material.
LSTM-GMM performs poorly on both sauce pouring andspreading tasks. It failed to lift the ladle after successfully
scooping sauce in 15 out of 20 of the pouring trials. When

Realworld Bimanual Tasks

Beyond single arm setup, we further demonstrate Diffusion
Policy on several challenging bimanual tasks. To enablebimanual tasks, the majority of effort was spent on extending
our robot stack to support multi-arm teleopration and control.
Diffusion Policy worked out of the box for these tasks
without hyperparameter tuning.

7.1

Observation and Action Spaces

The proprioceptive observation space is extended to include
the poses of both end-effectors and the gripper widths of bothgrippers. We also extend the observation space to include
the actual and desired values of these quantities. The image
observation space is comprised of two scene cameras andtwo wrist cameras, one attached to each arm. The action
space is extended to include the desired poses of both endeffectors and the desired gripper widths of both grippers.

7.2

Teleoperation

For these coordinated bimanual tasks, we found using
2 SpaceMouse simultaneously quite challenging for thedemonstrator. Thus, we implemented two new teleoperation
modes: using a Meta Quest Pro VR device with two
hand controllers, or haptic-enabled control using 2 Haption
Virtuose™ 6D HF TAO devices using bilateral positionposition coupling as described succinctly in the haptics
section of Siciliano et al. (2008). This coupling is performed
between a Haption device and a Franka Panda arm. Moredetails on the controllers themselves may be found in Sec.
D.1. The following provides more details on each task andpolicy performance.

7.3

Bimanual Egg Beater

The bimanual egg beater task is illustrated and describedin Fig. 11, using a OXO™ Egg Beater and a Room
Essentials™ plastic bowl. We chose this task to illustrate theimportance of haptic feedback for teleoperating bimanual
manipulation even for common daily life tasks such as
coordinated tool use. Without haptic feedback, an expert was
unable to successfully complete a single demonstration out
of 10 trials. 5 failed due to robot pulling the crank handle
off the egg beater; 3 failed due to robot losing grasp of
the handle; 2 failed due to robot triggering torque limit. In
contrast, the same operator could easily perform this task 10
out of 10 times with haptic feedback. Using haptic feedback
made it possible for the demonstrations to be both quicker
and higher quality than without feedback.
Result Analysis. Diffusion policy is able to complete thistask with 55% success rate over 20 trials, trained using
210 demonstrations. The primary failure modes for these

12

4

5

3

2

1

Figure 11. Bimanual Egg Beater Manipulation. The robot
needs to ⃝1 push the bowl into position (only if too close to theleft arm), ⃝2 approach and pick up the egg beater with the rightarm, ⃝
3 place the egg beater in the bowl, ⃝4 approach andgrasp the egg beater crank handle, and ⃝5 turn the crankhandle 3 or more times.
1
3

5
2

4

6

Figure 12. Bimanual Mat Unrolling. The robot needs to ⃝1
pick up one side of the mat (if needed), using the left or rightarm, ⃝
2 lift and unroll the mat (if needed), ⃝3 ensure that bothsides of the mat are grasped, ⃝4 lift the mat, ⃝5 place the matoriented with the table, mostly centered, and ⃝6 release the mat.

were out-of-domain initial positioning of the egg beater, or
missing the egg beater crank handle or losing grasp of it. The
initial and final states for all rollouts are visualized in 18 and
19.

7.4

Bimanual Mat Unrolling

The mat unrolling task is shown and described in Fig.
12, using a XXL Dog Buddy™ Dog Mat. This task wasteleoperated using the VR setup, as it did not require richhaptic feedback to perform the task. We taught this skill tobe omnidextrous, meaning it can unroll either to the left or
right depending on the initial condition.
Result Analysis. Diffusion policy is able to complete thistask with 75% success rate over 20 trials, trained using 162
demonstrations. The primary failure modes for these were
missed grasps during initial grasp of the mat, where thepolicy struggled to correct itself and thus got stuck repeating
the same behavior. The initial and final states for all rollouts
are visualized in 16 and 17.

7.5

Bimanual Shirt Folding.

The shirt folding task is described and illustrated in Fig. 13,
using a short-sleeve T-shirt. This task was also teleoperatedusing the VR setup as it did not require rich feedback
to perform the task. Due to the kinematic and workspaceconstraints, this task is notably longer and can take up
to nine discrete steps. The last few steps require bothgrippers to come very close towards each other. Having our
mid-level controller explicitly handling collision avoidance
was especially important for both teleoperation and policy
rollout.
Result Analysis. Diffusion policy is able to complete thistask with 75% success rate over 20 trials, trained using 284
demonstrations. The primary failure modes for these were

Figure 13. Bimanual Shirt Folding. The robot needs to ⃝1
approach and grasp the closest sleeve with both arms, ⃝2 foldthe sleeve and release, ⃝3 drag the shirt closer (if needed), ⃝4
approach and grasp the other sleeve with both arms, ⃝5 fold thesleeve and release, ⃝6 drag the shirt to a orientation for folding,⃝
7 grasp and fold the shirt in half by its collar, ⃝8 drag the shirt
to the center, and ⃝9 smooth out the shirt and move the arms
away.

missed grasps for initial folding (the sleeves and the color),
and the policy being unable to stop adjusting the shirt at theend. The initial and final states for all rollouts are visualized
in 20 and 21.

8

Related Work

Creating capable robots without requiring explicit programming of behaviors is a longstanding challenge in the field
Atkeson and Schaal (1997); Argall et al. (2009); Ravichandar
et al. (2020). While conceptually simple, behavior cloning
has shown surprising promise on an array of real-world
robot tasks, including manipulation Zhang et al. (2018);
Florence et al. (2019); Mandlekar et al. (2020b,a); Zenget al. (2021); Rahmatizadeh et al. (2018); Avigal et al. (2022)
and autonomous driving Pomerleau (1988); Bojarski et al.
(2016). Current behavior cloning approaches can be categorized into two groups, depending on the policy’s structure.
Explicit Policy. The simplest form of explicit policies
maps from world state or observation directly to action
Pomerleau (1988); Zhang et al. (2018); Florence et al.
(2019); Ross et al. (2011); Toyer et al. (2020); Rahmatizadeh
et al. (2018); Bojarski et al. (2016). They can besupervised with a direct regression loss and have efficient
inference time with one forward pass. Unfortunately,
this type of policy is not suitable for modeling multimodal demonstrated behavior. A popular approach tomodel multimodal action distributions while maintaining
the simplicity of direction action mapping is convert
the regression task into classification by discretizing theaction space Zeng et al. (2021); Wu et al. (2020); Avigal
et al. (2022). However, the number of bins needed toapproximate a continuous action space grows exponentiallywith increasing dimensionality. Another approach is tocombine Categorical and Gaussian distributions to represent
continuous multimodal distributions via the use of MDNs
Bishop (1994); Mandlekar et al. (2021) or clustering with

Diffusion Policy

offset prediction Shafiullah et al. (2022); Sharma et al.
(2018). Nevertheless, these models tend to be sensitive tohyperparameter tuning, exhibit mode collapse, and are still
limited in their ability to express high-precision behavior
Florence et al. (2021).
Implicit Policy. Implicit policies (Florence et al. 2021;
Jarrett et al. 2020) define distributions over actions by using
Energy-Based Models (EBMs) (LeCun et al. 2006; Du andMordatch 2019; Dai et al. 2019; Grathwohl et al. 2020;
Du et al. 2020). In this setting, each action is assigned
an energy value, with action prediction corresponding tothe optimization problem of finding a minimal energy
action. Since different actions may be assigned low
energies, implicit policies naturally represent multi-modal
distributions. However, existing implicit policies (Florence
et al. 2021) are unstable to train due to the necessity ofdrawing negative samples when computing the underlying
Info-NCE loss.
Diffusion Models. Diffusion models are probabilistic
generative models that iteratively refine randomly sampled
noise into draws from an underlying distribution. They canalso be conceptually understood as learning the gradient fieldof an implicit action score and then optimizing that gradientduring inference. Diffusion models (Sohl-Dickstein et al.
2015; Ho et al. 2020) have recently been applied to solve
various different control tasks (Janner et al. 2022a; Urain
et al. 2022; Ajay et al. 2022).
In particular, Janner et al. (2022a) and Huang et al.
(2023) explore how diffusion models may be used in thecontext of planning and infer a trajectory of actions that
may be executed in a given environment. In the context ofReinforcement Learning, Wang et al. (2022) use diffusion
model for policy representation and regularization with statebased observations. In contrast, in this work, we explore
how diffusion models may instead be effectively applied in
the context of behavioral cloning for effective visuomotor
control policy. To construct effective visuomotor control
policies, we propose to combine DDPM’s ability to predicthigh-dimensional action squences with closed-loop control,
as well as a new transformer architecture for action diffusion
and a manner to integrate visual inputs into the action
diffusion model.

13

9Limitations and Future Work

Although we have demonstrated the effectiveness ofdiffusion policy in both simulation and real-world systems,
there are limitations that future work can improve. First,
our implementation inherits limitations from behavior
cloning, such as suboptimal performance with inadequate
demonstration data. Diffusion policy can be applied toother paradigms, such as reinforcement learning Wang et al.
(2023); Hansen-Estruch et al. (2023), to take advantage ofsuboptimal and negative data. Second, diffusion policy has
higher computational costs and inference latency comparedto simpler methods like LSTM-GMM. Our action sequence
prediction approach partially mitigates this issue, but may
not suffice for tasks requiring high rate control. Future
work can exploit the latest advancements in diffusion model
acceleration methods to reduce the number of inference
steps required, such as new noise schedules Chen (2023),
inference solvers Karras et al. (2022), and consistencymodels Song et al. (2023).

10

Conclusion

In this work, we assess the feasibility of diffusion-based
policies for robot behaviors. Through a comprehensive
evaluation of 15 tasks in simulation and the real world,
we demonstrate that diffusion-based visuomotor policies
consistently and definitively outperform existing methods
while also being stable and easy to train. Our results also
highlight critical design factors, including receding-horizon
action prediction, end-effector position control, and efficientvisual conditioning, that is crucial for unlocking the fullpotential of diffusion-based policies. While many factors
affect the ultimate quality of behavior-cloned policies —including the quality and quantity of demonstrations, thephysical capabilities of the robot, the policy architecture,and the pretraining regime used — our experimental results
strongly indicate that policy structure poses a significant
performance bottleneck during behavior cloning. We hopethat this work drives further exploration in the field into
diffusion-based policies and highlights the importance ofconsidering all aspects of the behavior cloning process
beyond just the data used for policy training.

Wang et al. (2023) explore how diffusion models learned
from expert demonstrations can be used to augmentclassical explicit polices without directly taking advantage
of diffusion models as policy representation.

11

Concurrent to us, Pearce et al. (2023), Reuss et al.
(2023) and Hansen-Estruch et al. (2023) has conducted
a complimentary analysis of diffusion-based policies insimulated environments. While they focus more on effective
sampling strategies, leveraging classifier-free guidance forgoal-conditioning as well as applications in Reinforcement
Learning, and we focus on effective action spaces, ourempirical findings largely concur in the simulated regime.
In addition, our extensive real-world experiments provide
strong evidence for the importance of a receding-horizon
prediction scheme, the careful choice between velocity andposition control, and the necessity of optimization for realtime inference and other critical design decisions for aphysical robot system.

FundingAcknowledgement

We’d like to thank Naveen Kuppuswamy, Hongkai Dai,
Aykut Önol, Terry Suh, Tao Pang, Huy Ha, Samir Gadre,
Kevin Zakka and Brandon Amos for their thoughtful
discussions. We thank Jarod Wilson for 3D printing supportand Huy Ha for photography and lighting advice. We thank
Xiang Li for discovering the bug in our evaluation code on
GitHub.

This work was supported by the Toyota Research Institute, NSF
CMMI-2037101 and NSF IIS-2132519. We would like to thank
Google for the UR5 robot hardware. The views and conclusions
contained herein are those of the authors and should not beinterpreted as necessarily representing the official policies, either
expressed or implied, of the sponsors.

14

References
Ajay A, Du Y, Gupta A, Tenenbaum J, Jaakkola T and Agrawal
P (2022) Is conditional generative modeling all you need for
decision-making? arXiv preprint arXiv:2211.15657 .
Argall BD, Chernova S, Veloso M and Browning B (2009) A
survey of robot learning from demonstration. Robotics andautonomous systems 57(5): 469–483.
Atkeson CG and Schaal S (1997) Robot learning fromdemonstration. In: ICML, volume 97. pp. 12–20.
Avigal Y, Berscheid L, Asfour T, Kröger T and Goldberg K (2022)
Speedfolding: Learning efficient bimanual folding of garments.
In: 2022 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). IEEE, pp. 1–8.
Bishop CM (1994) Mixture density networks. Aston University.
Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B,
Goyal P, Jackel LD, Monfort M, Muller U, Zhang J et al.
(2016) End to end learning for self-driving cars. arXiv preprint
arXiv:1604.07316 .
Chen T (2023) On the importance of noise scheduling for diffusion
models. arXiv preprint arXiv:2301.10972 .
Chi C, Feng S, Du Y, Xu Z, Cousineau E, Burchfiel B and Song S(2023) Diffusion policy: Visuomotor policy learning via action
diffusion. In: Proceedings of Robotics: Science and Systems(RSS).
Dai B, Liu Z, Dai H, He N, Gretton A, Song L and Schuurmans D(2019) Exponential family estimation via adversarial dynamics
embedding. Advances in Neural Information Processing
Systems 32.
Deng J, Dong W, Socher R, Li LJ, Li K and Fei-Fei L (2009)
Imagenet: A large-scale hierarchical image database. In: 2009
IEEE conference on computer vision and pattern recognition.
Ieee, pp. 248–255.
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X,
Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S
et al. (2020) An image is worth 16x16 words: Transformers forimage recognition at scale. arXiv preprint arXiv:2010.11929 .
Du Y, Li S, Tenenbaum J and Mordatch I (2020) Improved
contrastive divergence training of energy based models. arXiv
preprint arXiv:2012.01316 .
Du Y and Mordatch I (2019) Implicit generation and generalizationin energy-based models. arXiv preprint arXiv:1903.08689 .
Florence P, Lynch C, Zeng A, Ramirez OA, Wahid A, Downs L,
Wong A, Lee J, Mordatch I and Tompson J (2021) Implicit
behavioral cloning. In: 5th Annual Conference on Robot
Learning.
Florence P, Manuelli L and Tedrake R (2019) Self-supervised
correspondence in visuomotor policy learning. IEEE Robotics
and Automation Letters 5(2): 492–499.
Grathwohl W, Wang KC, Jacobsen JH, Duvenaud D and Zemel
R (2020) Learning the stein discrepancy for training andevaluating energy-based models without sampling.
In:
International Conference on Machine Learning.
Gupta A, Kumar V, Lynch C, Levine S and Hausman K (2019)
Relay policy learning: Solving long-horizon tasks via imitation
and reinforcement learning. arXiv preprint arXiv:1910.11956 .
Hansen-Estruch P, Kostrikov I, Janner M, Kuba JG and Levine S(2023) Idql: Implicit q-learning as an actor-critic method withdiffusion policies. arXiv preprint arXiv:2304.10573 .

He K, Fan H, Wu Y, Xie S and Girshick R (2020) Momentum
contrast for unsupervised visual representation learning. In:
Proceedings of the IEEE/CVF conference on computer visionand pattern recognition. pp. 9729–9738.
He K, Zhang X, Ren S and Sun J (2016) Deep residual learning forimage recognition. In: Proceedings of the IEEE conference on
computer vision and pattern recognition. pp. 770–778.
Ho J, Jain A and Abbeel P (2020) Denoising diffusion probabilistic
models. arXiv preprint arXiv:2006.11239 .
Huang S, Wang Z, Li P, Jia B, Liu T, Zhu Y, Liang W and Zhu SC(2023) Diffusion-based generation, optimization, and planningin 3d scenes. arXiv preprint arXiv:2301.06015 .
Janner M, Du Y, Tenenbaum J and Levine S (2022a) Planningwith diffusion for flexible behavior synthesis. In: International
Conference on Machine Learning.
Janner M, Du Y, Tenenbaum J and Levine S (2022b) Planning withdiffusion for flexible behavior synthesis. In: Chaudhuri K,
Jegelka S, Song L, Szepesvari C, Niu G and Sabato S (eds.)
Proceedings of the 39th International Conference on Machine
Learning, Proceedings of Machine Learning Research. PMLR.
Jarrett D, Bica I and van der Schaar M (2020) Strictly batch
imitation learning by energy-based distribution matching.
Advances in Neural Information Processing Systems 33: 7354–
7365.
Karras T, Aittala M, Aila T and Laine S (2022) Elucidating thedesign space of diffusion-based generative models. arXiv
preprint arXiv:2206.00364 .
Khatib O (1987) A unified approach for motion and force control ofrobot manipulators: The operational space formulation. IEEE
Journal on Robotics and Automation 3(1): 43–53. DOI:10.
1109/JRA.1987.1087068. Conference Name: IEEE Journal on
Robotics and Automation.
LeCun Y, Chopra S, Hadsell R, Huang FJ and et al (2006) A tutorial
on energy-based learning. In: Predicting Structured Data. MIT
Press.
Liu L, Liu X, Gao J, Chen W and Han J (2020) Understanding
the difficulty of training transformers.
arXiv preprint
arXiv:2004.08249 .
Liu N, Li S, Du Y, Torralba A and Tenenbaum JB (2022)
Compositional visual generation with composable diffusion
models. arXiv preprint arXiv:2206.01714 .
Mandlekar A, Ramos F, Boots B, Savarese S, Fei-Fei L, Garg A andFox D (2020a) Iris: Implicit reinforcement without interaction
at scale for learning control from offline robot manipulation
data. In: 2020 IEEE International Conference on Robotics andAutomation (ICRA). IEEE.
Mandlekar A, Xu D, Martı́n-Martı́n R, Savarese S and Fei-Fei L(2020b) Learning to generalize across long-horizon tasks fromhuman demonstrations. arXiv preprint arXiv:2003.06085 .
Mandlekar A, Xu D, Wong J, Nasiriany S, Wang C, Kulkarni R,
Fei-Fei L, Savarese S, Zhu Y and Martı́n-Martı́n R (2021)
What matters in learning from offline human demonstrations
for robot manipulation. In: 5th Annual Conference on Robot
Learning.
Mayne DQ and Michalska H (1988) Receding horizon control
of nonlinear systems. In: Proceedings of the 27th IEEE
Conference on Decision and Control. IEEE, pp. 464–465.
Nair S, Rajeswaran A, Kumar V, Finn C and Gupta A (2022) R3m:
A universal visual representation for robot manipulation. In:
6th Annual Conference on Robot Learning.

Diffusion Policy

Neal RM et al. (2011) Mcmc using hamiltonian dynamics.
Handbook of markov chain monte carlo .
Nichol AQ and Dhariwal P (2021) Improved denoising diffusion
probabilistic models. In: International Conference on Machine
Learning. PMLR, pp. 8162–8171.
Pearce T, Rashid T, Kanervisto A, Bignell D, Sun M, Georgescu R,
Macua SV, Tan SZ, Momennejad I, Hofmann K et al. (2023)
Imitating human behaviour with diffusion models. arXiv
preprint arXiv:2301.10677 .
Perez E, Strub F, De Vries H, Dumoulin V and Courville A (2018)
Film: Visual reasoning with a general conditioning layer. In:
Proceedings of the AAAI Conference on Artificial Intelligence.
Pomerleau DA (1988) Alvinn: An autonomous land vehicle in aneural network. Advances in neural information processing
systems 1.
Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S,
Sastry G, Askell A, Mishkin P, Clark J et al. (2021) Learningtransferable visual models from natural language supervision.
In: International conference on machine learning. PMLR, pp.
8748–8763.
Rahmatizadeh R, Abolghasemi P, Bölöni L and Levine S (2018)
Vision-based multi-task manipulation for inexpensive robots
using end-to-end learning from demonstration. In: 2018 IEEE
international conference on robotics and automation (ICRA).
IEEE, pp. 3758–3765.
Ravichandar H, Polydoros AS, Chernova S and Billard A (2020)
Recent advances in robot learning from demonstration. Annual
review of control, robotics, and autonomous systems 3: 297–
330.
Reuss M, Li M, Jia X and Lioutikov R (2023) Goal-conditioned
imitation learning using score-based diffusion policies. In:
Proceedings of Robotics: Science and Systems (RSS).
Ridnik T, Ben-Baruch E, Noy A and Zelnik-Manor L (2021)
Imagenet-21k pretraining for the masses.
Ronneberger O, Fischer P and Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–
MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer,
pp. 234–241.
Ross S, Gordon G and Bagnell D (2011) A reduction of imitation
learning and structured prediction to no-regret online learning.
In: Proceedings of the fourteenth international conference
on artificial intelligence and statistics. JMLR Workshop andConference Proceedings, pp. 627–635.
Shafiullah NMM, Cui ZJ, Altanzaya A and Pinto L (2022) Behavior
transformers: Cloning $k$ modes with one stone. In: Oh AH,
Agarwal A, Belgrave D and Cho K (eds.) Advances in Neural
Information Processing Systems.
Sharma P, Mohan L, Pinto L and Gupta A (2018) Multiple
interactions made easy (mime): Large scale demonstrations
data for imitation. In: Conference on robot learning. PMLR.
Siciliano B, Khatib O and Kröger T (2008) Springer handbook ofrobotics, volume 200. Springer.
Sohl-Dickstein J, Weiss E, Maheswaranathan N and Ganguli
S (2015) Deep unsupervised learning using nonequilibrium
thermodynamics. In: International Conference on Machine
Learning.
Song J, Meng C and Ermon S (2021) Denoising diffusion
implicit models. In: International Conference on Learning

15

Representations.
Song Y, Dhariwal P, Chen M and Sutskever I (2023) Consistencymodels. arXiv preprint arXiv:2303.01469 .
Song Y and Ermon S (2019) Generative modeling by estimating
gradients of the data distribution.
Advances in neural
information processing systems 32.
Subramanian J and Mahajan A (2019) Approximate information
state for partially observed systems. In: 2019 IEEE 58th
Conference on Decision and Control (CDC). IEEE, pp. 1629–
1636.
Ta DN, Cousineau E, Zhao H and Feng S (2022) Conditionalenergy-based models for implicit policies: The gap between
theory and practice. arXiv preprint arXiv:2207.05824 .
Tancik M, Srinivasan P, Mildenhall B, Fridovich-Keil S, Raghavan
N, Singhal U, Ramamoorthi R, Barron J and Ng R (2020)
Fourier features let networks learn high frequency functions
in low dimensional domains. Advances in Neural Information
Processing Systems 33: 7537–7547.
Toyer S, Shah R, Critch A and Russell S (2020) The magicalbenchmark for robust imitation.
Advances in Neural
Information Processing Systems 33: 18284–18295.
Urain J, Funk N, Chalvatzaki G and Peters J (2022) Se (3)diffusionfields: Learning cost functions for joint grasp andmotion optimization through diffusion.
arXiv preprint
arXiv:2209.03855 .
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN,
Kaiser Ł and Polosukhin I (2017) Attention is all you need.
Advances in neural information processing systems 30.
Wang Z, Hunt JJ and Zhou M (2022) Diffusion policies as an
expressive policy class for offline reinforcement learning.
arXiv preprint arXiv:2208.06193 .
Wang Z, Hunt JJ and Zhou M (2023) Diffusion policies as an
expressive policy class for offline reinforcement learning.
In: The Eleventh International Conference on Learning
Representations.
URL https://openreview.net/
forum?id=AHvFDPi-FA.
Welling M and Teh YW (2011) Bayesian learning via stochastic
gradient langevin dynamics. In: Proceedings of the 28th
international conference on machine learning (ICML-11). pp.
681–688.
Wu J, Sun X, Zeng A, Song S, Lee J, Rusinkiewicz S
and Funkhouser T (2020) Spatial action maps for mobile
manipulation. In: Proceedings of Robotics: Science andSystems (RSS).
Wu Y and He K (2018) Group normalization. In: Proceedings of theEuropean conference on computer vision (ECCV). pp. 3–19.
Yang J, Zhang J, Settle C, Rai A, Antonova R and Bohg J (2022)
Learning periodic tasks from human demonstrations. In: 2022
International Conference on Robotics and Automation (ICRA).
IEEE, pp. 8658–8665.
Zeng A, Florence P, Tompson J, Welker S, Chien J, Attarian M,
Armstrong T, Krasin I, Duong D, Sindhwani V et al. (2021)
Transporter networks: Rearranging the visual world for robotic
manipulation. In: Conference on Robot Learning. PMLR, pp.
726–747.
Zhang A, McAllister RT, Calandra R, Gal Y and Levine S(2020) Learning invariant representations for reinforcement
learning without reconstruction. In: International Conference
on Learning Representations.

A
A.1

Diffusion Policy Implementation Details
Normalization

Properly normalizing action data is critical to achieve best
performance for Diffusion Policy. Scaling the min and maxof each action dimension independently to [−1, 1] works wellfor most tasks. Since DDPMs clip prediction to [−1, 1] ateach iteration to ensure stability, the common zero-mean
unit-variance normalization will cause some region of theaction space to be inaccessible. When the data variance issmall (e.g., near constant value), shift the data to zero-mean
without scaling to prevent numerical issues. We leave action
dimensions corresponding to rotation representations (e.g.
Quaternion) unchanged.

A.2

Rotation Representation

For all environments with velocity control action space, we
followed the standard practice Mandlekar et al. (2021) to use
3D axis-angle representation for the rotation component ofaction. Since velocity action commands are usually close
to 0, the singularity and discontinuity of the axis-angle
representation don’t usually cause problems. We used the
6D rotation representation proposed in Zhou et al. (2019) forall environments (real-world and simulation) with positional
control action space.

A.3

Image Augmentation

Following Mandlekar et al. (2021), we employed random
crop augmentation during training. The crop size for eachtask is indicated in Tab. 7. During inference, we take a static
center crop with the same size.

A.4

Hyperparameters

Hyerparameters used for Diffusion Policy on both simulation
and realworld benchmarks are shown in Tab. 7 and Tab.
8. Since the Block Push task uses a Markovian scripted
oracle policy to generate demonstration data, we found its
optimal hyper parameter for observation and action horizon
to be very different from other tasks with human teleop
demostrations.
During tuning, we found increasing the number ofparameters in CNN-based Diffusion Policy always improves
performance, therefore the optimal model size is limited by
the available compute and memory capacity. On the other
hand, increasing model size for transformer-based Diffusion
Policy (in particular number of layers) hurts performancesometimes. For CNN-based Diffusion Policy, We foundusing FiLM conditioning to pass-in observations is better
than impainting on all tasks except Push-T. Performance

7XEXI'22

:MWMSR'22

:MWMSR8VERWJSVQIV

                       
3FW,SVM^SRWXITW
4YWL8
7UYEVI

Figure 14. Observation Horizon Ablation Study. State-based
Diffusion Policy is not sensitive to observation horizon.
Vision-based Diffusion Policy prefers low but > 1 observation
horizon, with 2 being a good compromise for most tasks.

6IPEXMZI4IVJ'LERKI

Zhang T, McCarthy Z, Jow O, Lee D, Chen X, Goldberg K
and Abbeel P (2018) Deep imitation learning for complex
manipulation tasks from virtual reality teleoperation. In: 2018
IEEE International Conference on Robotics and Automation(ICRA). IEEE, pp. 5628–5635.
Zhou Y, Barnes C, Lu J, Yang J and Li H (2019) On the continuityof rotation representations in neural networks. In: Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern
Recognition. pp. 5745–5753.

6IPEXMZI4IVJ'LERKI

16








4YWL8(EXE)JJMGMIRG]

7UYEVI(EXE)JJMGMIRG]         
8VEMRMRK(IQS)TMWSHIW8VEMRMRK(IQS)TMWSHIW
(MJJYWMSR4SPMG]
0781+11

Figure 15. Data Efficiency Ablation Study. Diffusion Policy
outperforms LSTM-GMM Mandlekar et al. (2021) at every
training dataset size.

reported for DiffusionPolicy-C on Push-T in Tab. 1 used
impaiting instead of FiLM.
On simulation benchmarks, we used the iDDPM algorithm
Nichol and Dhariwal (2021) with the same 100 denoisingdiffusion iterations for both training and inference. We used
DDIM Song et al. (2021) on realworld benchmarks to reducethe inference denoising iterations to 16 therefore reducing
inference latency.
We used batch size of 256 for all state-based experiments
and 64 for all image-based experiments. For learning-ratescheduling, we used cosine schedule with linear warmup.
CNN-based Diffusion Policy is warmed up for 500 steps
while Transformer-based Diffusion Policy is warmed up for1000 steps.

A.5

Data Efficiency

We found Diffusion Policy to outperform LSTM-GMM
Mandlekar et al. (2021) at every training dataset size, as
shown in Fig. 15.

B
B.1

Additional Ablation Results
Observation Horizon

We found state-based Diffusion Policy to be insensitive
to observation horizon, as shown in Fig. 14. However,vision-based Diffusion Policy, in particular the variant with
CNN backbone, see performance decrease with increasing
observation horizon. In practice, we found an observation
horizon of 2 is good for most of the tasks for both state andimage observations.

Diffusion Policy

17

H-ParamCtrl To Ta Tp ImgRes

CropRes#D-Params #V-Params Lr

WDecay D-Iters Train D-Iters Eval

Lift
Can
SquareTransport
ToolHang
Push-T
Block Push
Kitchen

Pos
Pos
Pos
Pos
Pos
Pos
Pos
Pos

2
2
2
2
2
2
3
2

8
8
8
8
8
8
1
8

16
16
16
16
16
16
12
16

2x84x84
2x76x76256
2x84x84
2x76x76256
2x84x84
2x76x76256
4x84x85
4x76x76264
2x240x240 2x216x216 256
1x96x961x84x84
256
N/A
N/A
256
N/A
N/A
256

22
22
22
45
22
22
0
0

1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6

100
100
100
100
100
100
100
100

100
100
100
100
100
100
100
100

Real Push-T
Real Pour
Real Spread
Real Mug Flip

Pos
Pos
Pos
Pos

2
2
2
2

6
8
8
8

16
16
16
16

2x320x240 2x288x216 67
2x320x240 2x288x216 67
2x320x240 2x288x216 67
2x320x240 2x288x216 67

22
22
22
22

1e-4 1e-6
1e-4 1e-6
1e-4 1e-6
1e-4 1e-6

100
100
100
100

16
16
16
16

Table 7. Hyperparameters for CNN-based Diffusion Policy Ctrl: position or velocity control To: observation horizon Ta: action
horizon Tp: action prediction horizon ImgRes: environment observation resolution (Camera views x W x H) CropRes: random crop
resolution #D-Params: diffusion network number of parameters in millions #V-Params: vision encoder number of parameters inmillions Lr: learining rate WDecay: weight decay D-Iters Train: number of training diffusion iterations D-Iters Eval: number ofinference diffusion iterations (enabled by DDIM Song et al. (2021))

H-ParamCtrl To Ta Tp #D-params #V-params #Layers Emb Dim Attn Drp Lr

Lift
Can
SquareTransport
ToolHang
Push-T
Block Push
Kitchen

Pos
Pos
Pos
Pos
Pos
Pos
Vel
Pos

WDecay D-Iters Train D-Iters Eval

2
2
2
2
2
2
3
4

8
8
8
8
8
8
1
8

10 9
10 9
10 9
10 9
10 9
16 9
5 9
16 80

22
22
22
45
22
22
0
0

8
8
8
8
8
8
8
8

256
256
256
256
256
256
256
768

0.3
0.3
0.3
0.3
0.3
0.01
0.3
0.1

1e-4 1e-3
1e-4 1e-3
1e-4 1e-3
1e-4 1e-3
1e-4 1e-3
1e-4 1e-1
1e-4 1e-3
1e-4 1e-3

100
100
100
100
100
100
100
100

100
100
100
100
100
100
100
100

Real Push-T Pos 2

6

16 80

22

8

768

0.3

1e-4 1e-3

100

16Table 8. Hyperparameters for Transformer-based Diffusion Policy Ctrl: position or velocity control To: observation horizon Ta:
action horizon Tp: action prediction horizon #D-Params: diffusion network number of parameters in millions #V-Params: vision
encoder number of parameters in millions Emb Dim: transformer token embedding dimension Attn Drp: transformer attention
dropout probability Lr: learining rate WDecay: weight decay (for transformer only) D-Iters Train: number of training diffusion
iterations D-Iters Eval: number of inference diffusion iterations (enabled by DDIM Song et al. (2021))

B.2

Performance Improvement Calculation

For each task i (column) reported in Tab. 1, Tab. 2
and Tab. 4 (mh results ignored), we find the maximum
performance for baseline methods max baselinei andthe maximum performance for Diffusion Policy variant(CNN vs Transformer) max oursi . For each task, theperformance improvement is calculated as improvementi =max oursi −max baselinei(positive for all tasks). Finally, themax baselinei
average improvement is calculated as avg improvement =1 i
N ∑N improvementi = 0.46858 ≈ 46.9%.

C

Realworld Task Details

C.1

Push-T

Demonstrations 136 demonstrations are collected
and used for training. The initial condition is varied byrandomly pushing or tossing the T block onto the table. Prior
to this data collection session, the operator has performed
this task for many hours and should be considered proficientat this task.

C.1.1

Evaluation We used a fixed training time of 12hours for each method, and selected the last checkpoint
for each, with the exception of IBC, where the checkpointwith minimum training set action prediction MSE error due
to IBC’s training stability issue. The difficulty of training
and checkpoint selection for IBC is demonstrated in main

C.1.2

text Fig. 7. Each method is evaluated for 20 episodes, allstarting from the same set of initial conditions. To ensure
the consistency of initial conditions, we carefully adjusted
the pose of the T block and the robot according to overlayed
images from the top-down camera. Each evaluation episodeis terminated by either keeping the end-effector within theend-zone for more than 0.5 second, or by reaching the 60
sec time limit. The IoU metric is directly computed in the
top-down camera pixel space.

C.2

Sauce Pouring and Spreading

Demonstrations 50 demonstrations are collected,
and 90% are used for training for each task. For pouring,
initial locations of the pizza dough and sauce bowl are varied.
After each demonstration, sauce is poured back into thebowl, and the dough is wiped clean. For spreading, locationof the pizza dough as well as the poured sauce shape arevaried. For resetting, we manually gather sauce towards
the center of the dough, and wipe the remaining doughclean. The rotational components for tele-op commands are
discarded during spreading and sauce transferring to avoid
accidentally scooping or spilling sauce.
C.2.1

Evaluation Both Diffusion Policy and LSTM-GMM
are trained for 1000 epochs. The last checkpoint is used forevaluation.
Each method is evaluated from the same set of random
initial conditions, where positions of the pizza dough and
C.2.2

18

sauce bowl are varied. We use a similar protocol as in PushT to set up initial conditions. We do not try to match initial
shape of poured sauce for spreading. Instead, we make surethe amount of sauce is fixed during all experiments.
The evaluation episodes are terminated by moving thespoon upward (away form the dough) for 0.5 seconds, orwhen the operator deems the policy’s behavior is unsafe.
The coverage metric is computed by first projecting the
RGB image from both the left and right cameras onto
the table space through homography, then computing thecoverage in each projected image. The maximum coverage
between the left and right cameras is reported.

D

Realworld Setup Details

UR5 robot station Experiments for the Push-T task
are performed on the UR5 robot station.
The UR5 robot accepts end-effector space positionalcommand at 125Hz, which is linearly interpolated from the
10Hz command from either human demonstration or thepolicy. The interpolation controller limits the end-effector
velocity to be below 0.43 m/s and its position to bewithin the region 1cm above the table for safety reason.
Position-controlled policies directly predicts the desired endeffector pose, while velocity-controlled policies predicts thedifference the current positional setpoint and the previous
setpoint.
The UR5 robot station has 5 realsense D415 depth camera
recording 720p RGB videos at 30fps. Only 2 of the cameras
are used for policy observation, which are down-sampled to320x240 at 10fps.
During demonstration, the operator teleoperates the robot
via a 3dconnexion SpaceMouse at 10Hz.

Figure 16. Initial states for Mat Unrolling

D.0.1

D.1

Franka Robot Station

Experiments for Sauce Pouring and Spreading, Bimanual
Egg Beater, Bimanual Mat Unrolling, and Bimanual
Shirt Folding tasks are performed on the Franka robot
station.
For the non-haptic control, a custom mid-level controller
is implemented to generate desired joint positions fromdesired end effector poses from the learned policies. At
each time step, we solve a differential kinematics problem(formulated as a Quadratic Program) to compute the desired
joint velocity to track the desired end effector velocity. The
resulting joint velocity is Euler integrated into joint position,
which is tracked by a joint-level controller on the robot.
This formulation allows us to impose constraints such as
collision avoidance for the two arms and the table, safety
region for end effector and joint limits. It also enables
regulating redundant DoF in the null space of the end
effector commands. This mid-level controller is particularly
valuable for safeguarding the learned policy during hardware
deployment.
For haptic teleoperation control, another custom mid-level
controller is implemented, but formulated as a pure torquecontroller. The controller is formulated using Operational
Space Control Khatib (1987) as a Quadratic Program
operating at 200 Hz, where position, velocity, and torquelimits are added as constraints, and the primary spatial
objective and secondary null-space posture objectives are

Figure 17. Final states for Mat Unrolling

posed as costs. This, coupled with a good model of theFranka Panda arm, including reflected rotor inertias, allows
us to perform good tracking with pure spatial feedback, andeven better tracking with feedforward spatial acceleration.
Collision avoidance has not yet been enabled for this control
mode.
Note that for inference, we use the non-haptic control.
Future work intends to simplify this control strategy and only
use a single controller for our given objectives.
The operator uses a SpaceMouse or VR controller input
device(s) to control the robot’s end effector(s), and thegrippers are controlled by a trigger button on the respective
device. Tele-op and learned policies run at 10Hz, and the
mid-level controller runs around 1kHz. Desired end effector
pose commands are interpolated by the mid-level controller.
This station has 2 realsense D415 RGBD camera streaming
VGA RGB images at 30fps, which are downsampled to320x240 at 10fps as input to the learned policies.

D.2

Initial and Final States of Bimanual Tasks

The following figures show the initial and final state of fourbimanual tasks. Green and red boxes indicate successful andfailed rollouts respectively. Since mat and shirt are very flat
objects, we used a homographic projection to better visualize
the initial and final states.

Diffusion Policy

Figure 18. Initial states for Egg Beater

Figure 19. Final states for Egg Beater

Figure 20. Initial states for Shirt Folding

Figure 21. Final states for Shirt Folding

19