Diffusion Imitation from Observation
Authors: Bo-Ruei Huang
Chun-Kai Yang
Chun-Mao Lai
Dai-Jie Wu
Shao-Hua Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Imitation from Observation
Learning from Observation
Imitation Learning
Diffusion Model
Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Learning from Observation (LfO) aims to imitate experts by learning from state-only demonstrations without requiring action labels. 
Existing adversarial imitation learning approaches learn a generator agent policy to produce state transitions that are indistinguishable to a discriminator that learns to classify agent and expert state transitions. Despite its simplicity in formulation, these methods are often sensitive to hyperparameters and brittle to train. Motivated by the recent success of diffusion models in generative modeling, we propose to integrate a diffusion model into the adversarial imitation learning from observation framework. Specifically, we employ a diffusion model to capture expert and agent transitions by generating the next state, given the current state. Then, we reformulate the learning objective to train the diffusion model as a binary classifier and use it to provide ``realness'' rewards for policy learning. Our proposed framework, Diffusion Imitation from Observation (DIFO), demonstrates superior performance in various continuous control domains, including navigation, locomotion, manipulation, and games.

Diffusion Imitation from Observation
Bo-Ruei Huang Chun-Kai Yang
Chun-Mao Lai Dai-Jie Wu Shao-Hua Sun
Department of Electrical Engineering, National Taiwan University
Abstract
Learning from observation (LfO) aims to imitate experts by learning from state- only demonstrations without requiring action labels. Existing adversarial imitation learning approaches learn generator agent policy to produce state transitions that are indistinguishable to a discriminator that learns to classify agent and expert state transitions. Despite its simplicity in formulation, these methods are often sensitive to hyperparameters and brittle to train. Motivated by the recent success of diffusion models in generative modeling, we propose integrate a diffusion model into the adversarial imitation learning from observation framework. Specifically, we employ diffusion model to capture expert and agent transitions by generating the next state, given the current state. Then, reformulate the learning objective to train the diffusion model as a binary classifier and use it to provide "realness" rewards for policy learning. Our proposed framework, Diffusion Imitation from Observation (DIFO), demonstrates superior performance various continuous control domains, including navigation, locomotion. manipulation, and games. Project page: https://tturrbbtteeamnngg////// github io/DIFO
Introduction
Learning from demonstration (LfD) 26
x% 86 aims to acquire policies that can perform desired skills by imitating expert trajectories represented as sequences of state-action pairs, eliminat- ing the necessity of reward functions. Recent advancements LiD have enabled the deployment of reliable and robust learned policies in various domains, such as robot learning [17 20 28 30)
LfD's dependence on accurately labeled actions remains substantial limitation, particularly in scenarios where obtaining expert actions is challenging or costly. Moreover, most LiD methods assume that the demonstrator and imitator share the same embodiment, inherently preventing coos-embbddiment imitation.
strategy games 22
and self-driving 7
To address these issues, learning from observation (LIO) methods [66] 76
seek imitate experts from state-only sequences, thereby removing the need for action labels and allowing learning from experts with different embodiments. Schmidt and Jiang [66], Torabi et al. (75). Yang
35
[82
proposed learning inverse dynamic models (IDMs) that can infer action labels from state sequences and subsequently reformulate LfO as LfD. Nevertheless, acquiring sufficiently aligned data with the expert's s data distribution to train IDMs remains an unresolved challenge. On the other hand, adversarial imitation learning (AII.) 32 76 &T employs generator policy learning to imitate an expert, while a discriminator differentiates between the data produced by the policy and the actual expert data. Despite its simplicity in formulation, AlL methods can be brittle to learn and are often sensitive to hyperparameters 13
Recent works have explored leveraging diffusion models' ability generative modeling and achieved encouraging results in imitation learning and planning [29 31 44 For example, diffusion policies [6]
49 learn to denoise actions with injected noises conditioned on states, allowing for modeling multimodal expert behaviors. Moreover, Chen et al. [5] proposed to model expert state-action pairs with a diffusion model and then provide gradients to train a behavioral cloning policy to improve
Correspondence Shao-Hua Sun <shaohuas @tt..edu.ww.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

its generalizability. Nevertheless, these works require action labels, fundamentally limiting their applicability learning from observation.
In this work, we introduce Diffusion Imitation from Observation (DIFO), novel adversarial imitation learning from observation method that employs a diffusion model as discriminator to provide rewards for policy learning. Specifically, we design a diffusion model that learns to capture expert and agent state transitions by generating the subsequent state conditioning on the current state. We reformulate the denoising objective of diffusion models as binary classification task, allowing for the diffusion model to distinguish expert and agent transitions. Then, provided with the "realness" rewards from the diffusion model, the policy imitates the expert by producing transitions that look indistinguishable from expert transitions.
We compare our method DIFO to various existing LfO methods in various continuous control domains, including navigation, locomotion, manipulation, and games. The experimental results show that DIFO consistently exhibits superior performance. Moreover, DIFO demonstrates better data efficiency. The visualized learned reward function and generated state distributions verify the effectiveness of our proposed learning objective for the diffusion model.
2 Related work
Learning from demonstration (LfD). LfD approaches imitate experts from collected demonstrations, consisting of state and action sequences. Behavioral cloning (BC) [5]] 70 formulates LfD as a supervised learning problem by learning state-to-action mapping. Inverse reinforcement learning (IRL) 1 45 60) extracts a reward function from demonstrations and uses to learn policy through reinforcement learning. In contrast, this work aims to learn from state-only demonstrations, requiring no action label.
Learning from observation (LfO). LIO I 73 learns from state-only demonstrations, i.e., state sequences, making suitable for scenarios where action labels are unobservable or costly to obtain, and allowing for learning from experts with a different embodiment. To tackle LfO, one popular direction is learn an inverse dynamics model (IDM) for an agent that can recover an action for pair of consecutive states [66] 75 82). However, there is no apparent mechanism to efficiently collect tuples of state, next state, and action that align with the expert state sequences, which makes it difficult to learn good IDM. On the other hand, adversarial imitation learning from observation (AILfO) 23 37 76 resemble the idea of generative adversarial networks (GANs) [19], where an agent generator policy is rewarded by discriminator learning distinguish the expert state transitions from the agent state transitions. Despite the encouraging results, the AILfO trainings are often brittle and sensitive to hyperparameters [2] 13). Recent works also use generative models to predict state transitions and use the prediction to guide policy learning using tog-likelihood [2]. ELBO (84, or conditional entropy (25). However, these methods depend highly the accuracy of the generative models. In contrast, our work aims to improve the sample efficiency and robustness of AILfO by employing diffusion model as a discriminator.
Learning from video (LfV). Extending from LfO, LIV specifically considers learning from image- based states, i.e., videos, by leveraging recent advancements in computer vision, e.g., multi-view learning (69] image and video comprehension and generation [3] 12
16
33
43
65, (68. foundation
models 42), and optical flow and tracking 31 [0) Yet. these methods are mostly specifically
designed for learning from videos, and cannot be trivially adapted for vectorized states.
Diffusion models. Diffusion models are state of tht--thean art generative models capable of capturing and generating digh-dimensional data distributions 27772 Diffusion models have been widely
adopted for generating images [56] 61], videos [4. 3D structures [52], and speech [4] 53). Recent
works also have explored using the ability to model multimodal distributions diffusion models for
LMD [5 6 35 49 79], where expert demonstrations could exhibit significant variability [39 Our
work aims to employ the capability of diffusion models for improving AIRLfO.
3 Preliminary
3.1 Learning from observation
Consider environments represented as a Markov decision process (MDP) defined as tuple AS, P, P, yo. 7) of state space S, action space A, reward function r(s, a, s'), transition dynamics

---

P(s'|s, a), initial state distribution Po and discounting factor y. We define policy IT ((ass) that takes actions from state inputs and generates trajectories 7 = (sp, ao. , $/r)). The policy trained to maximize the sum of discounted rewards E (101)......I1))>> E y'r (s,, a, 81+1)
In imitation learning, the environment rewards cannot be observed. Instead, a set of expert demonstra- tions TE = [To, ........... TN [T ~ πæ} is given, which generated by unknown expert policy *E- We aim to learn the agent policy TA to generate a similar trajectory distribution with expert demonstrations. Moreover, in the learning from observation (LfO) setting, where expert action labels are absent, agents learn exclusively from state-only observations represented by sequences (50, $1 $1/)). We use the LiO setting in this work.
Inverse reinforcement learning (IRL). One of the general approaches to imitation learning is IRL. This approach learns reward function r from transitions, i.e., (s, a) in LfD or s') in LfO, that maximizes the reward of expert transitions and minimizes that of agent transitions. The learned reward function can thereby be used for reinforcement learning to train the policy to imitate expert.
3.2 Denoising Diffusion Probabilistic Models
Diffusion models have emerged as sfate-of-the-arr generative models capable of producing high- dimensional data and modeling multimodal distributions. Our work leverages the Denoising Diffusion Probabilistic Model (DDPM) 27 latent variable model that generates data through a denoising process. The training procedure of the diffusion model consists of forward and reverse processes. In the forward process, Gaussian noise is progressively added to the clean data, following a predefined noise schedule. The process is formulated as e is the Gaussian noise, denotes the time step within the whole process with step T and de is the scheduled noise level at the current time step. Conversely, the reverse process, denoted by Po (x [ (x,), is designed to reconstruct the original data by estimating the previously injected noise based on the given noise level. This is achieved by optimizing (.w(..)))) denotes the diffusion model.
varxo + /I dre, where Xo is the clean data,
||ee (x,, where
Approach
We propose Diffusion Imitation from Observation (DIFO), a novel learning from observation frame- work integrating a diffusion model into the AIL framework, which is illustrated in Figure Specif- ically, we utilize a diffusion model to model expert and agent state transitions; then, we learn an agent policy to imitate the expert via reinforcement learning by using the diffusion model to provide rewards based on how "real" agent state transitions are.
4.1 Modeling expert transitions via diffusion model
Motivated by the recent success in using diffusion models for generative modeling, we use a condi- tional diffusson model to model expert state transitions. Specifically, given a state transition (s, s'), the diffusion model conditions on the current state and generates the next state s'. We adopt DDPM (27) and define the reverse process as (s t-1 (s'., s), where t E T and ø is the diffusion model, which is trained by minimizing the denoising MSE loss:
La(s,s') = -ETT,e--NO..) ||c | (sss, t|s)((,
where denotes the noise sampled from Gaussian distribution and es denotes the noise predicted by the diffusion model. Once the diffusion model is trained, we can generate an expert next state conditioned on any given state by going through the diffusion generation process.
State-distance reward. To train a policy sample an action from the policy and obtain the next state s by interacting with the environment. Next, we generate predicted next state s, using the diffusion model. Then, to bring the state distribution of the policy closer to the expert's, we can optimize the policy using reinforcement learning by setting the distance the two next states d(s (,)) as a reward, where denotes some distance function that evaluates how close two states are. However, good distance function varies
imitate the expert from given state s, we can first
3

---

5
Single-step
Learning Parameters
Frowen Parameters
Denoising
Learning, Objective
Inpuls
Uiffustio Model
Diflusion Discriminator d
TW
FINSE
Enstroument
Modell
(s.ss')
Reward log(2 (2(2.1
Diffusion Discriminator
Train
Alternately
(a) Learning diffusson discriminator
(b) learning policy with diffusion reward
Figure 1: Diffusion Imitation from Observation (DIFO). We propose Diffusion Imitation from Observation (DIFO), novel adversarial imitation learning from observation framework employing a conditional diffusion model. (a) Learning diffusion discriminator. In the discriminator step the diffusion model learns to model state transition (s, by conditioning on the current state s and generates the next state s'. With the additional condition on binary expert and agent labels (CF/CA), we construct the diffusion discriminator to distinguish expert and agent transitions by leveraging the single-step denoising loss as likelihood approximation.
(b) Learning policy with diffusion reward. In the policy step, we optimize the policy with reinforcement learning according to rewards calculated based on the diffusion discriminator's 's output log(1 (s,
from one domain another. Moreover, predicting the diffusion model next state s& can be very time-consuming since it requires 7 denoising steps.
Denoising reward. We aim to provide rewards for policy learning while avoiding choosing distance function and going through the diffusion generation process. To this end, we take inspiration from Li al. (38], which shows that the denoising loss approximates the evidence lower bound (ELBO) of the likelihood. Our key insight is to leverage the denoising loss calculated from a state and the policy next state (s,s, s ') Ca short, as an indicator of how well the policy next state fits the expert distribution. That said, low La means that the policy produces a next state close to the expert next state, while a high La means that the diffusion model does not recognize this policy next state. Hence, we can use - Ls as reward to learn a policy to imitate the expert by taking actions to produce next states that can be recognized by the diffusion model. Note that this denoising reward can be computed using single denoising step.
4.2 Diffusion model as a discriminator
The previous section describes how we can use the denoising loss reward for policy learning via reinforcement learning. However, the policy can learn to exploit frozen diffusion model by discovering states that lead a low denoising loss while being drastically different from expert states. To mitigate this issue, we incorporate principles from the AlL framework by training the diffusion model to recognize both the transitions from the expert and agent. To this end, we additionally condition the model on a binary label e E {cg. cA). where eg represents the expert label and represents the agent label, both implemented as one-hot encoding, resulting in the following denoising losses given state transition s')::):
C (s, =
||e co t,, <x)
(2)
23 (s, = EE-T,
||C (s' t|s ca)l
(3)
With this formulation and an optimized diffusion model, an expert transition should yield a low LJ and a high LJ. while an agent transition should yield a high f and a low L Thus, we construct a diffusion discriminator that determine if transition is close to expert follows:
s=(s.s)) (C (s.
(4)

---

where o is the sigmoid function for normalization and Ae is a hyperparameter to control the sensitivity. To turn this diffusion discriminator as binary classifier classify agent and expert transitions, we train it to optimize the binary cross entropy (BCE):
LBCE = ( [og(1 - DD(ss (a.s")~=T, (s,
(5)
By optimizing LBCE, online interactions with the agent are leveraged as negative samples. Given expert transitions, the model should minimize L and maximize f.. resulting higher score closer I. Conversely, when the input is sampled from the agent, the model aims maximize 2 and minimize C outputting lower score closer The higher the score is, the more likely a transition is expert. Hence, we can learn a policy to imitate the expert using D& as rewards. In contrast to MLP binary discriminators used in existing AIL works like GAIL, which maps sigh-dimensional inputs to a one-dimensional logit, our diffusion discriminator learns to predict high-dimensional noise patterns. This is inherently more challenging to overfit, addressing one of the key instabilities in GAIL.
4.3 Diffusion Imitation from Observation
Wc present Diffusion Imitation from Observation (DIFO) adversarial imitation learning from observation framework that trains policy and discriminator in turns. In the discriminator step, the discriminator learns to classify expert and agent transition by optimizing LUCE- Furthermore, to ensure the diffusion loss of expert data is optimized so that it approximates the ELBO, the diffusion model also optimizes L by sampling from expert demonstrations,
MMSE = EITT, (... [lle- co(s' =.(ss
(6)
resulting in the overall objective:
CD = AMSECMSE + ΛBCECBCE.
where AMSE and ÁBCE are hyperparameters adjusting the importance of each term. In the policy step, to provide the policy rewards based on the "realness" Do of the agent transitions, we adopt the GAIL reward function (24]:
Ts = log(1 (s
(8)
where Do is computed with single denoising step. We justify the feasibility of sampling only denoising step in Section 5.8 We can optimize the policy using any RL algorithm. The DIFO framework is illustrated in Figure and the algorithm is presented in Appendix
5
Experiments
5.1
Environments
In this section, we introduce environments, tasks, and how expert demonstrations are collected. All environment trajectories, except CARRACING, are fixed-horizon to prevent biased information about Further details can be found in Appendix B
success 32
POINTMAZE: A navigation task for a 2-DoF agent with the medium maze, see Figure 2a A point agent is trained navigate from an initial position to a goal. The goal and initial position of the agent are randomly sampled. The agent observes its position, velocity, and goal position. The agent applies linear forces in the and directions to navigate the maze and reach the goal. We collect 60 demonstrations (36 000 transitions) using controller from Fu et al. [14].
ANTMAZE: A task containing both locomotion and navigation, which presents significantly more challenging variant of the POINT ZE, as shown in Figure The quadruped ant learns to navigate from an initial position to goal by controlling the torque of its legs, where both the goal and initial position of the ball are also randomly sampled. Notice that this environment serves a nigh-dimensional state space task with 29 29-dimension state space. We use 100 demonstrations (7000 transitions) from Minari [50].
We experiment with optimizing LMSE with agent data (CF)) leading unstable training (see Appendix
5

---

(a) POINTMAZE
(b) ANTMAZE
(c) FETCHPUSH
(d) ADROTTDOOR
(001
(c) WALKER
(f) OPENMICROWAVE
(g) CARRACING
(h) CLOSE
Figure 2: Environments & tasks. (a) POINTMAZE: A point agent (green) is trained to navigate to the goal (red). (b) ANTMAZE: A nigh-dimensional locomotion navigation task for an 8-DoF quadruped ant to navigate to the goal (red). (c) FETCHPUSH: A manipulation task to move block (yellow) to the target (red). (d) ADROITDOOR: A high-dimension manipulation task to undo the latch and swing the door open. (e) WALKER: A locomotion task for 6-DoF hopper to maintain at the highest speed while keeping balance. (f) OPENMICROWAVE: A manipulation task to control the robot arm to open the microwave with joint space control. (g) CARRACING: An image-based task to control the car to complete the track in the shortest time. (h) CLOSEDRAWER: An image-based manipulation task control the robot arm to close the drawer.
FETCHPUSH:
The goal to control a 7-DoF Fetch robot arm to push block to a target position table, see Figure 2c Both the block and target positions randomly sampled. The robot is controlled by small displacements of the gripper in XYZ coordinates, which has a 28-dimension state space and 4-dimension action space. We generate 50 demonstrations (2500 transitions) using an expert policy trained by SAC [21].
ADROITDOOR: A manipulation task to undo the latch and swing the door open, Figure 2d The position of the door is randomly placed. It is based on the Adroit manipulation platform 34 with 39-dimension state space and 28-dimension action space containing all the joints. It serves as a high-dimensional state and action space task. We use 50 demonstrations (10 000 transitions) from the dataset released by Fu al. [14]
WALKER: A locomotion task of a 6-DoF Walker2D in MuJoCo 77], as shown in Figure 2e goal is to walk forward by applying torques on the six hinges. Initial joint states added with uniform noise. We generate 1000 transitions using an expert policy trained by SAC [11.
The
open the microwave door, as shown Figure 21 The environment has a 59-dimension state space and a 9-dimension continuous action space control the angular velocity of each joint. It serves high-dimensional state and action space task. We use 5 demonstrations (300 transitions) from the dataset released by Fu al. [14]
OPENMICROWAVE A manipulation task to control 9-DoF Franka robot
CARRACING: An imsge control task aimed at directing a car to complete a track as quickly possible. Observations consist of top-down frames, shown in Figure 2g Tracks are generated randomly in every episode. The car has continuous action space to control the throttle, steering, and breaking. We generate 340 transitions using expert policy trained by PPO 67].
CLoosEDRAWERT AWER: An image-based manipulation task from Meta- World to control Sawyer robot to close drawer. Observations consist of fixed perspective frames, as shown in Figure 2h The robot has continuous action space to control the gripper in XYZ coordinates, and the initial poses of the robot and the drawer randomized in every episode. We generate 100 transitions using a scripted policy.
requires the agent 83

---

Export -X-BL
-C-
+ WAILIO
1Q41ame +T
DIPO-Uhom!
(Oas)
116
139
Succes Ratte (%)
Success Ratt (7)
Success Rate (%)
Succoss Rate (7%)
60-
30%
600
1008
MM
2006 90%
7006 IM 1.M TTM
414
36
100
Endronment Steps
Environment Steps
Enviranmenn Stops
Emvironment Steps
(a) POINTMAZE
(b) ANTMAZE
(c) FETCHPUSH
(d) ADROITDOOR
Sucoess Rate: (%)
230
Sucoes Rate (%)
15
Return
5
141 Environment Stops
31
411
th
182x
Environment Staps
Environment Steps (g) CARRACING
Environment Steps
(c) WALKER
OPERWWWVE
CLOSEDRAWER
Figure 3: Learning performance and efficiency. We evaluate all the methods with five random seeds and report their success rates in POINTMAZE, ANTMAZE, FETCHPUSH, ADROITDOOR OPENMICROWAVE, and CLOSEDRAWER, and their returns in WALKER, and CARRACING. The standard deviation is shown as the shaded area. Our proposed method, DIFO, demonstrates more stable and faster learning performance compared to baselines.
5.2 Baselines and variants
We compare our method DIFO with the following baselines:
Behavioral Cloning (BC) [5 learns a state-to-action mapping using supervised learning without any interaction with the environment. Note that BC is the only baseline having privileged access to ground truth action labels.
Behavioral Cloning from Observation (BCO) 75 first learns an inverse dynamic model through self-supervised exploration, and uses it to reconstruct action from state-only observation. BCO then uses these action labels to perform behavioral cloning.
Generative Adversarial Imitation from Observation (GAIfO) (76. trains a GAIL MLP discriminator taking state transitions ss, as input, instead of state-action pairs (s, a).
Wasserstein Adversarial Imitation from Observation (WAIFO) is LíO variant of WAIL 8I taking (s, 8') as input. WAIL replaces the learning objective of the discriminator from Jensen- Shannon divergence (GAIL) to Wasserstein distance.
Adverserial Inverse Reinforcement Learning from Observation (AIRL1O) is a LfO variant of AIRL [13. AIRL modifies the discriminator output disentangle task-relevant information from transition dynamics. Similarly to GALfO, AIRLfO takes (s, as input instead of (s, a).
Decoupled Policy Optimization (DePO) 40 decouples the policy into high-level state planner and an inverse dynamics model, utilizing embedded decoupled policy gradient and generative adversarial training.
Inverse soft-Q Learning for Imitation (IQ-Learn) 151
directly leams policy in Q-space from demonstrations without explicit reward construction. We use the state-only setting for LfO.
Optimal Transport (OT) [48] derives proxy reward function for RL by measuring the distance between probability distributions. We use the state-only setting for LfO.
In addition the existing methods, also compare DIFO with its variants:
DIFO-Non-Adversarril (DIFO-NA) follows the method introduced in Section which first pretrains a conditional diffusion model on expert demonstrations, and simply takes the denoising reward -La(s, s') for policy training.
7

---

Export -X-
BOO
+ WAILIO
AIRLID
1Q41amn *
DIFO-Uham!
DIFO-NA
(0as)
106
129
Succes Ratte (%)
Success Rate (7)
Success Rate (%)
Succoss Rate (7h)
12
80-
00-
2M
IM
24
IM
26
3M
Emvronmenn Steps
Environment Steps
Emvironmnn
Steps
Environment Steps
(a) 200 Trajectories
(b) 100 Trajectories
(c) 50 Trajectories
(d) 25 Trajectories
Figure 4: Data efficiency. We vary the amount of available expert demonstrations in ANTMAZE. Our proposed method DIFO consistently outperforms other methods when the number of expert demonstrations decreases, highlighting the data efficiency of DIFO.
DIFO-Unconditioned (DIFO-Uncond) removes the condition on s, and denoises both and s'. is optimized only with LECE- Namely, replacing the MLP discriminator with a diffusion discriminator from GAIfO. It serves baseline showing the effect of network architecture.
5.3 Experimental results
We report the success rates in POINTMAZE, ANTMAZE, FETCHPUSSI, and ADROITDOOR, and return in WALKER and CARRACING of all the methods in Figure 3 Each method is reported with the mean value and standard deviation with five random seeds for all the tasks. BC's performance is shown as horizontal lines since BC does not leverage environmental interactions. The expert's perfor- mance (gray horizontal lines) in goal-directed tasks, i.e., POINTMAZE, ANTMAZE, FETCHPUSH. ADROITDOOR, 100%. More details of training and evaluation can be found in Appendix G
Our proposed method DIFO consistently outperforms or matches the performance of the best- performing baseline in all the tasks, highlighting the effectiveness of integrating conditional diffusion model into the AIL framework. In ANTMAZE, ADROITDOOR, and CARRACING, DIFO outperforms the baselines and converges significantly faster, indicating its efficiency in modeling expert behavior and providing effective rewards even in high-dimensional state and action spaces. Moreover, DIFO presents more stable training results, with relatively low variance compared to other AIL methods. Notably, although BC has access to action labels, it still fails in most tasks with more randomness. This is because BC relics solely on learning from the observed expert dataset, unlike the LfO methods that utilize online interaction with environments, BC is susceptible and requires substantial amount of expert data achieve coverage of the dataset. The result indicates the significance of online interactions. OT only successfully learns in environments like ADROITDOOR, WALKER, and CLOSEDRAWER AWER, where trajectory variety is limited. OT computes distances at the trajectory level rather than the transition level, which requires monotonic trajectories, making it struggle in tasks with diverse trajectories. In contrast, our method generates rewards at the transition level, allowing us to identify transition similarities even when facing substantial trajectory variability.
covariate shifts 36 588 [9]
Variants of DIFO, i.e., DIFO-Uncond, and DIFO-NA, perform poorly in most tasks. DIFO-NA Iearns poorly in most of the tasks except CLOSEDDRAWER, underscoring diffusion loss could be reasonable metric for the discriminator while is still necessary to model agent online interaction data to prevent the diffusion model from being exploited by the policy. On the other hand, DIFO-Uncond performs comparably to other AIL baselines but shows instability across different tasks, this highlighting the importance of modeling transitions using a diffusion model.
We also verify DIFO's capability to model stochastic distribution in Appendix D
5.4 Data efficiency
To investigate the data efficiency, i.e., how much expert data is required for learning, we vary the number of expert trajectories in ANTMAZE and report the performance of all the methods in Figure Specifically, we use 200, 100, 50, and 25 expert trajectories, each containing 000, 7000, 3500, and 1750 transitions, respectively.
8

---

10-
1.0
0.-
m
Generated nest state
W
0.8
W
0.8
M
0.5-
0.4
8.4
Nest 0.4
Nett
04
10
10
0.0
0.7
C4 06
0n
1.0
0.0
3.7
44
34 0h
1.
0.0
07
0.4
0.0
0.0
02
0.4
0.
State
State
State
State
(a) Expert (8, 8')
(b) Generated (s, 8')
(c) GATfO reward
(d) DIFO reward
Figure 6: Reward function visualization and generated distribution SINE. (a) The expert state transition distribution. (b) The state transition distribution generated by the DIFO diffusion model.
(c-d) The visualized reward functions learned by GAIfO and DIFO, respectively. DIFO produces smoother rewards outside of the expert distribution, allowing for facilitating policy learning.
The results demonstrate that DIFO learns faster compared to all the baselines with various amounts of demonstrations, highlighting its sample efficiency. Specifically, the number of demonstrations decreases from 200 to 50, DIFO's performance drops modestly from an 80% success rate to 70%, whereas WAILFO, the best-performing baseline when given 200 expert trajectories. experiences a substantial decline to 20% success rate. Furthermore, when the number of demonstrations is reduced to 25, all other baselines fail learn, with success rates nearing zero. In contrast, DIFO maintains success rate of around 20%, underscoring its superior data efficiency. This data efficiency highlights DIFO's ' potential for real- world applications, where collecting expert demonstrations can be costly.
5.5 Generating data using diffusion models
To investigate whether the DIFO diffusion model can closely capture the expert distribution, generate trajectories with the diffusion model in POINTMAZE, Specifically, we take a trained diffusion discriminator of DIFO and autoregressively generate sequence of next states starting from an initial state sampled in the expert dataset. We visualize four pairs of expert trajectories and the corresponding generated trajectories in Figure
The results show that our diffusion model can accurately gen- erate trajectories similar to those of the expert. It is worth noting that the diffusion model can generate trajectories that differ from the expert trajectories while still completing the task, such as the example on the bottom right of Figure 5 where the diffusion model produces even shorter trajectories than the scripted expert policy. Additional expert trajectories and the corresponding generated trajectories are presented in Appendix E
Figure Generated trajectories under POINTMAZE. The green point marks the initial state. The red point marks the goal. The blue trace represents the generated trajectory and the orange trace represents the corresponding expert trajectory.
5.6 Visualized learned reward functions
We aim to visualize and analyze the reward functions learned by DIFO. To this end, we introduce toy environment SINE which both the state and action space are I-dimension with range [o, 1]. We generate expert state-only demonstrations by sampling from the distribution s' sin GTT + (0, 0.05²). The sampled expert state transitions are plotted in Figure 6a
Reconstructed distribution. Given the expert state distribution, we generate a distribution of next states using the diffusion model of DIFO and visualize the distribution in Figure [b The generated distribution closely resembles the expert distribution, which again verifies the modeling capability of the conditional diffusion model.
Visualized learned reward functions. visualize the reward functions learned by GAIfO and respectively. The result shows that the reward function
DIFO in SINE in Figure 6c and Figure 6d

---

(130) ++ (1.10") 7 (1.10" (1.10 (0L1)
Number Samples
I0e
Suceess Ratt (1g
Surcess Rate (%)
20-
Return
Return
.664 626 2004
206
1M
Environment Steps
Environment Steps
Environment Steps
Environment Steps
POINTMAZE
(b) WALKER
(a) POZNTMAZE
WALKER
Figure 7: The effect of AMSE and ABCE- We Figure 8: Different numbers of denoising step vary the values of AMSE and ΛBCE in POINTMAZE samples for reward computation. We vary the and WALKER, showcasing DIFO's s robustness to number of denoising step samples to compute re- hyperparameters and emphasizing the importance wards. The result indicates the number of samples of both LECE and LMSE-
does not significantly affect the performance.
learned by GAIfO drops dramatically once it deviates from expert distribution, while that of DIFO presents smoother contour the region outside the distribution, which allows for bringing learning agent closer to the expert even when agent's behaviors are far from the expert behavior.
5.7 Ablation study on AMSE and ABCE
We hypothesize that both LMSE and LECF important for efficiency learning. To examine the effect of LMSE and LECE and verify the hypothesis, we vary the ratio of AMSe and λBCE in POINTMAZE and WALKER, including LBE only and LMSE only, i.e. AMSE = 0 and λACE 0. As shown in Figure 7 the results emphasize the significance of introducing both LMSE and LBCE, since they enable the model to simultaneously model expert behavior (CMSE) and perform binary classification (CRCe). Without LMSE, the performance slightly decreases as it does not modeling expert behaviors. Without LECE, the model fails learn as it does not utilize negative samples, i.e., agent data. Moreover, when we vary the ratio of AMSE and ÂBCE, DIFO maintains stable performance, demonstrating DIFO is relatively insensitive to hyperparameter variations.
5.8 Ablation study on the number of samples for reward computation
To investigate the robustness of our rewards, we conducted experiments with varying numbers of denoising step samples in POINTMAZE and WALKER. We take the mean of losses computed from different numbers of samples, i.e., multiple t, to compute rewards. As presented in the Figure 8 the performance of DIFO is stable under different numbers of samples. As result, we use a single denoising step sample to compute the reward for the best efficiency. We also investigate the stability of rewards under different numbers of samples Appendix F
6 Conclusion
We present Diffusion Imitation from Observation (DIFO), novel adversarial imitation learning from observation framework. DIFO leverages conditional diffusion model as a discriminator to distinguish expert state transitions from those of the agent, while the agent policy learns to produce state transitions that indistinguishable from the expert's for the diffusion discriminator. Experimental results demonstrate that DIFO outperforms existing learning from observation methods, including BCO, GAIfO, WAIFO, AIRLFO, DePO, OT, and IQ-Learn, across continuous control tasks in various domains, including navigation, manipulation, locomotion, and mmage-bseed games. The visualization of the reward function learned by DIFO shows that it can generalize well to states unseen from expert state transitions by utilizing agent data. Also, the DIFO diffusion model is able to accurately capture expert state transitions and can generate predicted trajectories that are similar to those of expert's.
10

---

Acknowledgement
This work was supported by the National Science and Technology Council, Taiwan (113-2222-E-002 007-). Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry of Education, Taiwan.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
[2] Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Chris Pal, and Derek Nowrouzezahraii Adversarial soft advantage fitting: Imitation learning without policy optimization. In Neural Information Processing Systems, 2020.
[3] Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh. Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline rl from internet videos via value- function pre-training. International Conference on Robotics and Automation, 2023.
[4] Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv:2311.15122 2023.
[5] Shang-Fu Chen, Hsiang-Chin Chun Wang, Ming-Hao Hsu, Cau Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.
[6] Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
[7] Seongjin Choi, Jiwon Kim, and Hwasoo Yeo. Trajgail: Generating urban vehicle trajectories using generative adversarial imitation learning. Transportation Research Part C: Emerging Technologies, 2021.
[8] Daniel Coelho, Miguel Oliveira, and Vitor Santos. Rifold: Reinforcement learning from online demonstrations in urban autonomous driving. In Association for the Advancement of Artificial Intelligence, 2024.
[9] Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In International Conference on Robotics and Automation, 2024.
[10] Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2023.
[11] Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation, 2017.
[12] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Neural Information Processing Systems, 2023.
[13] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[14] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv:2004.0722 2020.
[15] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Neural Information Processing Systems, 2021.

---

[16] Dibya Ghosh, Chethan Anand Bhateja. and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Learning Representations. 2023.
[17] Alessandro Giusti, Jérôme Guzzi, Dan C. Cireşan, Fang-Lin He, Juan Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, Davide Scaramuzza, and Luca M. Gambardella. A machine learning approach to visual perception of forest trails for mobile robots. JEEE Robotics and Automation Letters, 2016.
[18] Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell, imitation: Clean imitation learning implementations. arXiv:2211.1197. 2022.
[19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 2014.
[20] Abhishek Gupta, Clemens Eppner, Sergey Levine, and Pieter Abbeel. Learning dexterous manipulation for soft robotic hand from human demonstrations. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
[22] Jack Harmer, Linus Gisslén, Jorge del Val, Henrik Holst, Joakim Bergdahl, Tom Olsson, Kristoffer Sjöö, and Magnus Nordin. Imitation learning with concurrent actions in 3d games. In IEEE Conference on Computational Intelligence and Games, 2018.
[23] Peter Henderson, Wei-Di Chang. Pierre-Lue Bacon, David Meger, Joelle Pincau, and Doina Precup. Optiongan:
Learning joint reward- polucyypppyyyyccccoooo options using generative adversarial inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence, 2018.
[24] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
[25] Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. European Conference on Computer Vision, 2024.
[26] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:
A survey of learning methods. ACM Computing Surveys, 2017.
[27] A Jain Ho. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[28] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lyneh, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, 2022.
[29] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
[30] Edward Johns. Coarse-to- fine imitation learning: Robot manipulation from a single demonstra- tion. In International Conference on Robotics and Automation, 2021.
[31] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. International Conference on Learning Representattons, 2024.
[32] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor.coo adversarial imitation learning. In International Conference on Learning Representations, 2019.
Addressing sample inefficiency and reward bias in
[33] Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline enables learning new tasks from a handful of trials. In Robotics: Science and Systems, 2022.
12

---

[34] Vikash Kumar. Manipulators and Manipulation in high dimensional spaces. PhD thesis, University of Washington. Seattle, 2016.
[35] Chun-Mao Lai, Hsiang-Chus Wang, Ping-Chun Hsich, Yu-Chiang Frank Wang, Min-Hung Chen, and Shao-Hua Sun. Diffusion-reward adversarial imitation learning. 2xXi:24455.1614 2024.
[36] Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsich, Jeffrey Mahler, Florian T. Pokorny, Anca D. Dragan, and Ken Goldberg. Shiv: Reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces. In International Conference Robotics and Automation, 2016.
[37] Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J. Lim. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems, 2021.
[38] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In International Conference on Computer Vision, 2023.
[39] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Neural Information Processing Systems, 2017.
[40] Minghuan Liu, Zhengbang Zhu, Yuzheng Zhuang, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization. In International Conference on Machine Leurning, 2022.
[41] Songxiang Liu, Dan Su, and Dong Yu. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv:2201. 2022.
[42] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, 2023.
[43] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training, International Conference on Learning Representations, 2023.
[44] Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, 2023.
[45] Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
[46] Takayuki Osa, Joni Pajarinen, Gerhard Neumann. J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 2018.
[47] Ricardo Palma, Antonio A. Sánchez-Ruiz, Marco Antonio Gómez-Martín, Pedro Pablo Gómez- Martín, and Pedro Antonio González-Calero. Combining expert knowledge and learning from demonstration in real-time strategy games. In Base-Based Reasoning Research and Development, 2011.
[48] Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2022.
[49] Tim Pearce. Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In International Conference on Learning Representations, 2023.
[50] Rodrigo Perez- Vicente, Omar Younis, John Balis, and Alex Davey. Minari: A dataset api for offline reinforcement learning, 2023.
13

---

[51] Dean A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 1991.
[52] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. In International Conference on Learning Representations, 2023.
[53] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, 2021.
[54] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation learning using variational models. Neural Information Processing Systems, 2021.
[55] Antonin Rafiin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
[56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[57] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention, 2015.
[58] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics, 2010.
[59] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[60] Stuart Russell. Learning agents for uncertain environments. In Anmual Conference on Computa- tional Learning Theory, 1998.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Neural Information Processing Systems, 2022.
[62] Tanmay Vilas Samak, Chinmay Vilas Samak, and Sivanathan Kandhasamy, Robust behavioral cloning for autonomous vehicles using end-to-end imitation learning. SAE International Journal of Connected and Automated Vehicles, 2021.
[63] Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, 1997.
[64] Oliver Scheel, Luca Bergamini, Maciej Wolczyk. Blažej Osiński, and Peter Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Conference on Robot Learning, 2022.
[65] Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Rein- forcement learning with videos: Combining offline observations with interaction. In Conference on Robot Learning, 2020.
[66] Dominik Schmidt and Minqi Jiang. Learning to act without actions. In International Conference on Learning Representations, 2024.
[67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms.
[68] Younggyo Seo, Kimin Lee, Stephen James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, 2022.
14

---

[69] Pierre Sermanet, Corey Lynch. Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. time-contrastive networks: Self-supervised learning from video. In International Conference on Robotics and Automation, 2018.
[70] Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone. In Neural Information Processing Systems, 2022.
[71] Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In International Conference on HHmman-Robu Interaction, 2022.
[72] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[73] Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. In International Conference on Machine Learning. 2018.
[74] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference On Intelligent Robots and Systems, 2012.
[75] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, 2018.
[76] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. In International Conference on Machine Learning, 2019.
[77] Mark Towers, Jordan Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, 2023.
[78] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models, 2022.
[79] Bingzheng Wang, Guoqiang Wu, Teng Pang, Yan Zhang, and Yilong Yin. Diffail: Diffusion adversarial imitation learning. In Association for the Advancement of Artificial Intelligence, 2024.
[80] Chuan Wen, Xingyu Lin, John So. Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. In Robotics: Science and Systems, 2023.
[81] Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv: 00.0000.8113. 11,, 2019.
[82] Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree- ment. In Neural Information Processing Systems, 2019.
[83] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning. 2020.
[84] Xingyuan Zhang, Philip Becker-Fhmmck, Patrick van der Smagt, and Maximilian Karl. Action inference by maximising evidence: zero-shot imitation from observation with world models. In Neural Information Processing Systems, 2024.
[85] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. In Neural Information Processing Systems, 2020.
[86] Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence, 2008.
15

---

Appendix
A Pseudocode of DIFO
Algorithm Diffusion Imitation from Observation (DIFO)
1: Input: Expert demonstrations TE ~ TE, policy ил and diffusion model De
2: while æ not converges do
3: Sample agent transitions Tù ~ TI by interacting with the environment
4: Calculate discriminator loss LD with Eq.
5: Update discriminator parameters & with VLD
6: Calculate IRL reward r (s, with Eq.
7: Update TA with respect to To with any RL algorithm
8: end while
B Environment & task details
The environments used for our experiments in Section 5 are from Gymnasium [10 77 We list
names, version numbers, dimensions of state and action spaces in Table All environments we used continuous action spaces.
Table Enviroments. Detailed setting of each Task.
Task
ID
Observation space
Action space
Fixed horizon
POINTMAZE
DointMazeMMedinee
2
True
ANTMAZE
AutMaze UMaze-v4
31
8
True
FETCHPUSH
FetchPush-v2
31
4
True
ADROITDOOR
AdroittanndDorrCutttomm
28
True
WALKER
Walker2d-v4
True
OPENMICROWAVE
FrankaKitchen-vl
59
True
CARRACING
CarRacing-ve -2
96 x 96 x 3
3
False
CLOSEDRAWER
N/A
641 x 64 x 3
4
True
Notably, we fix the horizon to prevent biased information about termination 32 in all environments except CARRACING since the goal of CARRACING is complete the track fast possible instead of goal completion. Notice that though the objective of WALKER is also not goal completion, the termination signal provides additional information about tumbling.
Preprocessing, In CARRACING, we preprocess the observation state by applying frame skipping with factor of 2, resizing, and converting the image to grayscale, resulting in 64 x 64 matrix. Finally, we stack frames to form the input state.
We list the details of the expert demonstration datasets we used in Section 5 and how we collect them in Table All our expert data incorporates stochasticity to enhance diversity in trajectories. We add a small amount of noise to the experts' actions, providing stochasticity and multimodality in expert behaviors.
Table 2: Expert observations. Detailed information on collected expert observations in each Task.
Task
of trajectories
transitions
Algorithm or Retrieved Source
POINTMAZE
60
360000
D4RL 14
ANTMAZE
100
7000
Minari 50
FETCHPUSH
2500
Own SAC expert
ADROITDOOR
50
10000
D4RL 14
WALKER
1000
Own SAC expert
OPENMICROWAVE
300
D4RL [4]
CARRACING
340
Own PPO expert
CLOSEDRAWER
100
Own Script expert
16

---

In Section 5.6 we introduce toy environment SINE. We generate 25 000 state-only transition (s, s'), with transition function:
= sin GTS + M/0, 0.05")
where M(u, o²) normal distribution noise with the mean value μ and the standard deviation 0.05.
C Optimizing LMSE with agent data
Our method optimizes LuSE (Eq. 6 to approximate the ELBO only using expert demonstrations. To investigate the effect of optimizing this MSE loss using agent data, experiment with optimizing LMSE with and without agent data on all tasks. The results are reported in Figure 9 We found that optimizing MMS with agent data can lead to slower and unstable convergence, especially in tasks with larger state and action spaces, e.g., ADROITDOOR, where optimizing LSSE leads to a 0% success rate. We hypothesize that optimizing LMSE leads to unstable training because, during the early stage of training. the agent policy changes frequently and generates diverse transitions. This diversity leads consistent distribution shift for the diffusion model, making the diffusion model unstable to learn. As a result, the overall performance can be less stable. Hence, our method designed to only optimize LMSE using expert demonstrations.
Success Rate (%)
Swccsss Rate (%S
Succers Rate (%)
Succes Rase (%)
ageer (Dhrr)
Steps
Environment Steps
Emironment Steps
Steps
(a) POINTMAZE
(b) ANTMAZE
(c) FETCHPUSH
(d) ADROITDOOR
120
Success Fatt ('))
Succoss Rata (%)
Rature
300
200
15
144
24
2/4
4M
18
2M
244
2/1
SM
12
244
1M
1059
2.06
200
426
50%
Environment Steps
Environment
Steps
Environment Steps
Emvironment
Steps
(e) WALKER
(I) OPEN MICROWAVE
(g) CARRACING
(h) CLOSEDRAWER
Figure 9: Optimizing MAS with and without agent data. We evaluate optimizing LMSE with and without agent data in all tasks. The results show that optimizing LUSE with agent data (red) leads to slower convergence and less stable performance.
100
D Stochastic environment
Success Rate (%)
We create stochastic version of ANTMAZE environment where Gaussian noise is added to the agent's 's actions before they are applied in the environment. The magnitude of the noise is 0.5, resulting in the actual action taken in the environment would be action action + 0.5A(O, 1). Given the action space this environment is [-1, this represents high level of stochasticity.
20
Stodestte Detewwekiik
5004
1dkt
2dn
Environment Steps
Figure 10: Stochastic environment. We apply addition Gaussian noises to ac- tions to create stochastic ANTMAZE. Our method DIFO maintains robust per- formance under large stochasticity.
Figure 10 shows that the performance of our method re- mains robust even under such high stochasticity, indicating our model's s ability adapt to stochastic environments ef- fectively.
17

---

E Full trajectories generations of POINTMAZE
More results of the POINTMAZE trajectory generation experiment in Section 5.5 can be found in Figure 11
Figure 11: Full result of generated trajectories under POINTMAZE. The blue trace represents the generated trajectory and the orange trace represents the corresponding expert trajectory, DIFO can accurately model expert behavior and generate successful trajectories.
F
The stability of rewards
To further verify the stability of the rewards under different numbers of denoising step samples, we present the standard deviation mean ratio of the rewards from 500 computations results in Table 3 The values are averaged from batch 4096 transitions. The result shows that sampling a single denoising step is enough to produce stable reward.
Table 3: Reward ratio. Standard deviation to mean rewards ratio over 500 computations, averaged from 4096 transitions. is the number of denoising step samples compute the reward.
Learning Progress
1
Y
2
5
10
20%
0.323
0.237
0.294
0.246
40%
0.234
0.199
0.206
0.230
60%
0.201
0.175
0.157
0.206
80%
0.157
0.152
0.150
0.190
100%
0.142
0.133
0.145
0.161
G
Training details
Our codebase inherits from Imitation [18]
open-source imitation learning framework based on Stable Baselines3 [55]. We then describe the implementation of each baseline:
GAIfO, AIRLfO, and BC follow the original implementation of Imitation.
IQ-Learn and OT are modified from SAC in Stable Baselines3 with the loss function borrow from the official implementations
We implemented BCO, WAIFO, and DePO.
18

---

Table 4: Hyperparameters. The overview of the hyperparameters used for all the methods in every task. We abbreviate "Discriminator" as "Disc." in this table.
Methed
Hygerparanetee
PoINTMaze
ANEMaze
PEscHPPPH
ADsOr/Docs
WALKIM
CARRACIN
Algusilm
SAC
SAC
P9O
Butch Size
128
128
128
125
128
Leaming
0/0011
0/0009
01000
0.0001
0.0001
0.000]
Trailing
0000000
500000
0U0003
0,0003
0,0003
DU0001
800
Estropy Wesgght
1.2 Waight
0.0001
0:001
0.0001
0.001
0,0001
0.001
0.0001
0.001
0.0001
0.001
0L0001
0.001
0,01
0.2
1,0
5.0
1.0
5.0
Enweroment Steps
00000200
500/000
100000000
00000000
30003000
Butch Size
64
64
64
Dis. Learzing Kate
0LEG01
0100011
0.0001
0L001
0000001
GAUIO
Hidken Dimcasions
Layers
125
64
Dic: Lowrning
0.0001
0.00013
0.0001
0.000
00000011
Dic. Buffer Size
0000
DOD0000
1000000
0000000
1 1000000
WATKO
Disc Hiddzn Dimessions
Dive. Hisklai Layers
128
124
125
128
128
128
Regulattze Epoilon
0.001
B1001
01001
0.001
00001
0.001
Regalanze Finction
Clippins
1
0,0003
0.0001
0.00001
0.0001
0.00001
AIRLIO
Disz Hudlen Dimeecions
Dac. Bafier Stzn
10000000
176
1000-00
1.28
0000
176
1000000
178
0:00000
128
10000000
178
Disc. Hidden Layers
Enwivomenn Steps
0000
3 C0000
500000
100000000
5 00000000
30000000
Butch Size
66
64
66
DePU
IDM, Basch Size
Lomnnng
0,0003
256
0.0003
256
0.0000
256
255
Envinommni Step-
10000000
0000000
1.000000 000
100000000
0000 300
256
256
256
Leaming
3x3005
Cálie Leaming
3x.10
Entmpy
10000000
D000000
1.600000
10000000
0000
00000000
Hatch
256
Rewand Scate
100
100
100
OT
Expest Sumples
10
10
10
10
10
10
caming Rair.
0,0003
0.00013
0.0009
0.000
0.0003
0.0001
Envireennni Steps
10000000
3 0000000
5000 000
100000000
5000000
50000000
Butch Size
64
64
64
Div: Leariing
0.0001
0.00011
0.000
0.0001
0U0001
Disc.
L000000
1000000
1000300
Jace
Weight A.
(Ous)
U-Net Units
(756, 200)
(256. 256,
256 2560
(15),
(256, 265
10
(756, 206)
Embeedinng Dimmnoon
126
1.58
196
92
Denoiiing Sunple Runge
(250. 7500
(250, 750)
(250) 750)
(250, 7500
(250, 750)
(250.
7500
Dermang Timesiop
1000
1000
1000
1000
1000
1000
Lagil Lagi Stale
10
10
10
10
10
10
Enwiromenn Steps
000000000
33 0000
10000000 000
100000000
5 000 2000
30000000)
Learting
0L00033
0.00011
000001
00000000
0000000
Weight. A.
10
10
10
L.-ee Units
(256, 256, 255)
(356, 256, 256)
(256. 256 1565
(256, 256, 256)
(256, 256 2.66
(256, 256, 256)
Enchelding Dimenson
128
124
128
129
128
128
Denosing Sumple Kange
(250) 7500
(250, 7503
(250) 7500
(250) 7500
(250) 750)
(250) 7550)
Dessiinn Tinesiop
1000
1000
1000
1000
1000
1000
Logie Scale Rovimemen Step-
00000000
3 00000000
0000000
100000000
5 00000000
30000000
Leorting
010001
Dise.
10000000
DOD000
1000000
LO00000
10000000
BCE
0.02
0.01 1
MSF Weiggt Juns A.
1
DIFO-NA
U. Units
512, 512, 512)
(512,
512,
10
(512, 512,, 512)
(112,
512,5122,222211 512)
10
(512, 1112.512. 512,512,,,1111111
10
(256, 256, 250)
10
Inchedding Dimenson Deneising Sumple Ringe Dermong Timestep Logii Ewiioonenn Steps
122%
250
2815
250
200
12%
(250, 7500
(250, 750)
(250. 750)
(250, 7500
(250) 750)
(250. 7500
1000
10001
1000
1000
1000)
1000
10
10
3100000
1:000000
50000000
19

---

G.1 Model architectures
Vector-based observation space. For tasks with vectorized state space, we implement a U-Net with MLP layers as our backbone of the diffusion model. The conditions are applied on every layer.
Image-based observation space. For tasks with image observations, we implemented a 2D U-Net as the backbone of the diffusion model with the diffiueess package provided by von Platen et al. 78.. which originally proposed by Ronneberger et al. [57]. The model contains down-sampling blocks and 3 up- sampling blocks. Each block consists of 2 convolution residual layers, with group normalization applied using 4 groups. The conditions are applied on every layer.
G.2 Hyperparameters
The hyperparameters of the policies and discriminators employed for all methods across all tasks are listed in Table 4 and We use Adam as the optimizer for all the experiments.
Table 5: SAC PPO training parameters.
Method
Hyperparameter
Learning Rate
0,0003
SAC
Batch
Discount Factoor y
0.99
Learning Rate
0.0001
Ratch Size
128
Discount Factor "y
GAE Clip
0.001
PPO
0.95
Value Function Coeflicient
0.5
Entropy Coefficient
Maximum gradient
Epochs
H
Computational resources
We used the workstations listed in Table Our method takes approximately 3 hours for each task. The shortest task POINTMAZE which takes around hour. The longest task is CARRACING which takes around 6 hours. As for reproducing all results including the baselines, it takes about 2000 GPU hours to run in series.
Table 6: Computational resources.
Workstation
CPU
GPU
RAM
Workstation
Intel Xeon w7-2475X
NVIDIA GeForee RTX 4090 x
125 GiB
Workstation 2
Intel Xcon w5-2455X
NVIDIA RTX A6000 x
125 GiB
Workstation
Intel Xeon W-2255
NVIDIA GeForce RTX 4070 Ti x 2
125 GiB
Workstation
Intel Xeon W-2255
NVIDIA GeForce RTX 4070 Ti x
125 GiB
I Limitations
This work presents novel learning from observation framework, DIFO. integrating a diffusion model into the AIL framework. Despite that DIFO achieves encouraging results in various domains, there are still some limitations. Firstly, our method takes state-only transitions (s, as the reward function, while the underlying optimal reward function could be in the form r(s, a, s'), where dynamics involved. This may lead to sub-optimal performance in tasks with delayed effects on actions. Secondly, our method assumes the state spaces of the agent and expert are identical as they share the same model, which limits cross-embodiment applications to some extent. Thirdly, our method highly relies on expert demonstrations, the presence of sub-optimal demonstrations may adversely impact the performance. Finally, due to the learning nature focused on discrimination, DIFO may not incorporate well with environmental rewards even if they accessible.
20

---

J Broader impact
Experimental results demonstrate that DIFO exhibits data efficiency, generalizability, and resistance to noisy environments, thereby enhancing its suitability for real-world applications. Learning from observation enables its deployment in scenarios where action data is costly or inaccessible.
However, our method inherits the nature of Adversarial Imitation Learning. One significant concern the potential negative impact safety when deployed in real-world settings, the exploration process may lead to unsafe actions. Additionally, imitation learning may capture and reinforce bias present in expert demonstrations, causing trapping in sub-optimal behaviors. These issues highlight the necessity of further research focused on reducing these negative impacts.
21

---

NeurIPS paper checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction reflect our paper's contributions and scope. We clearly claim that our novel adversarial imitation learning method, DIFO, integrates a diffusion model into the learning from observation framework. We acknowledge the limitations of existing methods and how DIFO addresses these challenges, aligning well with our paper's 's contributions and findings. Additionally, we demonstrate that DIFO has better data efficiency and performance various continuous control tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our method in Section I
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable. the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed implementation and hyperparameter information in Appendix G Additionally, we fix seeds in all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice,
the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to release code, expert datasets, and models recently.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
www.iiiissoooooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Sitips..ccpubbiee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have reported the experimental setting and details in Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported the error bars in every figure and discussed the statistical significance the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric emor bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the computing resources and computing time in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: hett://nnuuiisppe wwwprip...cppuulliiieeee htt.c//bbbcc/ticcccsiiiieeaaaaaa
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that this paper conforms the Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the broader impacts in Section !
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications. the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
needed point out
The authors should consider possible harms that could arise when the technology being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibilityy of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We gave credits and cited the diffusers package and existing imitation imple- mentations we built upon in Scction G.T
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pwpprswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question:
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations. and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "qUOdmQKaf5", "reviewer": "Reviewer_7W6h", "summary": "This paper leverages a diffusion model to learn a expert-state transition and additionally a discriminator that can differentiate expert and agent states. The paper conducts experiments on standard RL environments and demonstrate better performance and data efficiency.", "questions": "**Questions**\n- On page 8, line 298, the paper indicates that a \"smoother contour\" allows for bringing a learning agent closer to the expert. I was wondering if this property is only useful for specific environments but not in general. Dense reward may be helpful but often can misguide policy learning.\n\n**Possible typos**\n- Page 3, line 90: \"can't\" should be \"cannot\"", "limitations": "The approach leverages only expert data, it remains to extend this to setting with suboptimal demonstrations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple and its experimental results demonstrate strong performance and data efficiency\n- The paper is well written in general", "weaknesses": "- Under section 4.3, why do we only learn the MSE on the expert data but not also the agent data? My understanding is that both data will impact the discriminator output based on section 4.2.\n- Diffusion models are usually used for modelling multimodal distribution. I have two questions:\n\t-  Is the expert stochastic? Does the expert policy exhibit diverse behaviours?\n\t- Is it more convincing to have stochastic environments rather than purely deterministic environments?\n- The experimentation seems very limited for the amount of theory it provides---in particular there is no theory regarding whether the proposed objective is truly sound.\n- Tuning the $\\lambda$'s  seem difficult as they can vary by ~20% success rate according to figure 7. How should one address this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> the paper indicates that a \"smoother contour\" allows for bringing a learning agent closer to the expert. I was wondering if this property is only useful for specific environments but not in general. Dense reward may be helpful but often can misguide policy learning.\n\nWe thank the reviewer for the question. The objective of RL is to maximize the overall return. The RL algorithm should converge to the optimal policy as long as the reward function produces the highest reward at the optimal transition. A smoother contour helps guide the RL algorithm without misguiding the final policy. Our method's reward peaks align with expert distribution (see Figure 7(a) in the main paper), ensuring accurate guidance.\n\nMoreover, under the AIL framework, the reward function evolves alongside the policy. The reward function becomes sparser as the policy improves and eventually converges to a relatively sparse condition. Detailed theoretical guarantees can be found in the GAIL paper [1]. Besides, many previous imitation learning and inverse RL papers visualize the reward function with similar plots like ours [2, 3, 4, 5]. Hence, we believe that this approach is widely accepted for evaluating reward functions.\n\nLastly, we would like to highlight that we evaluate DIFO in diverse domains, including navigation, manipulation, locomotion, and control, in both state- and image-based environments. Our evaluation results demonstrate that DIFO is broadly applicable and not restricted to specific environments.\n\n> Why do we only learn the MSE on the expert data but not also the agent data?\n\nPlease see the overall response. \n\n> Is the expert stochastic? Does the expert policy exhibit diverse behaviors?\n\nAll of our expert data incorporates stochasticity to enhance diversity in trajectories. In the PointMaze environment, our expert trajectory data is generated using a breadth-first search planner with added stochasticity to create different path choices, resulting in various trajectories for the same start and end positions. In other environments, we add a small amount of noise to the experts' actions. These methods provide stochasticity and multimodality in expert behaviors. Our model effectively learns from such expert demonstrations, demonstrating its capability to handle multimodal distributions.\n\n> Diffusion models are usually used for modeling multimodal distribution.\n\nAs discussed in Section 5.5 and Section D of the main paper, we use a trained diffusion model to generate maze trajectories. As previously mentioned, our expert data exhibits a certain degree of multimodality, meaning that there are different paths in the dataset for the same start and goal locations. The generated results support our observations, as our model effectively produces multimodal possible paths. This demonstrates the benefit of modeling multimodal distributions using diffusion models.\n\n> Is it more convincing to have stochastic environments rather than purely deterministic environments?\n\nPlease see the overall response.\n\n> The experimentation seems very limited for the amount of theory it provides---in particular there is no theory regarding whether the proposed objective is truly sound.\n\nOur method builds upon the established theories of Diffusion Classifier [6] and GAIL [1], as described in Section 4. The Diffusion Classifier paper shows that the diffusion loss (Equations 2 and 3) approximates ELBO, effectively transforming a diffusion model into a classifier. GAIL proves that IRL is a dual problem of an occupancy measure matching problem. Consequently, running RL with the discriminator IRL reward, which recovers the primal optimum from the dual optimum, ensures that the resulting policy is the primal optimum. As a result, by adversarially training the discriminator and the policy, we can match the distribution of the agent's policy to that of the expert. By incorporating the diffusion model as the discriminator into the AIL framework, our method leverages the strengths of the diffusion model while maintaining optimality guarantees. We will revise the paper to include these details.\n\nWe believe our experiments are comprehensive. Please see the response to Reviewer AzJp. \n\n> Tuning the lambda seem difficult as they can vary by ~20% success rate according to figure 7. \n\nFigure 7 aims to investigate the effect of $\\lambda$, and therefore, we experimented with a very wide range of values. The performance only drops by 20% when the lambda is increased or decreased by two orders of magnitude from the optimal value we identify. In Figure 7(a), the performance of $\\lambda=0.1$ and $0.01$ is comparable, with overlapping standard deviations, indicating no statistically significant difference between them. Similarly, in Figure 7(b), although the performance appears to vary slightly, the standard deviations overlap. Therefore, we believe DIFO is not sensitive to $\\lambda$. Any $\\lambda$s in a reasonable range [0.1~0.001] can yield good performance.\n\n**References**\n \n[1] Ho et al. \"Generative adversarial imitation learning.\" In NIPS, 2016.\n\n[2] Garg et al. \"Iq-learn: Inverse soft-q learning for imitation.\" In NeurIPS, 2021.\n\n[3] Fu et al. \"Learning robust rewards with adversarial inverse reinforcement learning.\" In ICLR, 2018.\n\n[4] Ni et al. \"f-irl: Inverse reinforcement learning via state marginal matching.\" In CoRL, 2021.\n\n[5] Liu et al. \"Energy-based imitation learning.\" In AAMAS, 2021.\n\n[6] Li et al. \"Your diffusion model is secretly a zero-shot classifier.\" In ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EH9g6W5fcL", "reviewer": "Reviewer_AzJp", "summary": "This paper introduces Diffusion Imitation from Observation (DIFO), a novel approach to Imitation Learning from Observation (ILfO). DIFO innovates by employing a diffusion model as the discriminator, departing from the conventional feed-forward neural network approach. The authors leverage the connection between the diffusion model's training loss and the Evidence Lower Bound (ELBO) to construct a discriminator based on the relative likelihood of state-pairs originating from the expert rather than the agent. Experimental results across six environments demonstrate DIFO outperforming baseline methods.", "questions": "- There are other methods [1, 2, 3] that use metrics, i.e. likelihood, ELBO and entropy, from generative model for ILfO. I suggest discussing them in the related works.\n- How many samples of $t$ and $\\epsilon$ are needed to compute the discriminator score in eq. 4? Is the value stable across different samples?\n- Although you are motivated to improve the robustness of AILfO, I don't see a direct connection that using a diffusion model instead of an FFN should make things more stable, since in the end you are still running adversarial learning. Could you please elaborate on this?\n- In Figure 7, the lines of $(\\lambda_\\text{MSE}, \\lambda_\\text{BCE}) = (1, 10^{-2})$ and $(\\lambda_\\text{MSE}, \\lambda_\\text{BCE}) = (1, 10^{-1})$ are too similar to each other both in terms of colours and symbols and is really hard to distinguish. Could you at least change one of them to different colour or symbol?\n\nReference:\n\n[1] Escontrela *et al.*, Video Prediction Models as Rewards for Reinforcement Learning, NeurIPS 2023\n\n[2] Zhang *et al.*, Action Inference by Maximising Evidence: Zero-Shot Imitation from Observation with World Models, NeurIPS 2023\n\n[3] Huang *et al.,* Diffusion Reward: Learning Rewards via Conditional Video Diffusion", "limitations": "Yes, the author has discussed the limitations in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The utilization of diffusion models' loss as ELBO to build a discriminator presents a novel and potentially fruitful direction in the field of ILfO.", "weaknesses": "- While DIFO demonstrates sample efficiency in terms of expert demonstrations, it lacks efficiency in environment interactions.\n- The experimental scope is limited, with tests conducted on only six environments and comparisons made against relatively older baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> There are other methods [1, 2, 3] that use metrics, i.e. likelihood, ELBO and entropy, from generative model for ILfO. I suggest discussing them in the related works.\n\nWe thank the reviewer for providing the references. We will revise the paper to discuss these methods using metrics from generative models for ILfO.\n\n> How many samples of and are needed to compute the discriminator score in eq. 4? Is the value stable across different samples?\n\nWe sample a single denoising timestep (n=1) to compute the reward. As suggested by the reviewer, to investigate the robustness of our rewards with a varying number of timestep samples, we conducted experiments in PointMaze by averaging different numbers of timestep samples (n=1, 2, 5, 10) as rewards. The result presented in Figure R.3 in the PDF file attached to the rebuttal summary shows that our method can learn smoothly with a single sample.\n\nTo verify the stability of the rewards, we present the standard deviation to mean ratio of the rewards from 500 timesteps across different numbers of timestep samples in the table below. The values are averaged from a batch of 4096 transitions. The result below suggests that the reward is stable. \n\n|Learning Progress|n=1|n=2|n=5|n=10|\n|-|-|-|-|-|\n|20%|0.323|0.237|0.294|0.246|\n|40%|0.234|0.199|0.206|0.230|\n|60%|0.201|0.175|0.157|0.206|\n|80%|0.157|0.152|0.150|0.190|\n|100%|0.142|0.133|0.145|0.161|\n\n> Although you are motivated to improve the robustness of AILfO, I don't see a direct connection that using a diffusion model instead of an FFN should make things more stable\n\nWe hypothesize that the instability of most existing AIL frameworks arises from the tendency of FFN discriminators to overfit, resulting in the policy's inability to learn from discriminator rewards. Unlike FFN binary classifiers, which map a high-dimensional input to a one-dimensional logit, our diffusion model learns to predict high-dimensional noises, which is inherently more difficult to overfit.\n\nFurthermore, expert demonstrations typically exhibit significant variability, which could be difficult for FFNs to model effectively [1]. Diffusion models, however, are adept at handling such multimodality, providing a more robust approach for capturing the diverse patterns present in expert behavior. We will revise the paper to include these insights.\n\n> While DIFO demonstrates sample efficiency in terms of expert demonstrations, it lacks efficiency in environment interactions.\n\nWe respectfully disagree with the reviewer. Figure 3 in the main paper shows that our method, DIFO, is very efficient in environmental interactions, i.e., sample efficiency. In PointMaze, DIFO only requires about 175k environment steps to achieve a 60% success rate, while the best-performing baseline AIRLfO needs over 400k steps. Similarly, in AdroitDoor, DIFO achieves a 75% success rate with only 4M steps, yet the best-performing baseline WAILfO needs around 10M steps. That said, DIFO is very sample-efficient.\n\nThe table below presents the number of environment steps required for each method to achieve 50% expert performance. The result shows that our method, DIFO, can achieve the same performance with significantly fewer environmental steps (environment interactions). That said, DIFO is very sample-efficient.\n\n|Environment|Goal|DIFO (Ours)|BCO|GAILfO|WAILfO|AIRLfO|IQ-Learn|\n|-|-|-|-|-|-|-|-|\n|PointMaze|Success rate=50%|120k|100k|850k|x|310k|160k|\n|AntMaze|Success rate=50%|1.1M|x|x|x|x|x|\n|FetchPush|Success rate=50%|630k|x|510k|670k|x|x|\n|AdroitDoor|Success rate=50%|3.1M|x|x|4.9M|x|x|\n|Walker|Return=3000|1.0M|x|2.0M|1.7M|x|x|\n|CarRacing|Return=400|400k|x|1.1M|x|x|x|\n\n> The experimental scope is limited, with tests conducted on only six environments and comparisons made against relatively older baselines.\n\nWe conducted extensive experiments to evaluate our method across various aspects. \n- The six test environments we selected cover a wide range of domains, including navigation (PointMaze and AntMaze), manipulation (FetchPush and AdroitDoor), locomotion (Walker), and games (CarRacing). In contrast, most existing works [2, 3, 4, 5] focus only on locomotion tasks. \n- The tasks included in our work cover both vectorized-state-based environments (PointMaze, AntMazmethod'sPush, AdroitDoor, and Walker) and image-based environments (CarRacing and DrawerClose), while most existing works [2, 3, 4, 5, 6, 7, 8, 9] only consider vectorized-state-based environments.\n- Our work analyzes the data efficiency by varying the amount of available expert demonstrations and evaluating the learning performance of all methods in Section 5.4.\n- We designed and conducted a toy environment to visualize the reward functions learned by our proposed method and GAIfO.\n\n> More baselines\n\nPlease see the response to Reviewer Dab5.\n\n**References**\n\n[1] Li et al. \"Infogail: Interpretable imitation learning from visual demonstrations.\" In NIPS, 2017.\n\n[2] Liu et al. \"Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization.\" ICML, 2022.\n\n[3] Papagiannis et al. \"Imitation learning with sinkhorn distances.\" In ECML PKDD, 2022.\n\n[4] Ni et al. \"f-irl: Inverse reinforcement learning via state marginal matching.\" In CoRL, 2021.\n\n[5] Liu et al. “CEIL: Generalized Contextual Imitation Learning.”. In NeurIPS, 2023.\n\n[6] Garg et al. \"Iq-learn: Inverse soft-q learning for imitation.\" In NeurIPS, 2021.\n\n[7] Fu et al. \"Learning robust rewards with adversarial inverse reinforcement learning.\" In ICLR, 2018.\n\n[8] Ho et al. \"Generative adversarial imitation learning.\" In NIPS, 2016.\n\n[9] Xiao et al. \"Wasserstein adversarial imitation learning.\" arXiv, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P01EOgkcxf", "reviewer": "Reviewer_Dab5", "summary": "The paper introduces a novel method named Diffusion Imitation from Observation (DIFO), which integrates diffusion models into the adversarial imitation learning from observation (LfO) framework. Traditional adversarial imitation learning methods often struggle with hyperparameter sensitivity and training stability. DIFO leverages the generative capabilities of diffusion models to improve the imitation process. Specifically, the diffusion model generates the next state based on the current state, and its learning objective is reformulated as a binary classification task to distinguish between expert and agent transitions. This model then provides \"realness\" rewards to guide the policy learning process. The paper demonstrates that DIFO achieves superior performance across various continuous control tasks compared to existing LfO methods.", "questions": "1. The diffusion discriminator is different from prior methods in that we can sample from the distribution $D(s, s')$. Can the resulting discriminator itself be used as a policy when combined with an inverse dynamics model $p(a|s, s')$, and how does it perform during evaluation?", "limitations": "Authors listed the limitation of their method in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach by integrating diffusion models into the adversarial imitation learning from observation (LfO) framework, enhancing both stability and performance.\n2. The proposed DIFO method consistently outperforms existing LfO methods across various continuous control tasks, demonstrating improved data efficiency.", "weaknesses": "1. Although the authors name their paper \"Diffusion Imitation from Observation,\" the performance gain does not seem to be due to the use of the diffusion model. As shown in Section 5.7, using the diffusion loss alone demonstrates very poor results. The major contributing factor is the BCE loss and the discriminator as a whole. It appears more like using diffusion loss as regularization for discriminator training. Do other regularization techniques lead to similar improvements, and why are diffusion models more suitable in this setting (an intuitive explanation)?\n2. The LfO baselines in the experiments are rather outdated. More recent LfO baselines should be compared and discussed [1, 2].\n3. Minor problems:\n   1. In Line 111, there seems to be a missing symbol following the first tilde, and the $\\phi$ should be the \"parameters\" of the diffusion model.\n   2. In the Preliminary section, the symbol $t$ is used to denote the environment step, while later $t$ is again used as the diffusion step, which might cause confusion. It is common practice in diffusion-based RL papers to denote the diffusion step and environment step using different symbols, respectively written as superscripts and subscripts.\n   3. The \"Expert\" lines are missing in Figure 3 and Figure 4.\n\n\n[1] Liu, M., Zhu, Z., Zhuang, Y., Zhang, W., Hao, J., Yu, Y., & Wang, J. (2022, June). Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization. In *International Conference on Machine Learning* (pp. 14173-14196). PMLR.\n\n[2] Liu, Y., Dong, W., Hu, Y., Wen, C., Yin, Z. H., Zhang, C., & Gao, Y. Imitation Learning from Observation with Automatic Discount Scheduling. In *The Twelfth International Conference on Learning Representations*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> As shown in Section 5.7, using the diffusion loss alone demonstrates very poor results. The major contributing factor is the BCE loss and the discriminator as a whole.\n\nDiffusion models have shown their ability to become classifiers with diffusion loss [1]. DIFO-NA demonstrates using diffusion loss from an offline pre-trained model provides valid rewards, as shown in Figure 3(a), which indicates diffusion loss could be a reasonable metric for the discriminator. However, as the agent continues to learn, it will generate transitions that increasingly resemble those of the expert, causing the offline model to lose its ability to provide precise rewards eventually. Hence, we integrate the AIL framework to improve the discriminator simultaneously with the policy.\n\n> It appears more like using diffusion loss as regularization for discriminator training. Do other regularization techniques lead to similar improvements?\n\nWAILfO builds on GAIfO's work by incorporating gradient penalty as a regularization to improve performance. Additionally, our implementations of GAIfO, AIRLfO, and WAILfO already employ L2 regularization. The experimental results show that our proposed method outperforms these methods with various regularizations. We would like to highlight that these regularizations could also be applied to our method.\n\n> Why are diffusion models more suitable in this setting (an intuitive explanation)?\n\nWe hypothesize that there are two reasons why our diffusion model classifier performs better than the GAIL discriminator. First, the instability of GAIL arises from the tendency of discriminators to overfit, resulting in the policy's inability to learn from discriminator rewards. Unlike GAIL's MLP binary classifier, which maps a high-dimensional input to a one-dimensional logit, our diffusion model learns to predict high-dimensional noises, which is inherently more difficult to overfit. Second, diffusion models excel at modeling multimodal distributions and thus can outperform GAIL's discriminator when expert demonstrations exhibit significant variability. We will revise the paper to include these intuitions.\n\n> More recent LfO baselines should be compared and discussed [1, 2].\n\nWe thank the reviewer for these references. We will revise the paper to discuss these works. As requested by the reviewer, we additionally included the results of two recent methods: Decoupled Policy Optimization (DePO) [2] and Optimal Transport (OT) [3] provided by Reviewer P9FE, which is considered a more general version compared to ADS [4] provided by Reviewer Dab5. We report the results in Figure R.1 in the PDF file attached to the rebuttal summary. The results show that our proposed method outperforms both DePO and OT.\n\nDePO decouples the policy into a high-level state planner and an inverse dynamics model (IDM), utilizing embedded decoupled policy gradient and generative adversarial training. The decouple feature makes the planner and IDM transferable across different domains. However, it still suffers from the same challenges as the ones in GAIL, such as overfitting and the inability to model multimodality. Moreover, it also requires training with an IDM, making it more difficult to tune.\n\nOT uses proxy rewards derived from Sinkhorn distances rather than directly obtaining rewards from the raw logits of the discriminator, which enhances stability compared to AIL methods. However, OT computes distances at the trajectory level rather than the transition level, which requires monotonic trajectories. Consequently, OT performs well in environments like Walker and AdroitDoor, where trajectory variety is limited. However, it struggles in tasks with diverse trajectories, such as navigation, where the initial and goal states vary significantly. In contrast, our method generates rewards at the transition level, allowing us to identify transition similarities even when facing substantial trajectory variability. This flexibility enables our method to succeed in more complex environments with diverse trajectories.\n\nWe will include the results and discussion of these two methods in the revised paper.\n\n> The diffusion discriminator is different from prior methods in that we can sample from the distribution 𝐷(𝑠,𝑠′). Can the resulting discriminator itself be used as a policy when combined with an inverse dynamics model 𝑝(𝑎|𝑠,𝑠′)?\n\nAs suggested by the reviewer, we believe that combining an ideal inverse dynamics model (IDM) with our diffusion discriminator could indeed result in an effective policy. This potential is illustrated by our successful trajectory generation in PointMaze experiments (Section 5.5 and Figure 5), where our diffusion discriminator shows the capability to accurately predict the next state based on the current state.\n\nHowever, obtaining a high-quality IDM is challenging, as discussed in lines 27-29. Training IDMs requires data that is well-aligned with the expert's data distribution, while, collecting such data relies on having an effective policy, creating a deadlock. Even if this problem is solved, the planner may also produce invalid transitions for the IDM, posing challenges for both planners and IDMs.\n\nGiven these difficulties, our method directly learns a policy with the rewards produced by the diffusion discriminator instead of addressing the difficulties of obtaining an ideal IDM. We will revise the paper to make this clear.\n\n**References**\n\n[1] Li et al. \"Your diffusion model is secretly a zero-shot classifier.\" In ICCV, 2023.\n\n[2] Liu et al. \"Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization.\" In ICML, 2022.\n\n[3] Papagiannis et al. \"Imitation learning with Sinkhorn Distances.\" In ECML PKDD, 2022.\n\n[4] Liu et al. “Imitation Learning from Observation with Automatic Discount Scheduling.” In ICLR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ABxEst6CpR", "reviewer": "Reviewer_P9FE", "summary": "This paper introduces Diffusion Imitation from Observation, an adversarial imitation learning method using a conditional diffusion model as the discriminator for policy learning. DIFO learns a diffusion model on expert and agent state transitions, with an auxiliary binary classification objective to discriminate between the expert and the agent. Experiments show that DIFO can learn an imitation policy online, given only expert demonstration observations, outperforming relevant LfO baselines. Ablations and variants show that both loss terms are important for the diffusion model training, and justify the choice of diffusion model conditioning input.", "questions": "- It seems unclear to me why behavioral cloning, with access to ground truth action labels, is doing worse than LfO methods in Figure 3.\n- Does DIFO work with negative samples generated offline, e.g. adding Gaussian noise to the previous state, instead of policy online rollouts?", "limitations": "- Most experiments are on state-based environments. It would be convincing to see more evaluations on image-based environments.\n- This method requires online rollouts for collecting negative transitions for discriminator training. It would be great if this method can be shown to work completely offline.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Using conditional diffusion models for adversarial imitation learning from observations is a novel approach.\n- The loss objectives in the diffusion model are well-motivated and shown to be necessary for downstream policy performance.\n- The trained diffusion model captures the expert distribution, and exhibits some generalizability in the generated trajectories.\n- Experiment results show that DIFO works across multiple state-based and image-based environments, outperforming previous methods for learning from observation.", "weaknesses": "- This method requires online interactions to train the diffusion discriminator and the policy. One popular alternative approach in this setting (online + expert demonstrations) is optimal transport-based RL, which also works with only access to expert demo observations (optimal transport over expert and agent state trajectories as the reward). [\\[1\\]](https://arxiv.org/pdf/2008.09167) [\\[2\\]](https://arxiv.org/pdf/2206.15469) It would be convincing to see comparisons with this line of work.\n- It would be great to see more evaluations on image-based environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> This method requires online interactions to train the diffusion discriminator and the policy. One popular alternative approach in this setting (online + expert demonstrations) is optimal transport-based RL, which also works with only access to expert demo observations (optimal transport over expert and agent state trajectories as the reward). [1] [2] It would be convincing to see comparisons with this line of work.\n\nWe thank the reviewer for providing these references. We will revise the paper to discuss these works. As requested by the reviewer, we additionally implemented Optimal Transport (OT) [1]. OT uses proxy rewards derived from Sinkhorn distances rather than directly obtaining rewards from the raw logits of the discriminator, which enhances stability compared to AIL methods. \n\nWe report the result in Figure R.1 in the PDF file attached to the rebuttal summary, which shows that our method consistently outperforms OT across all evaluated tasks. We hypothesize that this is because OT computes distances at the trajectory level rather than the transition level, which requires monotonic trajectories. Consequently, OT performs well in environments like Walker and AdroitDoor, where trajectory variety is limited. However, it struggles in tasks with diverse trajectories, such as navigation, where the initial and goal states vary significantly. In contrast, our method generates rewards at the transition level, allowing us to identify transition similarities even when facing substantial trajectory variability. This flexibility enables our method to succeed in more complex environments with diverse trajectories.\n\nWe will revise the paper to include the results and the discussion.\n\n> It would be great to see more evaluations on image-based environments.\n\nAs requested by the reviewer, we additionally conducted experiments on the drawer close task introduced in Meta-World [2] using image-based states. This table-top manipulation task requires the agent to control a Sawyer robotic arm to close a drawer. An illustration of this task is shown in Figure R.5(a) in the PDF file attached to the rebuttal summary. \n\nFigure R.5(b) presents the learning efficiency of our proposed method and the baselines. Our proposed method achieves an 80% success rate with only 70k environment interactions, outperforming BC, BCO, GAIfO, WAILfO, AIRLfO, IQ-Learn, and DePO. \n\nWe will revise the paper to include the results of this new image-based environment.\n\n> It seems unclear to me why behavioral cloning, with access to ground truth action labels, is doing worse than LfO methods in Figure 3.\n\nWhile BC has access to ground truth action, it may suffer from compounding errors, i.e., the accumulation of errors from small initial deviations, caused by covariate shifts [3, 4, 5]. Because BC relies solely on learning from the observed expert dataset, unlike the LfO methods that utilize online interaction with environments, BC is susceptible to accumulating errors from states with small deviations, ultimately reaching an irrecoverable point. Therefore, BC is well-known for requiring a substantial amount of expert data to achieve coverage of the dataset and reduce unseen states. However, under our experiment setting with limited expert data, BC doesn’t have sufficient data to generalize to unseen states, leading to poor performance.\n\n> Does DIFO work with negative samples generated offline, e.g. adding Gaussian noise to the previous state, instead of policy online rollouts?\n\nWe thank the reviewer for the insightful idea. This idea of generating and utilizing negative samples offline for imitation learning resembles the idea of Implicit Behavior Cloning (Implicit BC) [6]. Implicit BC learns the joint probability of expert state-action pairs $p(s, a)$ via contrastive learning using expert data and offline generated negative data without policy online rollouts. \n\nWe believe it is possible to generate negative data offline by adding Gaussian noise to the previous state, as suggested by the reviewer, and learning our diffusion classifier to classify expert state transitions and the generated state transitions. However, unlike Implicit BC, our work focuses on learning from observation, where we do not have access to action labels in expert demonstrations. Hence, without online interactions, it would be impossible to know the action space, let alone learn a policy. Thus, we believe that online interactions are essential for policy learning.\n\n**References**\n\n[1] Papagiannis et al. \"Imitation learning with Sinkhorn Distances.\" In ECML PKDD, 2022.\n\n[2] Yu et al. \"Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning.\" In CoRL, 2019.\n\n[3] Ross et al. “Efficient Reductions for Imitation Learning.” In AISTATS, 2010.\n\n[4] Ross et al. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” In AISTATS, 2011.\n\n[5] Laskey et al. “SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces.” In ICRA, 2016.\n\n[6] Florence et al. “Implicit Behavioral Cloning.” In CoRL, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The attached PDF file contains the following content:
- **Two additional baselines (DePO and OT) and the results [Reviewer AzJp, Reviewer Dab5, Reviewer P9FE]**: We additionally include two recent and relevant baselines: Decoupled Policy Optimization (DePO, 2022) [1] and Optimal Transport (OT, 2022) [2] suggested by reviewer Dab5 and P9FE. These baselines represent two other major families of LfO approaches: inverse dynamics models (IDM) and sequence matching. The experiment results presented in Figure R.1 show that our method outperforms both DePO and OT across all tasks.
- **Optimizing the MSE loss with agent data [Reviewer 7W6h]**: We evaluate optimizing the diffusion model denoising loss $\mathcal{L}\_{\text{MSE}}$ with and without agent data in all tasks. The results presented in Figure R.2 show that optimizing $\mathcal{L}\_{\text{MSE}}$ with agent data leads to slower convergence and less stable performance, which justifies our choice of only optimizing $\mathcal{L}\_{\text{MSE}}$ with expert data.
- **Sampling different numbers of denoising timesteps for computing rewards [Reviewer AzJp]**: To examine how varying the number of denoising timestep sampled for computing rewards impacts policy learning, we experiment with different sample sizes (n=1, 2, 5, 10) in PointMaze. The results are shown in Figure R.3, indicating the effect of various numbers of samples is statistically insignificant, which justifies our choice of using one timestep sample for computing rewards.
- **Stochastic environments [Reviewer 7W6h]**: To investigate our diffusion model's ability to handle stochastic data, we created a stochastic AntMaze environment by adding strong Gaussian noise with a standard deviation of 0.5 to the actions. The results are shown in Figure R.4, demonstrating our method maintains robust performance under stochasticity.
- **Additional image-based Meta-World drawer close task [Reviewer P9FE]**: In response to the suggestion of adding more image-based tasks, we introduced the drawer close task from Meta-World [3]. This table-top manipulation task requires the agent to control a Sawyer robotic arm to close a drawer. Figure R.5(a) provides a screenshot of the task, and the learning efficiency is depicted in Figure R.5(b). Our method, DIFO, demonstrates superior performance compared to the baselines, including BC, BCO, GAIfO, WAILfO, AIRLfO, IQ-Learn, and DePO.

> [Reviewer 7W6h] Why do we only learn the MSE on the expert data but not also the agent data?

Please see the overall response. As pointed out by the reviewer, our method optimizes $\mathcal{L}\_{\text{MSE}}$ (Eq. 6) to approximate the ELBO only using expert demonstrations. We hypothesize that optimizing $\mathcal{L}\_{\text{MSE}}$ leads to unstable training because, during the early stage of training, the agent policy changes frequently and generates diverse transitions.

To investigate the effect of optimizing this MSE loss using agent data, we experiment with optimizing $\mathcal{L}\_{\text{MSE}}$ with and without agent data on all tasks. The results are reported in Figure R.2 in the PDF file attached to the rebuttal summary. We found that optimizing $\mathcal{L}\_{\text{MSE}}$ with agent data can lead to slower and unstable convergence, especially in tasks with larger state and action spaces, e.g., AdroitDoor, where optimizing $\mathcal{L}\_{\text{MSE}}$ leads to a 0\% success rate. We hypothesize that the rapidly changing agent distribution hinders learning the diffusion model optimizing $\mathcal{L}\_{\text{MSE}}$ with agent data. As a result, the overall performance can be less stable. Hence, we design our method to optimize $\mathcal{L}\_{\text{MSE}}$ using only expert data.

We thank the reviewer for the question. We will revise the paper to include this experiment and the discussion.

> [Reviewer 7W6h] Is it more convincing to have stochastic environments rather than purely deterministic environments?

We thank the reviewer for the suggestion. We additionally created a new stochastic AntMaze environment where Gaussian noise is added to the agent's actions before they are applied in the environment. The magnitude of the noise is 0.5, resulting in the actual action taken in the environment would be $action = action + 0.5 *\mathcal{N}(0,1)$. Given the action space of this environment is [-1,1], this represents a high level of stochasticity. 

We report the result in Figure R.4 in the PDF file attached to the rebuttal summary. The result shows that the performance of our method remains robust even under such high stochasticity, indicating our model's ability to adapt to stochastic environments effectively.  We appreciate the reviewer's insightful suggestion, which leads to this experiment that strengthens our paper. We will include this result in the revised paper.

**References**

[1] Liu et al. "Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization." In ICML, 2022.

[2] Papagiannis et al. "Imitation learning with Sinkhorn Distances." In ECML PKDD, 2022.

[3] Yu et al. "Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning." In CoRL, 2019.

Meta-review: This paper proposes Diffusion Imitation from Observation (DIFO), a diffusion-model method for learning from state-only demonstrations without action labels, within an adversarial imitation learning framework. DIFO utilizes a diffusion model to capture expert and agent state transitions, and reformulates the learning objective as a binary classification task to train the diffusion model as a discriminator, providing "realness" rewards for policy learning. The method demonstrates superior performance in various continuous control tasks, such as navigation, locomotion, and manipulation, compared to existing methods. DIFO also exhibits better sample complexity, making it suitable for real-world applications where expert demonstrations are limited. 

This paper highlights the effectiveness of diffusion models in improving the robustness and performance of imitation learning from observation (ILfO). All the reviewers agreed that this paper makes a good contribution to the conference.
Decision: Accept (poster)