DPAIL: Training Diffusion Policy for Adversarial Imitation Learning without Policy Optimization
Authors: Yunseon Choi
Minchan Jeong
Soobin Um
Kee-Eung Kim
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Adversarial Imitation Learning
Primary area: deep_learning

Abstract: Human experts employ diverse strategies to complete a task, producing to multi-modal demonstration data. Although traditional Adversarial Imitation Learning (AIL) methods have achieved notable success, they often collapse theses multi-modal behaviors into a single strategy, failing to replicate expert behaviors. To overcome this limitation, we propose DPAIL, an adversarial IL framework that leverages diffusion models as a policy class to enhance expressiveness. Building on the Adversarial Soft Advantage Fitting (ASAF) framework, which removes the need for policy optimization steps, DPAIL trains a diffusion policy using a binary cross-entropy objective to distinguish expert trajectories from generated ones. To enable optimization of the diffusion policy, we introduce a novel, tractable lower bound on the policy's likelihood. Through comprehensive quantitative and qualitative evaluations against various baselines, we demonstrate that our method not only captures diverse behaviors but also remains robust as the number of behavior modes increases.

DPAIL: Training Diffusion Policy for Adversarial Imitation Learning without Policy Optimization
Yunseon Choi
Minchan Jeong'
Soobin Um 4.2
Kee-Eung Kim
Kim Jaechul Graduate School of AI, KAIST
2 Department of AI, Kookmin University
{cys9506, kekim} @kaist.ac kr
Abstract
Human experts employ diverse strategies to complete task, producing to multi- modal demonstration data. Although traditional Adversarial Imitation Learning (AlI.) methods have achieved notable success, they often collapse theses multi- modal behaviors into single strategy, failing to replicate expert behaviors. To overcome this limitation, we propose DPAIL, an adversarial IL framework that leverages diffusion models as policy class to enhance expressiveness. Building on the Adversarial Soft Advantage Fitting (ASAF) framework, which removes the need for policy optimization steps, DPAIL trains diffusion policy using binary cross- cross-entropy objective to distinguish expert trajectories from generated To enable optimization of the diffusion policy, introduce novel, tractable lower bound on the policy's ' likelihood. Through comprehensive quantitative and qualitative evaluations against various baselines, demonstrate that method not only captures diverse behaviors but also remains robust as the number behavior modes increases.
Introduction
Many sequential decision- Dacision-making tasks involve multiple distinct expert behaviors to achieve the same goal, resulting in multi-modal demonstration data. For example, human demonstrations of robotic manipulation tasks often exhibit various grasping strategies, alternating between cautious and aggressive movements. Capturing such diversity important in imitation learning (IL), as an agent should reproduce the range of expert strategies rather than collapse to an average behavior. Generative modeling approaches offer a potential solution by learning rich distributions over expert behavior. In particular, diffusion models, such as the Denoising Diffusion Probabilistic Model (DDPM) 15] have emerged as powerful generative models capable of capturing complex, multi- modal data distributions. Recent work [22] suggests that diffusion- based policies, trained to mimic expert actions via behavioral cloning, can more faithfully reproduce stochastic and multi-modal human demonstrations than conventional uni-modal policies. These successes motivate to leverage diffusion models for IL, aiming to model diverse expert behaviors as a multi-modal distribution. However, a behavior cloned policy often struggles to generalize beyond the demonstrated data distribution, suffering from compounding errors when encounters novel states 225]
While behavioral cloning directly learns a policy from state-action examples via supervised learning. inverse reinforcement learning (IRL) methods instead infer reward function under which the expert is optimal and then train policy to maximize this reward 37.. By capturing the underlying objectives of expert behavior rather than specific actions, policies trained via IRL. exhibit better generalization and adaptability to variations in the environment or task dynamics. However, classical IRL was often impractical, requiring repeatedly solving full RL problem an inner loop. Adversarial Imitation
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Learning (AIL) reframes IRL as distribution sestribution-mtchinggg problem. enabling more efficient learning. In particular, Generative Adversarial Imitation Learning (GAIL) [14 introduced a GAN-like framework in which a discriminator is trained to distinguish expert from agent behaviors, and policy modeled as a generator trained via policy gradient to produce trajectories that the discriminator cannot tell apart from expert trajectories. This formulation effectively matches the occupancy measure of the learned policy to that of the expert, and GAIL and its variants [36, 20] have demonstrated strong performance on complex control tasks. However, the alternating optimization between the discriminator and policy makes such adversarial methods computationally expensive and sometimes unstable.
Recent research has sought to simplify AlL by removing the costly policy optimization inner loop. Barde et al. [6] proposed Adversarial Soft Advantage Fitting (ASAF), which avoids policy gradient updates altogether. In ASAF, the discriminator is structured to ouiput parametric policy distribution directly, conditioned on both the previous generator policy and a new learnable policy. When optimized to equilibrium, this discriminator effectively solves for the optimal imitation policy, eliminating the need for a separate policy update step, i.e. the updated policy obtained for free from the discriminator parameters. This elegant one-step formulation drastically simplifies training, cutting out the policy gradient updates. Crucially, however, ASAF assumes the policy class is one for which probabilities can be evaluated in closed form, e.g. policies modeled as Gaussian or normalizing flow. This is because the training objective of the discriminator involves computing policy densitites, an operation that is tractable for simple parametric flow-base based policies. This assumption breaks down for diffusion models: diffusion policies generate samples via an iterative denoising process, and their exact likelihood are generally intractable to compute. This incompatibility prevents a naive approach to ASAF to diffusion- policies.
In this paper, we bridge diffusion generative models with AIL without requiring any policy optimiza- tion step. Building on the insights of ASAF, we introduce a diffusion policy imitation framework that retains the ability to match expert distributions adversarially while harnessing the expressive power of diffusion models for multi-modal behavior. The key contribution is a novel training objective: we derive tractable lower bound to the ASAF objective that enables us to train diffusion policies efficiently despite the intractability of their exact densities. By optimizing this lower bound, our method avoids explicit computation of marginal diffusion densities, yet still directs the diffusion policy to fit expert demonstration distribution. Notably, our approach also complements recent advances that integrate diffusion models into AlL., For example, diffusion models have been used to strengthen the discriminator in AIL by modeling state sate-action distributions [18] 34]; in contrast, our focus in using diffusion the policy model itself and devising training algorithm that circumvents policy gradient updates.
We summarize our main contributions as follows:
1. Diffusion Policies for AIL:
To the best of our knowledge, we propose the first AlL framework that employs diffusion policies to model a multi-modal expert behavior distributions.
ASAF for Diffusson Policies: We adapt the ASAF algorithm 66) to diffusion- based policies by deriving a new training objective that bypasses intractable computation of marginal diffusion densities. We show that this surrogate objective is tight lower bound of the training objective in ASAF, making our algorithm an instance of minorizaioon-maxiiizzzooo algorithm [16].
3. Empirical Validation: We evaluate the proposed diffusion policy imitation method on standard IL benchmarks, including continuous control tasks in the MuJoCo simulator (32 and maze navigation tasks in the Maze2D environment [T]] Without the policy optimization loop, our diffusion-basse approach effectively learns to imitate the expert demonstrations on these tasks, demonstrating that diffusion policies can capture the breadth of expert behaviors while maintaining competitive performance with state-of-th thet-hart art imitation learning methods.
2 Related Works
2,1 Adversarial Imitation Learning (AIL.)
AIL recasts imitation learning as occupancy-measure distribution matching, removing the expensive full RI. inner loop required by classical IRL algorithms. GAIL [14 was one of the first to formulate this idea as a GAN-style minimax game between a policy generator and a discriminator, proving its effectiveness in digendimmnsional continuous-control tasks. A series of extensions broadened its
2

---

augments the adversarial objective with mutual information regularizer that pushes the policy to embed diverse behavioral patterns in latent variables, enabling to disentangle and reproduce multiple expert behavior modes. Building this idea, Ess-InfoGAlI. [IO] introduces a semi- coni-supervised learning approach that extracts more meaningful representations from a small amount of labeled demonstrations, thereby improving sample efficiency and stability. Although these variants enhance the ability to imitate diverse expert behaviors, they still depend on the policy update in the inner loop, inheriting the associated computational complexity and instability from alternating optimization.
expressiveness: InfoGAIL 20 13
To alleviate this burden, Adversarial Soft Advantage Fitting (ASAF) reformulates the AlL objective so that the policy update occurs implicitly inside the discriminator, eliminating the need for separate policy optimization. While computationally attractive, ASAF has so far been restricted to policy classes with probabilities evaluatable closed form, e.g. Gaussians or normalizing flows, because its objective requires exact density evaluation.
2.2 Diffusion Models in Sequential Decision Making
28 31 have demonstrated remarkable success in generative modeling, particularly in the domains of images and audio [8] (3 5], and now increasingly being applied to sequential decision-min making tasks. Diffusion-Bel [22 showed that diffusion- policies are more accurate in replicating diverse human demonstrations in robotic manipulation tasks under behavior cloning. Diffuser [17 uses diffusion model to generate trajectories guided by the reward function. Diffuser Decision uses conditional diffusion model with classifier-free guidance to generate trajectories. Diffusion QL [35] represents the policy as a diffusion model, and train it via Q-learning.
Diffusion models [5 30
Diffusion models have been also adopted in the AlL setting. DiffAIL 34 replaces the standard binary discriminator AIL with an unconditional diffusion model trained via diffusion loss, enabling more precise sccupancy-meant conditional diffusion model as the discriminator to improve its ability to distinguish between agent and expert state-action pairs.
matching of expert state-actin pairs. DRAIL [18] instead incorporates
Our work departs from these prior works by extending ASAF diffusion- based policies, whose exact densities are intractable to evaluate, via tractable lower bound on the ASAF objective. This development combines the efficiency and stability of ASAF with the rich, multi-modal expressivity of diffusion models, achieving adversarial imitation learning without any explicit policy optimization loop.
3 Preliminaries
3.1 Reinforcement Learning and Diffusion Models
Markoy Decision Process (MDP) We model the RL problem as an MDP, defined by the tuple Po. 7), where S is the state space, is the action space, P(s'|s, a) E [0, 1] is the
M MM(SAA, (S,A, state transition probability function, r(s, a) E R is the reward function, Po(s) E [o, 1] is the initial state distribution, and [0. 1] is discount factor. The objective is to learn a policy that maximizes the expected return, max E E 7 (rr(s, 0:)].
Diffusion Models Diffusion models [15] have emerged as powerful tools for RL tasks [17 19 Following [17]. we utilize diffusion probabilistic models [15] for generating trajectories. These models are composed of two stochastic processes: forward noising process and reverse denoising process. The forward process, shown in Eq. (I) is Markov chain with predefined Gaussian transition, where data T E T is iteratively corrupted according to fixed variance schedule:
a(
= :- /1 3,7"-1,,0 ,,,1),
for i E (1...., N and < 3, < The variance schedule 3; is chosen such that q(>^) approxi- mates 1). The reverse (denoising) process, shown in Eq. (2). is another Markov chain with paraneterized Gaussian transition, used for the data sampling distribution T0 ~ po(-):
Pe ( (77) = NN7 : (1) )')
3

---

where Ha (T', i) is typically parameterized by neural network. A common way to train this reverse process is via the variational lower bound. However, rather than predicting the mean μs, DDPM [T5] propose to predict the forward noise e, leading to the simplified loss function:
N |||
where 7' == Va,T" + VI C,C, and a; == II; (1 B2). Here, co is the noise prediction network that estimates the noise added to the clean sample 7 that leads to the noisy sample T'.
3.2 Adversarial Imitation Learning
This algorithm employs GAN-like training for AIL, where policy acts generator and an explicit discriminator D T - [0, 1] serves as a binary classifier to distinguish the trajectory T between the demonstration distribution pr induced by the expert policy ay and the imitation policy distribution Pr. The policy and the discriminator are trained with the minimax optimization objectíve:
GAIL [14]
min max L(D,Pt) D where C(D. = Bra llog D(T) + E.-p- -
GAIL alternates between updating discriminator and the policy. The discriminator is updated using the binary cross-entropy loss to differentiate expert samples from generated ones. The policy is then updated through RL by using the discriminator output as the reward, e.g. - [log(1 D(T))
This algorithm, while adopting the same GAIL objective in Eq. (4), introduces the structured discriminator, motivated by the analytical solution of the inner maximization (9. Formally, for any pair of policies л and n², define
ASAF 66
(T)
(T) Pr (T) + PT(
Then, the solution to the inner maximization in Eq. 4 is equal to the expert distribution, π² argmax, s'' :T) TH for any policy म, and the solution the outer minimization is also equal to the same solution, i.e. argmin, max,' L(Dx, , argmin, L(Da, = TE.
Based on this result, practical implementation of ASAF sets the policy from the previous iteration and updates the policy n' to maximize the discriminator objective L.
Method
Our goal is to employ a diffusion- based policy to capture the multi-modal behaviors observed in expert demonstrations. AlL methods typically involve training discriminator and a generator via mínimax objective, which often leads to training instability. becomes too accurate, it cause vanishing gradients for the generator, hindering the learning process. When applying such adversarial training to diffusion policies, these issues are further exacerbated. Specifically, computing policy gradient from reward signals requires backpropagation through the entire diffusion sampling steps, resulting in substantial computational cost and additional instability (21)
particular, when the discriminator
Therefore, we propose to train the diffusion policy using ASAF for improved efficiency and stability, as ASAF does not require an explicit discriminator separate policy gradient steps. The fundamental bottleneck, however, that ASAF requires efficient evaluations p- (T)),, which is intractable for diffusion models. To address this issue, we derive lower bound of the discriminator training objective that is tight at the optimal solution. With this lower bound as the surrogate objective, our algorithm, Diffusion Policy Adversarial Imitation Learning (DPAIL), becomes an instance of Minorization-Maxiiizaae (MM) algorithm [6].
DPAIL: Diffusion Policy for Adversarial Imitation Learning without Policy Optimization In line with Janner et al. [17], we assume trajectory-leve' diffusion models po(r), instead single-step models. Throughout this paper, denote 7 as trajectory to reuse notations from diffusion models.

---

First, we focus on the first term of Eq. (4) that involves expert demonstrations, Using the structured discriminator Eq. ( the maximization of the discriminator can be formulated as:
max E, ppp [log Daua, @ (T7) max Ey-pp log or loog Pa(r") (+0) )
(6)
Paou
When ()") and pa- (T") are modeled diffusion processes, it is computationally expensive to directly calculate the marginal densities. We first leverage the fact that the perturbed transitions ((r"T===1) of pe and Paas are the same for all forward diffusion steps i E [1, N since they have the same variance schedule. This allows us to rewrite the log density ratio as follows:
Po(T)) Paau (r") = log Ih ((T")=(1) po(TT) I ((T")r²1)) Paue (
log II Po po(T ((+
log
IT Pa (7+) q(T
The last equality comes from the fact that the reverse and forward processes coincide when the diffusion process reaches equilibrium, with q(T being Gaussian that will be canceled out. In addition, taking the expectation of this log density ratio over 7 7!:N ((1(7)) does not change anything since it still ratio of marginal densities. Thus, we can rewrite Eq. (6) as
II Po 119 ]
max E Po
log σ
199 log I Pan (ri-1
max E Eoo [fo(79] (7)
Finally, applying Jensen's incquality to the concave function log (.). we can derive a lower bound of f:
fo(T9) 2 Ei (Aogo(N.( (N.[KL(g(TT 70) / (-+) - (L( 70) ||po|
where in ((
Since q(---~11' 1)', is Gaussian by Bayes'rula rule, and both po(r"~117") and (r= are parame- terized Gaussians, all KL divergences in Eq. (8) admit closed-form expressions. Following (15.. we do not predict the mean μe of pe directly. Instead, we train noise prediction network under
which the mean is given by: (o((T i) = रा-म. (T' - (T'), Substituting this into Eq.
yields the first term of the DPAI.. objective as:
- (0, gold ,0) Er,e log a - (T,, -aep--llc-ool (1(
(9)
Here, Caes is the noise prediction network for poos, NO I). N), and 7¹ va.7 (1-ag) (1-ag)c.
For the second term of Eq. (4) that involves generative sample =0 from Pas, the lower bound of the objective derived similarly due to the symmetry of the sigmoid function - a alog
Pa
Poon
(log
Pand pe
(2) DRAIL (0, gold (00) = N. (l)oo)))( -go (Fu
(10)
Eq. 9 encourages accurate noise prediction for noised expert data by maximizing the gap in prediction errors between co and Cama. In contrast, Eq. (10) discourages accurate noise prediction for noised generative data in cg. Detailed derivations are provided in Appendix
Monotonic improvement
We employ the lower bound of Eq. (8) P (0,
Eya~pe
(0ld gold --0] + EEpo Paold
C (0, gold =9]
as surrogate objective. Importantly, our algorithm becomes an instance of minorization- maximization algorithm since (1) we optimize with a surrogate objective which a lower bound of the original objective (2) the lower bound is tight at the convergence gold = 0. This guarantees monotonic improvement of the original training objective. Further explanations are provided in
Appendix A
5

---

Algorithm DPAIL
Input:
expert trajectories D = fn J Randomly initialize pe, and set Pauu t- pdo for k E [O...., K] do
Collect trajectories Dauss = (Fn [ using Poo by interacting with environment
Update Oa-1 by optimizing the following loss in Eq. (9) and Eq. TT):
BK+1 arg maxo EDoDD [ (0, gold, 79)] + E E you [ (0, gold, =9)]
Pietdd < P02+1
end for
Practical implementation In our approach, the diffusion policy generates fixed-horizon sub- trajectories length H, instead of producing full trajectories. We then execute the resulting H actions sequentially in the environment. To condition on the current state at the start of the sampling process, we overwrite the corresponding state variable every diffusion step with the current observed state. The detailed procedures for action execution and sampling appear in Algorithm 2 and in Appendix. Since there is discrepancy between the trajectories sampled entirely from the generator ps: and those obtained from the environment, we clip the denoising error ||e Egu(T', i)) to prevent it from being too large. The complete DPAIL algorithm is summarized in Algorithm Additional implementation details can be found in Appendix D
Comparison with DRAIL DRAIL [8] was introduced to incorporate diffusion models into the discriminator in GAIL. It employs conditional diffusion models for the discriminator, with the loss function given by:
Can( (T. c) = E, ( -e|1
(11)
where c E fct, } denotes real or fake labels. This loss function forms the basis of the diffusion- based discriminator D& T + 1/.
(T = o (fair(7", (c) L (((,((),)))))))))))))(()))))(( ,c+))=
which is trained using a binary cross-entroes cross-entropy loss to predict 1 for expert samples and 0 for generated samples. The overall diffusion training objective of DRAIL is:
E Exo [log a [Laif(r", - cc))]] 1Ezoopp, o (Fair(F)) ( -ain c )}} (12)
Both DRAIL (Eq. (12) and DPAIL (Eq. (9) rely on differences in noise predictions errors within sigmoid function. In DRAIL, this difference arise from comparing predictions conditioned on two different class labels et and c using the same noise prediction network Eg. In contrast, DPAIL computes the difference using two distinct noise prediction networks, ep and Epac. Despite this similarity formulation, the two approaches differ fundamentally in how the diffusion model used. DRAIL trains unimodal Gaussian policy via RL, using the diffusion models purely reward signal, which limits its ability to represent modal distributions. DPAIL, on the other hand, directly leverages the generative capacity of diffusion models to reproduce the diverse behavior in the expert demonstrations.
Extension to lanent-conditioned diffusion policies Unlike InfoGAIL 220,, our method does not inherently learn latent representations, which makes difficult to guide the policy's 's behavior toward specific modes. However, the DPAIL framework can extended to incorporate latent variables a for mode conditioning, as this is compatible with derived lower bound. To enable this, define latent-conditioned diffusion model where the joint probability of the generative process p:N7 factorizes
N
DNN
0:N 2) 9(T )a(z) Il (o
where g(=) is the latent prior and g is the noise prior. To derive the corresponding tractable lower bound, we follow similar procedure to the main DPAIL. formulation; the detailed derivation is provided in Appendix F

---

Experiments
In this section, we evaluate our method across navigation and control tasks, including Maze2d and MuJoCo environments. We begin with quantitative results that demonstrate its effectiveness at modeling multi-modal expert demonstrations. Next, we provide qualitative trajectory visualization to illustrate its ability to reproduce diverse behaviors. We then investigate how performance varies with the size of the expert dataset and the number of behavior modes. Finally, we wrap up with the analysis the effects of the trajectory horizon H and the number of diffusion sampling steps N.
5.1 Experimental Setup
Environment We conduct experiments in six environments: (1) HalfCheetah-v33 and (2) Walker2d- v3: The goal of these environments is to control a robot to move in the desired direction, including running forward and backward.
this setup, a quadruped ant robot is tasked with moving in one of four directions, including forward, backward, left, or right. (4) AntGoal-v3: This environment requires a quadruped ant robot to navigate to target position. We define eight target locations, evenly distributed along circle with radius of 20. The target position is not observable in the state representation.
(3) Ant-v3:
(5) mazze2d-medium of three goal positions in medium-sizze sized maze. (6) maazeddlaarge-- Similar to the medium-sized maze task, but in a larger maze with five goal positions.
A point robot tasked with navigating to one
Multi-modal demonstration dataset For MuJoCo environments, we pre-train M expert policies using SAC, where each policy corresponds to one M behavior modes. We then sample K sets of expert demonstrations using these pre-trained policies, with each set consisting of 10 trajectories in MuJoCo. For Maze2d environments, we utilize the D4RL [IT] dataset to collect demonstrations consisting of trajectories from initial positions goal positions. Specifically, use 15 episodes for mazeed-medium-- and 30 episodes for maze2d-large-ve
Baselines
We compare our method against the following baselines:
BC 4 24 learns Gaussian policy via supervised learning, a mapping from observed states the corresponding expert actions.
Diffusion 17 trains diffusion policy models to predict action sequence conditioned on the state via supervised learning.
GAIL 14] learns a Gaussian policy by jointly training a generator and a discriminator. The discriminator tries to distinguish trajectories produced by the policy from expert demonstrations, while the policy is optimized to fool the discriminator.
DiffAIL 34 integrates diffusion models into AIL by using the diffusion model loss as reward. employs an unconditional diffusion model in the state-action reconstruction loss. Unlike GAIL, the reward does not come from the estimated value of (5).
DRAIL [8] combines diffusion models with GAIL by using conditional diffusion models
discriminator that performs binary classification.
is an extension of GAIL that trains Gaussian Mixture Models (GMM) to capture multi-modal behaviors by incorporating an unsupervised representation. It uses uniform categorical distribution as the prior for the GMM. ASAF 6 is an alternative approach to train Gaussian policy without policy optimization in AIL.
InLoGAIL 20
5.2 Experimental Results
To compare the performances of methods, use normalized scores for MuJoCo tasks and success rates for Maze2d tasks. For MuJoCo tasks, we compute returns across all modes, select the maximum value, and normalize it relative the corresponding expert performance, following [0 77. This score is high even if the agent captures only single mode of the expert behavior. In Maze2d tasks, success defined by whether the agent reaches one of the goals. We also evaluate behavioral diversity by measuring the entropy of the mode index that yields the maximum return during evaluation. In Maze2d, the goal index closest to the agent's 's final state is used for entropy calculation. Table summarizes the results.

---

Table I: Normalized score (Score) and entropy (Entropy) for MuJoCo and Maze2d tasks. Each experiments is conducted using 5 different random seeds, and we collect 50 episodes for each seed. We report the scores mean + standard error.
Environment
Metric
BC
Diffasion
GAIL
DifAIL
DRAIL
InfaGAIL
ASAF
DPAIL
HoliCheetib-v3
Score
0.61
=0.00
0.6
0.99
2.89
0.68
1.020001
Entropy
0.58
0.33
0.61
Walker20-e3
Score
0000
2000
072
0.13
0.33
0.55
0.00
0.78
Eatropy
0.12
0.41-0.10 +0.10
0.0+0000
0.06
0.21 +0.11
0.44
0.40 002
0.11
Ant-13
Entropy
Score
0.18
0.28
0.48
2.22
1.19
0.01 0000
0.02
1.22
0.02
1.26
0.03
201
0.00
1
0.56
1.20
AmGoolle3
Score
Entropy
0,01
1.46
0.22
1.52
0.57
0004
1.72
0,48
8.78
1.52
1.73
mowe2l-mdiimmxl
Seore
0.58
=.03
0
0.05
+.00
0.76
0.64
a.90
Eatropy
0.66
0.93
0.12
0.00 20.00
0.00
0.81 2015
0.75
0.91
maxe2d-lurgo-vi
Fatropy
Sexre
0.61
0..66
0:94
0.05-0.20
0.10+011
0.94
0.96
0.45
0,95
0.74
+.05
0.54
0.70
1,00
0.04
Average
Score
Entropy
0.34 10
0.93
0.19-
0.6+
0.63
0.14
0.50
0.59 0.
2..3
0.98 20220
0.13
0.76
0.35 0.13
0.90
0.83 0.06
HallCheetah
Antioal
-599
-0
-20
(x) DRAIL (Ouns)
(b) GAIL
[c) DIfAIL
(6) DRAIL
[e] InfoGAIL
(0 ASAF
Figure I: Learned behaviors of baseline methods and DPAIL (ours) in MuJoCo tasks. The first row shows HalfCheetah-v3, where the task is to move forward and backward (#x-axis) from a randomly initialized position around (0, 0). Each plot illustrates five different trajectories generated from the same policy. The second row shows AntGoal-v3, where the task is to reach one of eight target positions distributed around circle from randomly initialized position near (0, 0). Each plot illustrates ten different trajectories generated by the same policy.
In HalfCheetah-v3, all methods demonstrate strong performance except for BC. Specifically, GAIL, DiffAll., and ARI reach expert-level performance, but, yield zero or near-zero entropy, suggesting they collapse to single mode. While InfoGAIL attains higher entropy, it shows relatively lower performance, which indicate the challenges of unsupervised representation learning. In contrast, Diffusion and DPAIL achieve high performance and high entropy.
For more complex tasks with numerous behavior modes, such as Ant-v3 and AntGoal-v3, most baseline methods (including Diffusion) perform poorly. GAIL, DiffAIL, DRAIL, and ASAF fail to train, resulting in low returns and high entropy. These observations suggest instability when learning from multiple modes-agents frequently oscillate among modes and fail to learn a stable policy, Notably, training failures frequently yield nearly random behaviors, which lead to high entropy. For example, most baseline methods fail on Ant-v3, culminating in low performance but high entropy. Although InfoGAIL leverages the GMM to model diverse behaviors, it also exhibits low performance. This suggests that combining RL policy training with unsupervised representation learning remains as a challenging problem.
Meanwhile, Diffusion, an naive offline approach, sometimes outperforms several online methods in terms of returns and entropy (e.g., in Ant-v3 and Maze2d), likely due to the stability of its training process to effectively handle multiple modes. However, it struggles to perfectly imitate expert behaviors with limited datasets. In contrast, DPAIL consistently achieves strong performance alongside high entropy across all tasks. Even in environments with many behavior modes, DPAIL successfully avoids mode-collapse due to the capability of diffusion models, and successfully learn expert behaviors through the adversarial training. This demonstrates the effectiveness of our approach.
8

---

Diffusion
GAIL
InToGAIL
Normalized Score
0.5
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.0
0.0
3
10
20
10
20
5
10
20
(8) mode
modes
(c)
(d) modes
Figure 2: Performance with respect to the number of demonstrations and modes. The x-axis denotes the number of expert trajectories. error in Ant-v3 environment.
The graph depicts the average score across seeds with standard
To qualitatively analyze the learned behaviors of our algorithm and baselines, we visualize trajectories sampled from policies in Figure In MuJoCo tasks, DPAIL demonstrates diverse and successful behaviors across tasks. In contrast, most algorithms exhibit mode-collapse, where their learned behaviors concentrated in only few modes. Although InfoGAIL has high entropy, it struggles to imitate even a single mode. We present additional qualitative results of the learned behaviors in the Maze tasks in Appendix E
5.3 Further Analysis
Impact of the number of demonstrations and modes To further assess DPAIL, analyze its performance across different demonstration dataset sizes ((3,5 5, 10, 20) trajectories) and the number of modes, ((1, 2, 3, 4)) in Ant-v3 environment, where most baseline methods fail to learn policies. The results are presented in Figure
In single-mode settings, all methods achieve stronger performance than in four-mode settings. However, as the number of modes increases, most algorithms (excluding Diffusion and DPAIL) experience significant performance degradation, ultimately failing train. These findings highlight the limitations of traditional AIL methods in handling multi-modal expert demonstrations.
While both Diffusion and DPAIL exhibit performance declines as the number of modes increases, they do not fail to train policies. They maintain minimum normalized scores 0.2 (Diffusion) and 0.4 (DPAIL) in the trajectories demonstration and 4-modes setting. Notably, DPAIL consistently outperforms Diffusion when the number of demonstrations is limited. However, once the number of demonstrations becomes sufficiently large (e.g. 20 trajectories), Diffusion performs comparably to DPAIL. These results demonstrates effectiveness of DPAIL for scenarios requiring imitation from limited, multi-modal demonstrations. Extended quantitative analyses supporting Figure are provided Table 5 in Appendix.
Inference compute versus planning horizon H DPAIL incurs higher inference cost due to N-step diffusion sampling for action generation. reduce this cost, we generate H -teep sub-trajectoriis and execute the resulting H actions sequentially, requiring diffusion sampling only every H time-steps.
However, longer H introduces the compounding errors between pre- dicted and the state transitions. Figure 3 presents the trade-off be- tween inference time per action (Time Cost) and performance (Score) across different values H. As shown, shorter horizon yield better perfor- mance at higher time cost, while longer horizons reduce inference over- head bur degrade performance due to the accumulated prediction errors.
Time Soore
1000
08
Time Cost (md)
100
SAL
Score
Time Cost
0.5
10
82
Full
16
32
64
128
(6) mist-mmedium-
(b)
Antiicalis
Figure Inference compute versus planning horizon H

---

Diversity versus diffusion sampling step N The number of diffusion steps N di- rectly influences the model's ability to approximate the expert's multi-modaa distribution. To evaluate this effect, we vary N and report both entropy and task perfor- mance in Figure 4 As N increases, entropy rises, indicating richer behav- ioral diversity, while performance re- mains nearly unchanged. This find- ings indicate that at low N, the policy captures only a few dominated modes, whereas higher N values allow it to represent broader spectrum of be- haviors.
I Erc. Score
0.9
Entropy
0.8
0.7
10
20
40
50
(1)
20
(e)
msse-meddumm
(b) AntGssl-33
Figure 4: Diversity versus diffusion sampling step
6 Conclusion and Discussion
We presented Diffusion Policy for Adversarial Imitation Learning (DPAIL), framework integrates diffusion models into Adversarial Imitation Learning (AIL) to effectively capture multi-modal expert behaviors. DPAIL avoids the mode-collapse and instability issues faced by many existing methods when learning from multi-modal demonstrations. Extensive experiments on MuJoCo and Maze2d tasks demonstrate that DPAIL consistently achieves high returns and maintains high diversity, even in challenging environments with numerous expert modes.
7 Limitations and Future Work
While our approach effectively models multi-modal behaviors using diffusion models, it requires more computation costs for action decision compared to other baselines. This overhead arises from the iterative sampling process used by diffusion models. Several works [26] 12] have proposed methods accelerate sampling in diffusion process. Incorporating these techniques into DPAIL is a promising direction to enhance its efficiency. Furthermore, while we demonstrate that DPAIL is compatible with latent-conditioned diffusion policies, we did not provide empirical validation. We leave this as future work. Additionally, other directions exist for guiding policy behavior toward specific modes beyond training a latent-conditioned policy. For instance, recent research (29 has explored guidance mechanisms that steer the reverse diffusion process toward desired outcomes. Extending our framework with such mechanisms could further improve controllability in multi-modal settings.
Broader Impacts
The potential applications of our method extend across various fields, particularly in robotics and industrial automation, where learning from demonstrations crucial. In these domains, our approach can significantly enhance autonomous decision-making by enabling effective imitation learning from multi-modal expert demonstrations, where conventional algorithms often fail. By successfully capturing this behavioral diversity, our work contributes to the advancement of more capable and deployment.
reliable AI for
Acknowledgments
This work was supported by Institute for Information & Communications Technology Promotion (IITP), funded by MSIT, through the Information Technology Research Center (ITRC) program and other projects RSo NDSS2024-003992
Development of an AI Simulator for Creating Transparent Compounds that Can Be Altered for Tactile Sensation: No.RS NARS-20224004578899
AI Research Hub Project; RRS-2022-11203111 Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects: R02--2000-12009400 Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing: No.RS- 2019-11190075, Artificial Intelligence Graduate School Program (KAIST); LTP-2025-RS-2002 00436857).
10

---

References
[1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. Machine Learning, Proceedings of the Twenty-firsl International Conference ICML, volume 69 of ACM International Conference Proceeding Series. ACM, 2004.
[2] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. conditional generative modeling all you need for decision-making? In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[3] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.
[4] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. Intelligent Agents [St. Catherine's 's College, Oxford, July 1995]. Oxford University, 1999.
[5] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Hermann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel. and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation. In SIGGRAPH Asia 2024 Conference Papers, 2024.
[6] Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Chris Pal, and Derek Nowrouzezahrai. Adversarial soft advantage fitting: Imitation learning without policy optimization. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, 2020.
[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousincau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Roboties: Science and Systems XIX, 2023.
[8] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems, 2021.
[9] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy models, 2016. URL https
//axxiv. www.:///g///bbbeeeee.e 03852
[10] Huiqiao Fu, Kaiqiang Tang, Yuanyang Lu, Yiming Qi, Guizhou Deng, Flood Sung, and Chunlin Chen. Ess-infogail: Semi-supervised imitation learning from imbalanced demonstrations. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023.
[11] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, 2020.
[12] Zhengyang Geng, Ashwini Pokle, and J. Zico Kolter. One-step diffusion distillation via deep equilibrium models. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems, 2023.
[13] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gauray S. Sukhatme, and Joseph J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 2017.
[14] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, 2016.
[15] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[16] David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 2004.
[17] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Rescarch. PMLR, 2022.
[18] Chun-Man Lai, Hsiang- Chun Wang. Ping Chun Hsich, Yu-Chiang Frank Wang, Min Min-Hung Chen, and Shao- Hua Sun. Diffuuion-ren reward adversarial imitation learning. In The Thirty-eight Annual Conference Neural Information Processing Systems, 2024.

---

[19] Wenhao Li. Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[20] Yunzhu Li, Jiaminn Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems. 2017.
[21] Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference on Machine Learning, 2025.
[22] Tim Pearce. Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgeseu, Sergio Val- carcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In JEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR, 2022.
[24] Stéphane Ross and Drew Bagnell, Efficient reductions for imitation learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS, 2010.
[25] Stéphane Ross. Geoffrey J. Gordon, and Drew Bagnell, A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS, 2011.
[26] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations. ICLR, 2022.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[28] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[29] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and and- controllable generation. Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMI.R, 2023.
[30] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, 2019.
[31] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa, Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS. IEEE, 2012.
[33] Bram Wallace, Mcihua Dang, Rafacl Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diftusion model alignment using direct preference optimization. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR. IEEE, 2024.
[34] Bingzheng Wang. Guoqiang Wu, Teng Pang. Yan Zhang, and Yilong Yin. Diffail: Diffusion adversarial imitation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[35] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou Diffusion policies expressive policy class offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[36] Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
[37] Brian D. Ziebart, Andrew Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI, 2008
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No]" provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"), In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction should clearly state the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 7
13

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
14

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /ttppp.cc/pbllc/guudeeee http://odeubbbiiiooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Standard Err is used
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attts::/nneurpps wttp:/rppp...........ceeeeeeaeea
Answer:
[Yes]
Justification:
We preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader Impacts
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No License Problem
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy /tte::////errrpssccc///feeeeeeep for what should or should not be described.
19

---

A Details of the Derivation
A.1 Derivations of Eq. and Eq. 10
In this section, we provide detailed derivations of Eq. 9 and Eq. TT). Given the expert sample 1:N (1 17"),
70 ~ PE. the reverse process po po(r" (r'). and the corresponding forward process
B logo
E, (E log
i=
i Pou 7- 1r+)
log pol ( (- 1
logo
Pade (-
po ( (
=Brsope
log o
( log
Pau (---1|-1)
E -p log po (T (7(-1177) 77))
EBaape
log a
Paue (T+ i 117))
4pi log po - 170)
Eya-pg
E logo
(concavity)
pa (T (r--1 1
2(T--1/1 70)
Q(T--1177 177
B logo (EE 1 log P (
log
pa po(T(-1(79)
(13)
r²) The posterior q(7"~177', ,-',,1))))))) is tractable via Bayes' rule, since both 1(7"~1|7") and q(-())=-1,
q(''(r' are Gaussian distributions. The posterior q(7 (7--11pp | 70) can be written as the following:
1 mo) (",,)''7))))331
where (x(7',79)=- -
(+-
2(T',79) = r'=vor VI-a, = e. If we parameter-
ize the model po as (NPo o), then the KI. divergence between these isotropic Gaussians reduces to the squared error between the means: ( 7²) us (T, i)ll2. Ho et al. [15] proposes
to parameterize (i0 using a predicted noise ca, such as:
(+
B (0(T'))) ,0)
ns ( ()
VT-
v1 -1-a a
Substituting this into the squared error gives:
[rs("' 79) po(7"
B1
2(7" :00))
ue((i-0))
3
vI a
1
VI Br (+'- B
7'
c .2)
VI
Br
vI ai
-I-a
B:
{ = ,0)
(1 (1)
- 0z)
Therefore, minimizing the KL divergence equivalent to minimizing ||e - (- Substituting this into Eq. 13 leads to the following equation:
B
log a N C. (Ile Eguut (r), :)]1 - ca(T)))))3)
Here, C. (1- e A(O, 1). and -¹ ~ q(T"(>"), thus في = Vast"+ (1- a.)e. Follow- we consider the weight N C. as fixed constant value over in practical
ing Wallace et al. 33 implementation. Similarly, given the generative sample 70 Pous, the reverse process Paea )==
and the corresponding forward process qou (fI:N = ((FliN(=9) (since they have the same
20

---

variance schedule),
N
Pipoul
By Bposs
loga
B B log
Po(
(fi |
N
Poou (T
po (i)) ( |
=Byarppoll
logo
Σ
log
am1
Ngg.N~((FL log Pauu (7+ (71)
log
po po(
N Σ
Bez 1 1151,00) log nad (Fí
log a
add ( (
fof NE log Paue 1 |
2 Bpa
Binu((..N,,Fi+ log
(concavity)
pol p= (-(-1(7()
1 1 =0)
117 (()))))
Byd Ppous
BLuuLL,) log
10, log po (7*
log
Pauu (7' (7t-1171)
Epo logo (N. C, (lle co(F), ||c Eguu (F1 :)]{3) ]
C [1-A](1-a)) (O,, I) and F ~~ g(F')70), thus, 7t va,+0+ (1 - az)e.
A.2 Monotonic Improvement
In this section, we show that maximizing the surrogate objective guarantees a monotonic improvement in the original training objective. Revisiting Eq. (8) we denote the lower bound by o0r. where
0g gold.
I pa Pa(
fo(7") log
[Eyii Megtinggr log
I1; pap (7--1/71)
2 M logo [NE f)r,,r)) log po(T(-117') paz (r
90 (77).
Since loga() concave function, the lower bound (r)) is always less than or equal to the original fo(7") for all 0 E O: (1) fe(r²) Z g0n.0 (70.) Moreover, when 0 Os, the log term in both sides becomes zero, yielding (2) fer (r9) go, 02 (7 log 1/2.
If we maximize (r instead of f)(r"):
Ok+1 arg max 9o, 00 (TT))
BEB
then the following inequality holds:
So (7") 2 90. 11 (7") 2 90 (7 = So, (7
Therefore, this procedure guarantees monotonic improvement of the original objective.
A.3 Comparison with DPO-Diffusion
33 is algorithm for aligning diffusion models with human preferences by considering ranked pairs (7 7 that indicate a preference for T over 7 Built upon the Bradley- Terry (BT) model and the bijectivity between reward and policy, DPO-Diffusion optimizes the diffusion model Po with reference distribution Pret via:
DPO-Diffusion
I:N log pa (
E. log (G
log
(14)
Pret T1:N
Pre Pre((T!:NN)
21

---

where Tul --((-> and T!.. ~g(-1T)). This objective function can be reformulated in terms of noise prediction
E log o [[[lew Ge(tt i)ll -llew co( f)ll + ||e: -lcr-co co ( O - ||e: - (1a (1)(7))))))))
(15)
(r (p.17)
where en and correspond to Tue and rf. eo and ére are the noise prediction network for pa and pref-
By comparing the training objective functions, we draw interesting observation: if we denote expert samples as Tu and generative samples
with the reference model corresponding to the generator. the sigmoid in Eq. (15) is applied the sum of the error difference on expert and generative samples. In contrast, DPAIL evaluates these two eπror differences in indvidual sigmoid functions.
This distinction arises from DPO- Diffffsiffss'ss derivation via the BT model, whereas DPAIL is based on the binary discriminator. Moreover, although DPO-Diffusion also aims to handle multi- distributions, it targets offline RL and requires preference data, setting apart from DPAIL's focus direct expert imitation without preference annotations.
B Action Execution and Diffusion Sampling in DPAIL
In DPAIL, the diffusion policy generates fixed-horizon sub-trajectories of length H. The resulting H actions are executed sequentially in the environment, so action sequence generation is performed every H environment steps. The action execution procedure is detailed in Algorithm 2 To condition on the current state at the start the sampling process, we overwrite the corresponding state variable each diffusion step with the current observed state. The sampling procedure is detailed in Algorithm
Algorithm 2 Action execution
Algorithm Sampling
1: 80 env.reset))
I: Observe the current state 8t, T IO(,
2: for step = 0, 1, 2... do
2: for diffusion step N..... do
3: if t%H == 0 then
(0,
4:
Sample actions H::b ~ Pan (-1st)
4:
VI-A (T'-
81 vI-a; Soma(T', (T',))) 0zz
5: end if
6: a, + Get (t%H) action in ag.
5: Replace the initial state in 7 with St.
7:
Tt, env.step(a;)
6: end for
8: end for
Get action sequence do:h from 70.
C Environment Details
HalfCheetah-v3 and Walker2d-v3 The goal of these tasks is move the agent forward and backward along the x-axis as quickly as possible while maintaining balance. The state includes joint angles, angular velocities and the x-coordinate. Each expert trained using reward function based on the the forward reward, ex-coordinate velocity.
Ant-v3 and AntGoal-v3 The goal of these tasks is to control a four-legged ant robot to move forward, backward, left, right fast as possible while maintaining balance (Ant-v3), and to navigate to one of eight target positions evenly distributed around a circle with a radius 20 (AntGoal-v3). The state includes joint angles, angular velocities, and the (x.y) coordinates. Each expert trained using a reward function based on the the forward reward, #x-coordinate velocity dy-coordinate velocity in Ant-v3. In AntGoal-v3, each expert is trained using reward function based on the distance between the goal and the current robot's position. We use the 10 trajectories per mode expert demonstrations, with each trajectory consisting of 1 transitions.
maze2d-mediume and maze2d-large-vi navigate to one of the target positions. In the medium-sized maze, target positions are ((1.0, 6.0), (6.0, 5.0), (6.0, 1.0),, while in the large-sized maze, they are ((1.0, 10.0), (3.0, 8.0), (7.0, 10.0), (5.0.
The goal of these tasks is to control a point robot to
22

---

4.0), (7.0, 1.0)). Expert demonstrations are selected from D4RL dataset We use the 15 episodes for maze2d-medium-v/ and 30 episodes for mazzeddlarge-vl.
D Implementation Details
Policy gradient method We use PPO 27 to train policies and GAE(A) to compute advantage in GAIL, DiffAlL., DRAIL and InfoGAlL,, The corresponding hyperparameters for PPO are provided in Table At each k-th iteration, we perform m-steps rollout in the environment. The corresponding hyperparameter settings for each algorithm are provided in Table 2
Diffusion and DPAIL Both Diffusion and DPAIL. utilize the same U-Net architecture with residual blocks consisting of temporal convolution and group normalization, following [17 We use N = 50 diffusion steps in both Diffusion and DPAIL for all tasks. Additionally, we normalize the state values before feeding them into the network. For DPAIL, we clip the norm value of ||e Egout (z' i)]l not to be larger than 0.2.
GAIL, DI, DRAIL and ASAF For GAIL, DiffAIL. DRAIL, and ASAF, we use a multi-layer perceptron (MLP) with two hidden layers of size [64, 64] for the Gaussian policy. We also normalize the state values before feeding them into the policy network. The discriminator in GAIL is MLP with two hidden layers of size [100, 100]. The discriminator architectures both DiffAIL and DRAIL are based MLP U-Net structure based on the official repository and N 50 diffusion steps.
InfoGAIL For InfoGAIL, to for all tasks. We concatenate the one-hot encoding of the latent variable with the state and use the resulting vector input to a Gaussian policy. We also normalize the state values before feeding them into the policy network. The discriminator network and class prediction network InfoGAlL. share MLP with two hidden layers of size (100, 100) and output the corresponding values. For the coefficient of unsupervised regularization term, we perform a greedy search over the range [0.1, 0.2, 0,3, 0,5].
discrete latent variables, setting the number of latent variables
Form of the reward in GAIL, DiffAIL,, DRAIL and InfoGAII. For Mujoco tasks, we use a commonly adopted reward function of the form r(s, a) - log(1 D(s,a)), which acts survival bonus, encouraging agents to survive longer in the environment to accumulate more rewards. For Maze tasks, we use the reward function r(s, a) log(D(s, a)). which serves as a penalty signal. This is well-suited for goal-reaching tasks, as it incentivizes the agent to reach the goal as quickly as possible. In AntGoal-và v3, we adopt the survival-style reward r(s, a) = log(1 D(s, a)) and we find this to work well in practice.
Details of ASAF ASAF aims match the trajectory distribution under stationary policy (a)s). For n. the trajectory distribution (=(T) is decomposed as P=(T) P(so) _ 1 ((arls)) ss, at). To optimize Eq. 4 for trainable policy wo and generator policy TG. ASAF defines the discriminator in policy space as
Dx Digdesso No (T)= = (1og log PiT (T)
Prr (7) )
Σ loggl((cl))) - log ag (aolst)
(16)
where the transition probability P(s at) cancels out, leaving only to the ratio of policy terms. In practice, ASAF segments trajectories into windows of length w, updates ny via binary cross- entropy loss, and then sets Tg-u as the updated Te to the next iteration. This procedure iteratively updates ve until convergence. We offer the ASAF algorithm in Algorithm
hhtps:/gittuu..om//araa wtwm//araaaaaommdatttonnonaaaaa
hhups://githubb "tanps//gituub..om/jmmnnn com/janner//diiffuee Putps///iitubb.co/////aaooo
23

---

Algorithm Adversarial Soft Advantage Fitting (ASAF)
Input: expert trajectories DE = { Ny Randomly initialize no, and set Tguas t- TTpp for = {o... K} do
Nguus
Collect trajectories Daau Jn= using aaas by interacting with environment Update 02~1 by optimizing the following loss:
0x+1 = arg max Er [log Dx Dagueino (T) + Er [log (1 - D gois se (r))
where Dr (T) is defined in Eq. (16
Paae 5 10
end for
Table 2: Hyperparameters used for baselines across various environments.
Method
Hyperparameter
HaliChectab-v3
Walker2d-v3
Ant-v3
AntGoal-v3
mazc33-mcdium--1
maze2d-largo-vl
0.0002
0.0002
0.0002
Diffusion
borizon Ff Epoch
4
4
4
16
16
1000
1000
policy Ir
0.0003
0.0003
0.0003
0.0003
0.0003
0,0003
discriminator
GAIL
8 rollout length Iteration
50000
50000
50000
50000
10000
10000
policy Ir
0.0003
0.0003
0.0003
0.0003
0.0003
0,0003
discriminator à rollout length m lterationn JC
0.0002
0.0002
0.0002
DiffAIL
50000
50000
50000
50000
10000
10000
policy Ir
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
DRAIT.
discriminator # rollout length Iteration
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
50000
50000
50000
50000
10000
10000
policy discriminator coef MI rollout length a Iteration &
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
InfoGAIL
0.2
0.2
0.1
0.1
0.3
0.3
50000
10000
10000
200
200
10000
600
100
100
policy Ir
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
ASAF
window
64
64
64
64
# rollout longth m
50000
50000
50000
50000
10000
10000
# Iteration K
200
200
400
400
100
100
Ir harizum f!
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
DPAIL
16
# rollout length on
10000
10000
10000
10000
5000
5000
# Iteration K
200
200
200
200
200
200
Table 3: PPO training hyperparameters used for each task.
Hyperparameter
HalfCheetah--33
Walker2d-v3
Ant-v3
AntGoal-v3
maze2d-medium-vl
maze2d-large-vl
clipping range e discount factor y
0.2
0.2
0.2
0.2
0.99
0.99
0.995
0.995
gac parameter X
0.97
0.97
0.97
0.97
0.97
0.97
# epoch per iteration
50
50
50
50
40
40
24

---

E Additional Experimental Results
We present the learned behaviors of baselines methods and DPAIL in Figures and 7 demonstration behaviors are visualized in Figure 5 of imitation learning methods, we measure the entropy the learned behaviors to quantify diversity. Additionally, to assess the similarity between the learned trajectories and expert demonstrations, compute Maximum Mean Discrepancy (MMD) between their respective state-action distributions, using RBF kernel with 20 bandwidths, as shown in Table Since MMD quantifies the divergence between distributions, lower values indicate better recovery of all modes present the expert distribution.
The expert To evaluate the multi- learning capability
200
-103
-200
HalfCheetah-v3
(c)
(d) AntSoaliva
manedd medium-v1
large
Figure 5: Expert demonstrations across 6 tasks.
Me dediuum
Large
(a) DRNIL (Ours)
(b) GAIL
DifFAIL
[d) DRAIL
(el InfoGALL
(1) ASAF
Figure 6: Learned behaviors baseline methods and DPAIL (ours) in Maze2d tasks. The first row
depicts monneddmedium-vll while the second row depicts maze2d-large-vi. Each graph illustrates different trajectories generated by the same policy. The initial position is marked with circle, and the goal positions are marked with stars.
Walker2d
200
300
300
200
200
-200
-20
-200
-22
-200
(a) DRAIL (Ouns)
[b) GAIL
(c) DHÍAIL
(c) DRAIL
(e) InfoGAILL
(f) A5AF
The first row shows Walker2d-v3, where the task is to move forward and backward (#xxaxis). The second row shows Ant-v3, where the task move forward, backward, left and right (#x-axis, #y-axis). Each plot illustrates ten different trajectories generated by the same policy.
Figure 7: Learned behaviors of baseline methods and DPAIL (ours) in MuJoCo tasks.
E.1 Impact of the Number of Demonstrations and Modes
provide additional experimental results on the impact of number of demonstrations and modes in Ant-v3 (Table 5) and AntGoal-v3 (Table In both tasks, increasing the number of modes generally degrades the performance of most algorithms. However, DPAIL exhibits greater robustness, benefiting from the expressiveness of diffusion models.
25

---

Table 4: MMD(L) between state-action distributions between expert demonstrations and learned be- haviors. DPAIL has the lowest value most tasks, indicating better recovery of expert distributions.
Environment
BC
Diffusion
GAIL
DiffAlL
DRAIL
InfoGAIL
ASAF
DPAIL
HallCheetah-v3
0.029
0.027
0.038
0.038
0.045
0.039
0.031
0.025
Walker2d-x3
0.127
0.091
0.165
0.182
0.212
0,099
0.190
0.096
Ant-v3
0.334
0.018
0,281
0.341
0.335
0.282
0.311
0.012
AniGoal-v3
0.571
0.082
0.385
0,443
0.401
0.192
0.551
0.021
maze2d- 1
4.9e-4
9.1e-5
4.8c-4
4.2e-4
4.5e-4
5.0e-4
5.5c-4
8.0e.5
maze2d large vl
3.1e.3
1.0e44
3.0e 4
1.5e
5.0e 4
5.3e
5.9e-3
9.2e.5
Table 5: Normalized score (Score) and entropy (Ent) on varying the number of demonstrations and modes in A3t-v3.
of moddes
# of demos
metrics
BC
Didfusion
GAIL
DiHAIL
DRAIL
InfoGAIL
ASAF
DPAIL
Soore
0.13:0m0
0.38
0.39
0.50.:..00
0.37.0.12
0.23+0.01
0.27
0.48
Fat
0.000000
0.00+
0.00-0.00
0.00.000
0:00-0.00
0.00-0.00
0.00+000
0.00-0.00
Suore
0.15=0006
0:52025
0.39
0.55+002
0.22
020:000:
0.22
0.53
bat
0.0000000
0.00
0.000000
0.00:0000
0.00
0.00=0000
0.00
0.00:0.00
10
Fatt
Score
0.1720.00
0.0000000
0.27
000
0.50
0.0000000
0.50
0.00
0.00 0.00
00.28
0:23:00000
0.00=00000
0.27
0.000
0.67
0.00-00000
20
Score
Fint
0.00=0000
0.1020.0
0,74
4,00
0.00
0.1100112
0.69
0,00+000
0. 0.00
0.25
0.00=0.00
0.25
0.65
0.00 +0000
0.72
0.00
Score
0.10
0.40
0.17
0.
0.28
0.12
0.10
0.47
Fast
0.75-0.9)
0.60
0:36-0030
0.20-10
0.52+0222
0.42
0
Soore
0:14:0066
0.4
0.17:625
0.13
0.26
0.15:00:
0.34
0
2
10
Ent
Score 0.18200:15
Ent 2.71
0.882000
0.521000
0.69
0.59
0.18
0.25
0.3
0.52
0.26-10.29
0.66:022
0.31 20.23
0.31 +0.18
0.19
0.13:20.22
0.53
0.65
0.15
0,38
0.46+00:2
0.00
0.00
0.60
20
Eat 2025
Soore 0.15:000
0.06
k.09
0.44
0.16
0.51
0.
0.36
0.18
0.52
0.12:001
0.43
0.12
0.09
0.63
Swore
0.0-0000
0.13:
0.12+001
0.08 =008
0:0000001
0.53
last
0.322032
0.85 10.11
0.76=0.17
0.82
0.98 =0.10
0.95 0.05
0..12
1
Soore
0.060005
0.44
-031
0.1
0.000005
0.03-000
0.01 0002
0.55 00.0
Bat
0.0:0046
10.5
0.730041
0.68
0.57 0..30
0.
0.79
0
L0
Score
0.06:000
0.4300177
0.120.11
0.13+011
+0,5
0.02
0.01
0.15
Fatt 0.00-6.2
0.91 0015
082-0.18
0000
0.69
0.90-
0.75
1 +0.50
20
Swore 0.00
But
1.00
0.88
0.60
083:0014
0.11
0.15
0.96
0.00
0.1220..0
0.000
1.06
0.65
0.05
0.14
20.14
Suore
0.21
00800000
0.02+000
0:02
0.00
0.39
Ent
0.90=0.21
1,18
085
1,22+00s
1.0 0.050000
1.07 00
1.01 +0.19
1.20=0.11
Score
0.0900.01
0.28
0.0000000
0.04
0.07 20.09
0.022.00:
0.00
0.42
Fast
0.5500.2
0.17+006
0.5-0.00
1,31 0004
1.07 0..70
-.19-0.0
0.89 0.16
1.20
10
Ent 0.75
Score 0.06
0.4810111
1.22
0.11
1.19=0.10
000
10
0.02
1.26
1.04
0.03
0.00
1.07
0.56
1.21
0.050005
0.60+002
000000000
0.03
0.02.am.
0.04
Ent
1.13
1.29
1.30
+.30
0.30
0.87
0.15
Table 6: Normalized score (Score) and entropy (Ent) on varying the number of demonstrations and modes in AntGoal-v3.
modess
of demos
metrics
BC
Diffusion
GAIL.
DiEAAIL
DRAIL.
InfoGAII.
ASAF
DPAIL
3
Ent
Soore
0.
0.0020.00
0.18
0.00
0.740008
0.00=0.00
0.94:00:
0.00+0.00
0.00
0.78
0.62=0044
0.0020.00
0.00
0.40
0.80=0.02
0.0020.00
5
Score
Bat 0.000000
0.3:
0:001000
0.651021
084-0.00
0.000000
0.96
0.001000
0.00
0.68::000
0.00:0.00
0.00:0000
0.43
0.3:0.08
0.0:0.00
10
Ent 0.0000.00
Soore
0446-0.14
0.79
0,00
0.0000.00
0:92-000
0.94:002
0.00
0.93-004
0.00
0.00
0.64:011
0.47
0.00
0.
0.00
Store
0.52=020
0.860005
0.94:001
0007
0.65
0.53
0.00
batt 0.0000000
0.00
0.0000000
0.00
0.00
0.00=0.00
0.00
0.00
Score 0.080000
0.51 2008
0.68
0001
0.72
0:45
0.02
0.00
Fist 0.50
0.4
02200.177
0009
0.23 +..10
0.54-021
0.51 +0.27
0.68 +0.12
5
But 0.56
Scoree 0.1020.0
0.4
0.542011
0.15:0.11
0.74
0.86
(164 0005
0.73
0.20
0.64:0.17
0.13
0.33
0.04
0.71 10.17
2.70
10
Fat 0.68-0.10
Score 0.1320.00
0.58
0.424034
0.17-0.17
0.77
0.80
0.53
0.72
0.68
0.49
0.65+035
2.03
0.06
20
Ent 0.54.0.13
Soore 015:0000
0.33
0.8
0.3
0.156.11
0.86
0.62+009
0.11
0.76
0.11
0.77
0.16
0.517
0.11
0,87 0.04
Score
00:000000
0.37
0.52=6.13
0.22
0.54
0.40:022
0.02
0.64
Ent
0942000
0.99
1.02=0.28
1002
I.
1.04
0.
0.92
1..4
Soore
003-0.0
0.3
0.54-0.0
19 19
0.50
0.0
0.66 00000
Ent
0.720.00
1.0
1.000.30
1.09
20.18
0.13
0.8220.30
1.22
Store
000
0.45
0.79099
0.66
0.67
0.52=0114
0.15
0.05
latt
10020007
1.05
03820:111
0.94
0.96
0.22
0.56
1015
20
Fur 0.5:
Soore 0.2-0.00
1.00
0.30
0.54-0:09
1.00
0
1.09
0.50
1.05 +0.1
1.23
0.38-
0.03+0044
0.33
0.66 +000
1.16
3
But 136
Score 0.020000
0.20
L501033
04330.111
0.27
0.3
1.
0.39
1.29
0.39.0.11
1.77
2.00
1.4
0.52
1.70=0.27
Suore 003-0000
0,23
047=000
0.40
0.5
4:00
0.5
Fat
0.8-6.5)
1.56
1.78-0.17
1.81
-.79
1.65
1.7-0041
Suore 0012004
0.22
0.58
0.35
=0.15
0.18
0.01 20.00
0.00
Ent 1.4620.88
1.52
1.31
1.75
1.77
0.
1.57
20
Score 0.0100000
Fat
+.51-0.54
0.53
1.18
0.58
1.30-0.14
0.45
1.70
0.41 20.10
+.75+0.23
0.46
1.79-0.1
1.46
10.00
0.74
1.78 -044
26

---

F Derivation with latent variable 2
We define conditional diffusion models with an latent variable 2, where the joint probability po(7O:N factorizes as:
Po(T .2) ((T II [Po(T
where 9(2) is the prior and (( is the standard normal distribution. To evaluate the log-ratio log (o() (r0)) we rewrite with the posterior ((2)79) as:
log (T
I ((- ((2179) po(T)) I po =) ((2) p(r~) I Paua (r 2) q(2) q(TNN
Paua (TO) I ((T))//11) ((2 Paial (TO)
where ((T") and (2) cancel out. Taking expectation of this log-ratio over F I:N q(-17 2
(-117)) does not change the value.
Il po Po 11'.2) 22) I Paou (r 1 2)
max PON
logo
log
Given the expert sample 7" ~ pr, the approximate variational posterior (=(2)="), the reverse process po(ri-117?, 2), and the corresponding forward process ((r1:N179) these lead to the following lowerbound:
Po ----111111 (T',2) 2) I Pauu (7 1 1 ,)
Eya-pg
log a
E (a1o)
log
ydud matt (+o
pol (T | 2) pau - ; z)
log
E
log
plid Pi (100)
po ( [T(,2) 2))
=Epampe
log
Σ EE
log
Pare (7 (T 1|-,, 2)
ii:h No(" (NN 1ro)
po ( 1 | 2)
= Epaope
log a
zm 70) i >''ng(r')) "9(7" 17
log
Paua (T' 1|7', 2)
log NNEi Lng(ti I log Po (7- 17/ (T(,2)
Epo
E
(m)((1,)).
Paue (7- 1 2)
(concavity)
2~go(=17") (((")())))+))
q(T"-217' 17) 70) log Daue P (7--1 2) po( 2)
|r' (0)
Eya-pe
E (n logo 1ight log
) ~(0(1(1))))))))
C. (0, Gold, -old,>0,22 ,0,=) = Eyo,zi. log o NN Ci (lle - Egoul 2 -||- = -
go (=( NOO, I), and و³ (777) thus 7 Vait (1-ar L..e,). Similarly, for given generative samples F ~ Poon, we can also get C(PAn (0, gold, =0, 2). The overall objective becomes:
Here, C. (1-A)X(-a.)'
c (0, gold x7,2)) E [log g0(70.20 2) ((7)))
[-beAn (0, @ gold 7 +,,))) +E F0 Pimpposs
E.E +OmpEs 20go(=1170
soio (3/70)
where the last term encourages learning meaningful unsupervised representation go(=))) by maximizing mutual information.
27


Reviews:
[{"review_id": "PSJJ3bi0xn", "reviewer": "Reviewer_96w4", "summary": "The paper studies bridging diffusion-based and adversarial-based imitation learning strategies. The key technical contribution lies in deriving a novel learning objective which integrates the adversarial soft advantage fitting (ASAF) objective into the diffusion model optimization. The proposed approach is tested under MuJoCo and Maze2D environments, justifying that the proposed approach brings the benefits of diffusion and adversarial approaches and outperforms existing imitation learning baselines.", "questions": "Please feel free to response to the issues listed in the weaknesses part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths:\n\n1. The proposed approach is based on sound derivation of variational lower bound. \n\n2. The performance gain shown from the experiments is solid. \n\n- Weaknesses:\n\n1. The motivation needs further clarification. The paper does not provide clear explanations on why bridging diffusion and adversarial approaches is important. Does this help to address some specific open problem in imitation learning, or just bring the benefit of both approaches together? Answering this question is important for justifying the importance and value of the work.\n\n2. The paper does not clearly explain the key technical challenge that the proposed approach tackles. Intuitively, a straightforward approach to bridging diffusion and adversarial learning is to utilize diffusion model as the policy, and directly add both diffusion and adversarial training objectives together to form a single objective. Why we need an alternative method derived in the paper? Justifying this would be important to evaluate the technical contribution in the paper.\n\n3. Even though the results shown in the experiments verify the performance gain of the proposed approach, the benchmark tasks are relatively standard ones, which do not sufficiently show that the proposed approach can address some significantly challenging tasks that previous approaches fail to solve. \n\n- Overall justification:\n\nOverall, even though I think the paper is relatively technically sound, the weaknesses discussed above make me feel that the novelty and contributions of the paper are not sufficiently justified.", "was_revised": true, "final_justification": "Overall, I think this a technical sound paper which would be of interests in the imitation learning area. The author responses address my concerns in the reviews, thus I increase my score.", "rebuttal": "**(Weakness 1) Motivation**\n\nIn real-world settings, expert demonstrations are often limited in quantity and exhibit multi-modality. For example, there may be multiple distinct but valid strategies to solve the same task. Diffusion models are particularly well-suited to capture such multi-modal behavior due to their strong generative modeling capabilities. However, when only a few expert demonstrations are available, purely supervised learning on these data can lead to overfitting even for diffusion models.\nTo address this, we bridge diffusion modeling with adversarial learning by leveraging diffusion-generated samples as negative examples. This enables the model to contrast expert data against generated ones, thereby promoting better alignment with expert distribution[1,2] and enhancing robustness[3] and generalization[4,5] of the learned policy.\n\n[1] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, Chen et al.,ICML 2024.\n\n[2] Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization, Barde et al., NeurIPS 2020.\n\n[3] On the Convergence and Robustness of Adversarial Training, Wang et al., ICML 2019.\n\n[4] Adversarial Training Helps Transfer Learning via Better Representations, Deng et al., NeurIPS 2021.\n\n[5] Improved OOD Generalization via Adversarial Training and Pre-training, Yi et al., ICML 2021.\n\n **(Weakness 2) Necessity of an alternative approach for diffusion policies in AIL**\n\nAIL methods typically involve training a discriminator and a generator in a minimax objective, which often leads to training instability. In particular, when the discriminator becomes too accurate, it can cause vanishing gradients for the generator, hindering the learning process.\nWhen applying such adversarial training to diffusion policies, these issues are further exacerbated. Specifically, computing policy gradient based on reward signals requires backpropagation through the entire diffusion steps, resulting in substantial computational cost and additional instability [6,7]. \nTo address these challenges, several recent works have been proposed. For instance, [6] introduces a technique that gradually increases the number of trainable diffusion steps, while [7] proposes a reverse sampling method that estimates the mapping from noise to clean data to stabilize training.\nHowever, these techniques still require careful scheduling and additional sampling, making the training process complex.\nIn contrast, our proposed method, DPAIL, builds upon the ASAF framework, which allows us to sidestep these difficulties. By eliminating both the explicit discriminator and policy gradient-based optimization, DPAIL enables stable and efficient training of diffusion models, allowing them to be effectively applied to multi-modal generation.\n\n[6] Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards, Hu et al., CVPR, 2025.\n\n[7] Efficient Online Reinforcement Learning for Diffusion Policy, Ma et al., ICML, 2025.\n\n**(Weakness 3) Evaluation on challenging manipulation task**\n\nWe conducted additional experiments on two challenging environments: FrankaKitchen and adroit, which go beyond standard benchmarks and better reflect the complexity of real-world tasks.\nThe FrankaKitchen environment involves goal completion tasks that require the agent to perform household activities, such as opening a microwave or turning on a light switch.\nFor experiments, we collected a mix of 5 trajectories per task for 4 tasks: ‘microwave’, ‘kettle’, ‘light switch’, and ‘kettle’. \nIn addition, we evaluate DPAIL on the adroit pen manipulation task, which requires a dexterous robot hand to control and orient a pen. This task is particularly challenging due to its high-dimensional and contact-rich nature. We used 25 human demonstrations collected from [8].\nThe normalized returns are shown below:\n| |BC|Diffusion|GAIL|DiffAIL|DRAIL|InfoGAIL|ASAF|DPAIL|\n|-|-|-|-|-|-|-|-|-|\n|franka-kitchen-v0 |0.09 (0.12)|0.52 (0.05)|0.32 (0.02)|0.43 (0.13)|0.56 (0.07)|0.07 (0.08)|0.07 (0.11)|**0.69** (0.10)|\n|ardoit-pen-v0 | 0.20 (0.19) | 0.89 (0.06) | 0.01 (0.00) | 0.01 (0.02) | 0.01 (0.04) | 0.00 (0.01) | 0.40 (0.13) | **1.21** (0.19) | \n\nThese results demonstrate that DPAIL consistently outperforms baseline methods, validating its effectiveness in realistic scenarios.\n\n[8] D4RL: Datasets for Deep Data-Driven Reinforcement Learning, Fu et al., arxiv."}, {"review_id": "19OSCtJcBv", "reviewer": "Reviewer_9YY9", "summary": "This paper introduces DPAIL, a novel framework for Adversarial Imitation Learning (AIL) that utilizes diffusion models as a policy function. The core innovation of DPAIL lies in its ability to learn diverse expert behaviors without requiring policy optimization, a common and computationally expensive step in traditional AIL methods. By building on the Adversarial Soft Advantage Fitting (ASAF) framework and introducing a tractable lower bound for training diffusion policies, DPAIL overcomes the limitations of mode collapse or instability when dealing with complex, varied demonstrations. The research demonstrates DPAIL's effectiveness in capturing diverse behaviors and maintaining robust performance across various continuous control and navigation tasks, even with limited demonstration data.", "questions": "How does the scaling factor $N$ affect optimization? Does clipping the denoising error itself suffices?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n+ The paper introduces the AIL framework using diffusion models as policies without requiring policy optimization based on the ASAF framework. \n+ The paper derives a tight lower bound on the ASAF objective, enabling efficient training through the minimization-maximization approach. \n+ The paper presents extensive experiments to demonstrate better performance than baselines works. \n\n### Weakness\n- Though the proposed approach does remove policy optimization for improved training efficiency, the proposed approach does come with higher computational cost and longer inference time. \n- The scaling $N$ in eq. (9-10) is making the loss inside the sigmoid function large. This likely leads to the plateau region of the sigmoid function, hindering optimization.", "was_revised": true, "final_justification": "Thank you for the authors for the detailed responses. I feel my concerns have been addressed and my original rating is reasonable. I would maintain my score.", "rebuttal": "**(Weakness 1) High computation cost and longer inference time**\n\nAs noted in the section on limitations, DPAIL incurs longer inference time due to the use of diffusion models as policies. This is a trade-off for their high expressiveness and ability to capture multi-modal behaviors, which are essential for our goal for modeling diverse expert demonstrations.\nThere has been promising progress in accelerating diffusion model inference (e.g., through distillation or fast samplers), and integrating such techniques into our framework is a promising direction for improving its scalability and real-time applicability, which we leave as future work.\n\n**(Weakness 2 & Question 1) Scaling factor $N$ inside the sigmoid function**\n\nIn our experiments, we set $N=50$, and we did not observe any numerical issues or optimization instability resulting from scaling factor $N$.\nAlthough $N$ increases the magnitude of the input to the sigmoid function in Eq. (9-10), this did not push it into the flat region of sigmoid in our experiments.\nThis is because DPAIL resets the parameter via $\\theta^{\\text{old}} \\leftarrow \\theta$ at every iteration $k$, which causes the difference in noise prediction errors, the input of the sigmoid, to be zero at the start of each iteration. \nAs a result, the input to the sigmoid remains within a manageable range. Specifically, for N = 50, the value typically lies within [0.0~8.0], where the log-sigmoid still provides meaningful gradients.\n\nFor larger scaling factors (e.g N $\\geq 100$), we investigated whether clipping the input to the log-simoid function would be necessary for the optimization. To assess this, we conducted additional experiments with N=100, both with and without clipping the input (e.g to a maximum value of 9). \n| task      | HalfCheetah | Walker | Ant | AntGoal | maze2d-medium | maze2d-large |\n|-|-|-|-|-|-|-|\n|       clip | 1.02 | 0.79 | 0.53 | 0.69 | 1.00 | 1.00 | \n| wo clip | 1.01 | 0.81 | 0.57 | 0.67 | 1.00 | 1.00 |\n\nIn practice, we observed no significant difference in performance between them, likely because the input to the log-sigmoid resets to near-zero at the start of each iteration. However, the clipping may still be beneficial when using very large values of N for the effective optimization."}, {"review_id": "XJnDMJulCc", "reviewer": "Reviewer_MA5d", "summary": "This paper introduces DPAIL, a novel adversarial imitation learning (AIL) framework that uses diffusion models as policies to better capture diverse expert behaviors in multimodal demonstration data. Building on the Adversarial Soft Advantage Fitting (ASAF) framework, DPAIL avoids policy optimization and instead uses a binary cross-entropy objective to distinguish between expert and generated trajectories. The method demonstrates good performance and robustness across various baselines, particularly as the number of behavior modes increases.", "questions": "Some symbols in the equation is not clearly defined. In Eq. (6), how is the equation satisfied? What is the definition of $\\sigma$?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. This paper is well-written and easy to follow. The literature review is comprehensive and most related works are discussed.\n\n2. This paper provides theoretical analysis to support the proposed objective function, which makes the method more convincing.\n\n3. Empirical results from different aspects demonstrate the effectiveness of the proposed method, with several highly related works.\n\nCons:\n\n1. This paper proposes a method that integrates the diffusion policy with the adversarial learning process. However, I do really think the authors' claim \"without policy optimization\" seems to be misleading. The first time I saw the title of this paper, I thought there was no policy update process within the adversarial imitation learning framework. After I read the whole paper, it seems that there still exists the optimization process of diffusion policy.\n\n2. The formulation of discriminator in Eq. 5 is defined by the optimal result of the inner optimization loop in adversarial imitation learning. How does this formulation make sense? I understand this formulation is given by [5], however, as this work builds the foundation of this paper, the authors should make this point more clear.\n\n3. I am also curious about the construction of multi-modal demonstration dataset. The authors claim that they use SAC to train the policy and select K expert policies during the training. This is more likely to be under the setting of imitation learning with imperfect demonstrations.", "was_revised": true, "final_justification": "During the rebuttal, most of my concerns have been resolved. I think this is an interesting paper with good quality. Though there are some clarification issues, it could be addressed at the camera-ready phase.", "rebuttal": "**(Weakness 1) Clarifying the phrase “without policy optimization”**\n\nThe phrase “without policy optimization” in our paper specifically refers to the absence of reward-driven policy gradient updates, which are commonly used in adversarial imitation learning frameworks.\nOur proposed method, DPAIL, does not perform RL optimization steps, such as policy gradients of value-based updates that maximize expected rewards.\n\nInstead, we train the diffusion policy using a lower bound objective derived from the discriminator, which does not incorporate any reward signal. As such, the optimization does not involve computing gradients of a reward-based objective with respect to the policy parameters. We will clarify this distinction in the paper.\n\n**(Weakness 2 & Question 1) Clarifying the discriminator formulation in Eq. (5) and Eq. (6)** \n\nIn Eq. (5), we define the structured discriminator $D\\_{\\pi, {\\pi'}}$ as\n$D\\_{\\pi, \\pi'}(\\tau) = {{p_{\\pi'} (\\tau) }\\over{ p\\_{\\pi'} (\\tau) + p\\_\\pi (\\tau)}}$, \nwhich matches the form of the optimal discriminator in the inner maximization of eq. (4):\n\n$\\max\\_D \\mathcal{L}(D, p_\\pi)  = \\max_D \\mathbb{E}\\_{\\tau \\sim p\\_E}[\\log D(\\tau)] + \\mathbb{E}\\_{\\tau \\sim p_\\pi} [\\log (1 - D(\\tau))]$. \n\nAt optimality, the derivative of $\\mathcal{L}(D, p\\_\\pi)$ with respect to $D$ should be zero for $\\tau$: \n\n$\\nabla\\_D \\mathcal{L}(D, p\\_\\pi) = \\frac{p\\_E(\\tau) }{ D(\\tau) } - \\frac{p\\_\\pi(\\tau)}{1-D(\\tau)} = 0$.\n\nRewriting this gives:\n$\\frac{1-D(\\tau)}{ D(\\tau) }= \\frac{p\\_\\pi(\\tau) }{p\\_E(\\tau)}$.\n\nTherefore, the optimal solution is :\n$D^{*}\\_{\\pi, \\pi\\_E} = \\frac{ p\\_{\\pi\\_E(\\tau)} }{ p\\_{\\pi\\_E}(\\tau) + p\\_\\pi (\\tau) } = \\frac{1}{1+ \\frac{p\\_\\pi (\\tau)}{p\\_{\\pi\\_E(\\tau)}}}=\\frac{1}{1+ \\exp(-\\log\\frac{p\\_{\\pi\\_E}(\\tau)}{p\\_{\\pi}(\\tau)})} = \\sigma \\left(\\log\\frac{p\\_{\\pi\\_E}(\\tau)}{p\\_{\\pi}(\\tau)} \\right)$,\n\nwhere $\\sigma(\\cdot)$ is the sigmoid function.\n\nThus, our definition in eq. (5) generalizes this form by replacing the expert policy $\\pi\\_E$ with a variable $\\pi’$. \nWe will clarify this in the paper.\n\n**(Weakness 3) How to construct multi-modal demonstration**\n\nTo construct a multi-modal demonstration dataset, we use K sufficiently trained expert policies, each using a distinct and different reward function. For example, in AntGoal-v3, each reward function corresponds to the negative distance to a different target goal position from the current position.\nWe collect demonstrations from all K experts and aggregate them into a single dataset. We consider all collected trajectories as perfect demonstrations.\nFor detailed reward functions, please refer to Appendix B (Environment Details)."}, {"review_id": "6RIpPVvQp4", "reviewer": "Reviewer_hTNL", "summary": "This paper introduces DPAIL, an adversarial Imitation Learning (AIL) framework that leverages diffusion models as a policy class to enhance expressiveness. Building upon the Adversarial Soft Advantage Fitting (ASAF) framework, which reformulates AIL to remove explicit policy gradients, the authors address the challenge that exact likelihood evaluation is intractable for diffusion models. To overcome this, they derive a tractable lower bound for the discriminator objective, making it feasible to optimize the diffusion policy within the ASAF paradigm. The resulting approach retains the expressiveness of diffusion models while ensuring the stability and efficiency of adversarial training. Extensive experiments on continuous control (MuJoCo) and navigation (Maze2D) tasks demonstrate that DPAIL can successfully imitate diverse expert strategies, maintaining high task performance and behavioral diversity, especially in environments with multiple behavior modes where traditional methods tend to collapse to a single mode.", "questions": "1.Could the authors clarify how hyperparameters for all baselines (e.g., GAIL, DiffAIL, InfoGAIL) were tuned? Were the same budgets and random seeds applied?\n2. If the same tuning effort was not feasible, what measures were taken to ensure a fair comparison?\n3. Do the authors have any preliminary ideas or results on integrating latent variables or guided diffusion for mode conditioning? Could such an extension be compatible with the current ASAF lower bound design?", "limitations": "The authors have explicitly acknowledged the main limitations of their work in the dedicated Limitations section. They correctly point out that:\n(1) The iterative sampling cost of diffusion policies introduces significant computational overhead at inference time compared to simpler policy classes.\n(2) The method currently lacks latent controllability, making it harder to guide the imitation policy towards a user-desired behavior mode compared to methods like InfoGAIL.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The paper is technically solid overall, providing a well-motivated derivation of a tractable lower bound for training diffusion-based policies under the ASAF framework. \n2. Experimental results are extensive and conducted on a variety of benchmark tasks (MuJoCo, Maze2D) with different levels of multi-modal behaviors, and the metrics include both normalized returns and entropy for diversity, which are appropriate and relevant. The implementation details are carefully discussed, and the paper provides reproducibility information, e.g., hyperparameters, baseline setups.\n3. The method addresses a practical challenge in imitation learning—efficiently learning from diverse demonstrations without costly policy gradients—an issue of broad interest in the field. Combining the strengths of AIL (robust generalization) and diffusion models (multi-modal expressivity) can have significant implications for robotics and other sequential decision-making tasks.\n\nWeaknesses：\n1.The method’s practical scalability and applicability in real-world scenarios with strict latency constraints remain somewhat unclear. \n2. Hyperparameter sensitivity is only partially discussed. \n3.While the integration of ASAF with diffusion models is novel, parts of the loss design (e.g., using noise prediction errors) share structural similarities with prior works like DRAIL and DiffAIL. The paper could strengthen its argument about how its approach is fundamentally different.", "was_revised": true, "final_justification": "After considering the rebuttal, author clarifications, and discussions with other reviewers , my recommendation remains Borderline Accept, with the following reasoning:\n\nResolved Issues：\n(1) Differences from DRAIL / DiffAIL – The authors provided a clear and convincing explanation of the fundamental differences, particularly the implicit discriminator formulation in DPAIL, which justifies its ability to capture multi-modal behaviors.\n(2) Hyperparameter Tuning Fairness – Detailed descriptions of the tuning process for baselines (including rollout lengths, iteration counts, and seeds) addressed concerns about fairness in comparisons.\n(3) Sensitivity to Additional Hyperparameter – Additional experiments showed performance trends with varying generated sample counts, clarifying that the method is reasonably robust once above a threshold.\n\nRemaining Concerns:\n(1) Inference Latency and Real-World Scalability – The method’s reliance on diffusion policies still results in noticeably higher inference time, with no concrete acceleration strategies implemented; this limits immediate deployment potential.\n(2) Evaluation Scope – Experiments remain focused on simulated environments; real-world or noisy/heterogeneous scenarios are not tested, leaving generalizability uncertain.\n\nI assign strong weight to the technical novelty and solid methodological grounding, as the rebuttal clarified key differentiators and experimental fairness. However, the limited evaluation scope and scalability concerns are significant practical drawbacks, preventing a full accept. Overall, the paper is technically sound and makes a meaningful contribution, but the above limitations temper my enthusiasm, leading to a borderline accept recommendation.", "rebuttal": "**(Weakness 1) Inference latency and reward-world applicability**\n\nAs noted in the section on limitations, DPAIL incurs longer inference time due to the use of diffusion models as policies. This is a trade-off for their high expressiveness and ability to capture multi-modal behaviors, which are essential for our goal of modeling diverse expert demonstrations.\nThere has been promising progress in accelerating diffusion model inference (e.g., through distillation or fast samplers), and integrating such techniques into our framework is a promising direction for improving scalability and real-time applicability, which we leave as future work.\n\n**(Weakness 2) Sensitivity of the additional hyperparameter**\n\nIn addition to the hyperparameter planning horizon $H$ and sampling step $N$ discussed in the experiments section, we conducted additional sensitivity analysis on the number of generated samples per iteration in AntGoal-v3 environment.\nThe results are shown below:\n\n|# of generated samples per iteration | 0 | 1000 | 5000 | 10000 | 20000 |\n|-|-|-|-|-|-|\n|Normalized Score| 0.43 | 0.51 | 0.65 | 0.67 | 0.68 |\n\nOur findings indicate that when the number of generated samples is too small, performance degrades due to insufficient learning from the generated samples. However, once the sample size exceeds a certain threshold, performance improves and then saturates, exhibiting only marginal gains with further increases. \n\n**(Weakness 3) Differences from DRAIL and DiffAIL**\n\nWhile DPAIL shares similarities with prior works such as DRAIL and DiffAIL, the underlying mechanisms are fundamentally different.\nBoth DRAIL and DiffAIL use explicit diffusion-based discriminators, and then used to provide rewards for training a separate, uni-modal Gaussian policy, often via RL.\nIn contrast, DPAIL does not use an explicit discriminator at all. Instead, the discriminator is implicitly defined by the diffusion policy itself through the ASAF framework. This implicit formulation enables DPAIL to capture and reproduce diverse behaviors from multi-modal demonstrations, whereas DRAIL and DiffAIL are limited in this regard since their diffusion models are used solely as a reward signal for training unimodal policies.\n\n**(Question 1 & 2) Hyperparameter Tuning**\n\nWe made efforts to ensure a fair and meaningful comparison across all baseline algorithms.\nFor PPO-based baselines (e.g., GAIL, DiffAIL, InfoGAIL), we performed hyperparameter tuning on several factors that impact performance.\nFor example, we searched over the # of rollout-length per iteration, using values [5e3, 1e4, 5e4, 1e5]. In our experiments on MuJoCo tasks, rollout lengths of $\\geq 5 \\times 10^4$ generally led to stable and successful training, so we selected those as default settings for the PPO-based baselines.\n\nAnother hyperparameter is K, the # of iterations. Since different algorithms require different numbers of iterations $K$ to reach their converged performance, we determined a sufficiently large value of $K$ for each method and task during hyperparameter tuning.\nThis ensures that performance is evaluated after sufficiently training rather than at a fixed iteration cutoff.\nWhile our method requires fewer environment interactions to converge, we allowed other baselines to consume more interactions if needed to reach their converged performance. \nThis ensures a fair comparion based on fully trained policies, rather than under a fixed interaction budget. \nAll experiments were run using the same set of random seeds (0-4). For details about the tuned hyperparameters, please refer to Appendix D (Implementation Details).\n\n**(Question 3) Extension to latent variables and Conditional Diffusion Policies**\n\nYes, we have preliminary ideas on integrating latent variables $z$ for mode conditioning, and such an extension is compatible with the current DPAIL lower bound. Below, we briefly outline the idea.\nWe define a conditional diffusion model with a latent variable $z$, where the joint probability $p\\_\\theta(\\tau^{0:N}, z)$ factorizes as:\n\n$p\\_\\theta(\\tau^{0:N}, z) = q(\\tau^N)q(z) \\prod\\_{i=1}^{N} p\\_\\theta(\\tau^{i-1} | \\tau^i, z)$.\n\nwhere $q(z)$ is the prior of $z$, $q(\\tau^N)$ is a standard normal distribution.\n\nTo evaluate the log-ratio $\\log \\frac{p\\_\\theta(\\tau^0)}{p\\_{\\theta^\\text{old}}(\\tau^0)}$, \nwe rewrite it with the posterior $q(z|\\tau^0)$ as:\n\n$\\log \\frac{p\\_\\theta(\\tau^0)}{p\\_{\\theta^\\text{old}}(\\tau^0)} \n= \\log \\frac{\\prod\\_{i=1}^N q(\\tau^i|\\tau^{i-1})}{\\prod\\_{i=1}^N q(\\tau^i | \\tau^{i-1})}\n\\frac{q(z|\\tau^0)}{q(z|\\tau^0)}\n\\frac{p\\_\\theta(\\tau^0)}{p\\_{\\theta^\\text{old}}(\\tau^0)} \n=\\log \\frac{\\prod\\_{i=1}^N p\\_\\theta(\\tau^{i-1}|\\tau^{i}, z)}{\\prod\\_{i=1}^N p\\_{\\theta^{\\text{old}}}(\\tau^{i-1} | \\tau^{i}, z)}\n\\frac{q(z)}{q(z)}\n\\frac{q(\\tau^N)}{q(\\tau^N)}\n$,\n\nwhere $q(\\tau^N)$ and $q(z)$ cancel out. \nTaking expectation of this log-ratio over $\\tau^{1:N} \\sim q(\\cdot | \\tau^0), z~\\sim q(z|\\tau^0)$ does not change the value since it is still a ratio of marginal. Thus, the objective can be rewritten as: \n\n$\\max\\_\\theta \\mathbb{E}\\_{\\tau^0 \\sim p_E} \\left[ \n\\log \\sigma \\left (\n\\mathbb{E}\\_{\\substack{z \\sim q\\_(z | \\tau^0),\\\\\\\\ \\tau^{1:N} \\sim q(\\tau^{1:N} | \\tau^0)}}\n\\log \\frac{\\prod\\_{i=1}^N p\\_\\theta(\\tau^{i-1}|\\tau^{i}, z)}\n{\\prod\\_{i=1}^N p\\_{\\theta_{\\text{old}}}(\\tau^{i-1}|\\tau^{i}, z)}\n\\right) \\right].$\n\nGiven the expert sample $\\tau^0\\sim p_E$, the approximate variational posterior $q_\\phi(z|\\tau^0)$, the reverse process $p_\\theta(\\tau^{i-1}| \\tau^i, z)$, this leads to the following lowerbound:\n\n$\\begin{align*} &\\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}} \n\\left [ \\log \\sigma \\left(\n\\mathbb{E}\\_{\\substack{z \\sim q\\_\\phi(z|\\tau^0) \\\\\\\\ {\\tau^{1:N} \\sim q(\\tau^{1:N}| \\tau^0)}}}\n\\log \\frac{\\prod\\_{i=1}^{N} p\\_\\theta(\\tau^{i-1}|\\tau^{i},z)}\n{\\prod\\_{i=1}^{N} p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}|\\tau^{i},z)}\n\\right) \\right] \\\\\\\\\n&= \\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}}\n\\left [ \\log \\sigma \\left( \n\\mathbb{E}\\_{\\substack{z\\sim q\\_\\phi(z|\\tau^0) \\\\\\\\ {\\tau^{1:N}\\sim q(\\tau^{1:N}| \\tau^0)}}}\n\\sum\\_{i=1}^{N} \n\\log \\frac{p\\_\\theta(\\tau^{i-1}| \\tau^i, z)}{p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}| \\tau^{i}, z)}\n\\right) \\right]  \\\\\\\\\n&= \\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}}\n\\left [ \\log \\sigma \\left( \n\\sum\\_{i=1}^{N}\n\\mathbb{E}\\_{\\substack{z\\sim q_\\phi(z|\\tau^0) \\\\\\\\ {\\tau^{1:N} \\sim q(\\tau^{1:N}| \\tau^0)}}}\n\\log \\frac{p\\_\\theta(\\tau^{i-1}| \\tau^i, z)}{p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}| \\tau^{i},z)}\n\\right) \\right]  \\\\\\\\\n&= \\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}}\n\\left[ \\log \\sigma \\left( \n\\sum\\_{i=1}^{N} \\mathbb{E}\\_{\\substack{z\\sim q\\_\\phi(z|\\tau^0) \\\\\\\\ \\tau^{i-1}, \\tau^{i} \\sim  q(\\tau^i | \\tau^0) q(\\tau^{i-1}|\\tau^i, \\tau^0)}}\n\\log \\frac{p\\_\\theta(\\tau^{i-1}| \\tau^i, z)}{p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}| \\tau^{i},z)}\n\\right) \\right]  \\\\\\\\\n&\\geq \\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}}\n\\left [ \\mathbb{E}\\_{\\substack{i \\sim \\mathcal{U}(1, N),\\\\\\\\ z\\sim q\\_\\phi(z|\\tau^0), \\tau^{i} \\sim q(\\tau^i|\\tau^0)}}\n\\log \\sigma \\left( {N} \\mathbb{E}\\_{\\tau^{i-1} \\sim q(\\tau^{i-1}|\\tau^i, \\tau^0)}\n\\log \\frac{p\\_\\theta(\\tau^{i-1}| \\tau^i, z)}{p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}| \\tau^{i}, z)}\n\\right) \\right] (\\text{concavity}) \\\\\\\\\n&= \\mathbb{E}\\_{\\tau^{0} \\sim p\\_{E}}\\left [\n\\mathbb{E}\\_{\\substack{i\\sim \\mathcal{U}(1, N),\\\\\\\\ z\\sim q\\_\\phi(z|\\tau^0), \\tau^{i} \\sim q(\\tau^i|\\tau^0)}}\n\\log \\sigma \n\\left( N \\mathbb{E}\\_{\\tau^{i-1} \\sim q(\\tau^{i-1}|\\tau^i, \\tau^0)}\n\\log \\frac {q(\\tau^{i-1}|\\tau^i, \\tau^0)}{p\\_{\\theta^{\\text{old}}}(\\tau^{i-1}| \\tau^i,z)}\n-\\log \\frac {q(\\tau^{i-1}|\\tau^i, \\tau^0)}{p\\_\\theta(\\tau^{i-1}| \\tau^{i},z)}\\right) \\right] \n\\end{align*}$\n\n$\\therefore \\mathcal{L}\\_{\\text{DPAIL}}^{(1)}(\\theta,\\theta^{\\text{old}},\\tau^0, z) = \n\\mathbb{E}\\_{\\tau^0, z, i, \\epsilon}\n\\\\Big[ \\log \\sigma \\\\Big (N\n\\\\left (\\\\| \\epsilon - \\epsilon\\_{\\theta^{\\text{old}}}(\\tau^i,z,i)\\\\|^2\n\\- \\\\| \\epsilon - \\epsilon\\_\\theta(\\tau^i,z,i)\\\\|^2\n\\\\right )\\\\Big )\\\\Big],\n$\n\nHere, $z \\sim q\\_\\phi(z|\\tau^0)$, $\\epsilon \\sim \\mathcal{N}(0, I)$, and $i \\sim \\mathcal{U}(1, N)$.\n\nSimilarly, for given generative samples $\\bar{\\tau}\\sim p\\_{\\theta^\\{\\text{old}}}$, we can also get $ \\mathcal{L}\\_{\\text{DPAIL}}^{(2)}(\\theta,\\theta^{\\text{old}},\\bar{\\tau}^0, z)$.\n\nThe overall objective becomes:\n\n$\\therefore\n\\mathbb{E}\\_{\\substack{\\tau^0 \\sim p\\_E, \\\\\\\\ z \\sim q\\_\\phi(z|\\tau^0)}}\\\\left[\n\\mathcal{L}^{(1)}\\_{\\text{DPAIL}} (\\theta, \\theta^{\\text{old}}, \\tau^0,z) \\\\right]\n\\+\n\\mathbb{E}\\_{\\substack{\\bar{\\tau}^0 \\sim p\\_{\\theta^{\\text{old}}}, \\\\\\\\ z \\sim q\\_\\phi(z|\\bar{\\tau}^0)}} \n\\\\left[ \\mathcal{L}^{(2)}\\_{\\text{DPAIL}}(\\theta, \\theta^{\\text{old}}, \\bar{\\tau}^0, z) \\\\right] + \n\\text{MI}\\_\\phi(\\bar{\\tau}^0, z)\n$\n\nWhere the last term encourages learning meaningful unsupervised representation $z \\sim q\\_\\phi(z|\\bar{\\tau})$ by maximizing mutual information."}]

Author rebuttal:

Meta-review: The paper studies adversarial imitation learning with diffusion policies. To avoid the high computational cost and instability of RL-based policy optimization, the paper adopts the Adversarial Soft Advantage Fitting framework. On the upside: 
- the reviewers acknowledge the principled derivation of the optimization objective
- the reviewers also acknowledged the solid experiments
- the reviewers addressed hyperparameter tuning fairness and sensitivity in the rebuttal.

on the downside: 
- the reviewers are still concerned about the higher computational cost and longer inference time of the proposed approach
- the reviewers think that experiments remain focused on simulated environments; real-world or noisy/heterogeneous scenarios are not tested, leaving generalizability uncertain.

In summary, the paper is on the borderline.
Decision: Accept (poster)