Diffusing States and Matching Scores: A New Framework for Imitation Learning
Authors: Runzhe Wu
Yiding Chen
Gokul Swamy
Kianté Brantley
Wen Sun
Venue: ICLR 2025 Poster (2025)
Keywords: imitation learning
diffusion model
score matching
Primary area: reinforcement learning

Abstract: Adversarial Imitation Learning is traditionally framed as a two-player zero-sum game between a learner and an adversarially chosen cost function, and can therefore be thought of as the sequential generalization of a Generative Adversarial Network (GAN). However, in recent years, diffusion models have emerged as a non-adversarial alternative to GANs that merely require training a score function via regression, yet produce generations of higher quality. In response, we investigate how to lift insights from diffusion modeling to the sequential setting. We propose diffusing states and performing *score-matching* along diffused states to measure the discrepancy between the expert's and learner's states. Thus, our approach only requires training score functions to predict noises via standard regression, making it significantly easier and more stable to train than adversarial methods. Theoretically, we prove first- and second-order instance-dependent bounds with linear scaling in the horizon, proving that our approach avoids the compounding errors that stymie offline approaches to imitation learning. Empirically, we show our approach outperforms both GAN-style imitation learning baselines and discriminator-free imitation learning baselines across various continuous control problems, including complex tasks like controlling humanoids to walk, sit, crawl, and navigate through obstacles.

Published as a conference paper at ICLR 2025
DIFFUSING STATES AND MATCHING SCORES: A NEW FRAMEWORK FOR IMITATION LEARNING
Runzhe Wu
Yiding Chen Coreell University ed2
Gokul Swamy
Cornell University rw646@corne11.cm edu
Carnegie Mellon University
gswamydandrew. cmu. edu
Kianté Brantley
Wen Sun
Harvard University
Cornell University
kabrantley@g harvard. edu
ws4550cornel1.cm edu
ABSTRACT
Adversarial Imitation Learning traditionally framed as a two-player zero-sum game between learner and an adversarially chosen cost function, and can therefore be thought of as the sequential generalization of a Generative Adver- sarial Network (GAN). However, in recent years, diffusion models have emerged as a non-adversarial alternative to GANs that merely require training a score function via regression, yet produce generations of higher quality. In response, we investigate how to lift insights from diffusion modeling to the sequential setting. We propose diffussng states and performing score-matching along diffused states to measure the discrepancy between the expert's s and learner's s states. Thus, our approach only requires training score functions to predict noises via standard regression, making it significantly easier and more stable to train than adversarial methods. Theoretically, we prove first- and second-order instance-dependent bounds with linear scaling in the horizon, proving that our approach avoids the compounding errors that stymic offline approaches to imitation learning. Empirically, we show approach outperforms both GAN-style imitation learning baselines and discriminator-free imitation learning baselines across various continuous control problems, including complex tasks like controlling humanoids to walk, sit, crawl, and navigate through obstacles.
INTRODUCTION
Fundamentally, in imitation learning (IL, Osa et al. (2018)), we want to match the sequential behavior of an expert demonstrator. Different notions of what matching should mean for IL have been proposed in the literature, from f-divergences (Ho & Ermon, 2016; Ke et al., 2021) to Integral Probability Metrics (IPMs, Müller (1997); Sun et al. (2019); Kidambi al. (2021); Swamy et al. (2021); Chang et al. (2021); Song et al. (2024)). To compute the chosen notion of divergence that the learner can then optimize it is common train
from the expert demonstrations discriminator (i.e. a classifier) between expert and learner data. This discriminator is then used as reward function for policy update, an approach known inverse reinforcement learning (IRL, Abbeel & Ng (2004); Ziebart et al. (2008)). Various losses for training discriminators have been proposed (Ho & Ermon, 2016; Kostrikov al., 2019; Fu al., 2017; Ke et al., 2021; Swamy al., 2021; Chang et al., 2024), and IRL. has been applied in real life in domains like routing (Barnes al., 2023) and LLM fine-tuning (Wulfmeier al., 2024).
As observed by Finn et al. (2016), inverse RL. can be seen as the sequential generalization of Generative Adversarial Network (GAN) (Goodfellow et al., 2020). However, even the single-shot setting, GANs are known to suffer from issues like unstable training dynamics and mode collapse (Miyato et al., 2018; Arjovsky & Bottou, 2017). In contrast, Score-Based Diffusion Models (Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021) are known to be more stable train and produce higher quality samples domains like audio and video (Rombach et al., 2022; Ramesh et al., 2022; Kong et al., 2021). Intuitively, diffusion models corrupt samples from target distribution with

---

Published as a conference paper at ICLR 2025
noise and train a generative model to reverse this corruption. Critically, these models are trained via score-matching: a non-adversarial, purely regression-base based procedure in which a network is trained to match the score (i.e., gradient of the log probability) of the target distribution.
In fact, prior work has shown that diffusion models are powerful policy classes for IL due to their ability to model multi-modal behavior (Chi et al., 2023). However, these approaches are purely offline behavioral cloning (Pomerleau, 1988), and therefore can suffer from the well-known com- pounding error issue (Ross al., 2011) that affects all offline approaches IL (Swamy al., 2021).
Taken together, the preceding points beg the question:
Can we lift the insights of diffusion models to inverse reinforcement learning?
Our answer to this question is SMILING: a new IRI. framework that abandons unstable discriminator training in favor of simpler, score-matching based objective. We first fit score function to the expert's s state distribution. Then, at each iteration of our algorithm, we fit a score function to the state distribution of the mixture of the preceding policies via standard agression-based score matching, before using the combination of these score functions to define a cost function for the policy search step. Our framework treats diffusion model training (score- matching) as a black box which allows us to transfer any advancements in diffusion model training (e.g., better noise scheduling or better training techniques) to IRL. In theory, rather than optimizing either an f-divergence or an IPM, this corresponds to minimizing a novel sequential generalization of the Fisher divergence (Johnson, 2004) we term the Diffusioo Score Divergence (DS Divergence).
We demonstrate the advantages of our framework in both theory and practice. In theory, we show that our approach can achieve first- and second-order instance-dependent regret bounds, as well as a linear scaling in the horizon to model- moseellmisppeiification errors arising from expert not being realizable by the learner's policy class or potential optimization error in score-matching and policy search. Thus, we establish that, by lifting score-matching to IRL, SMILING provably avoids compounding errors and achieves instance-dependent bounds. Intuitively, the second-order bounds mean that the performance gap between our learned policy and the expert automatically shrinks when either the expert or the learned policy has low variance in terms of their performance under the ground-truth reward (e.g. when the expert and dynamics are relatively deterministic). This is because when we perform score-matching, we are actually minimizing the squared Hellinger distance between the learner and the expert's state distributions, which has shown to play an important role in achieving second-ordes regret bounds in the Reinforcement Learning setting (Wang et al., 2024c). The ability achieve instance-dependent bounds demonstrates the theoretical benefit of Diffusion Score divergence over other metrics such as IPMs and f-divergences. In practice, we show that under in the IL from observation only setting (Torabi et al., 2018: Sun et al., 2019), SMILING outperforms adversarial GAN-based IL baselines, discriminator-free IL baselines, and Behavioral Cloning' on complex tasks such as controlling humanoids to walk, sit, crawl, and navigate through poles (see Figure 3). This makes SMILING the first IRL. method to solve multiple tasks on the recently released HumanoidBench benchmark (Sferrazza et al., 2024) using only the state information of the expert demonstrations. We release the code for all experiments at https:://///.. /gg thub.
2 RELATED WORKS
Inverse Reinforcement Learning. Starting with the seminal work of Abbeel & Ng (2004), var- ious authors have proposed solving the problem of imitation via inverse RL (Ziebart et al., 2008; Ratliff et al., 2006; Sun et al., 2019; Kidambi et al., 2021; Chang et al., 2021). As argued by Swamy et al. (2021), these approaches can be thought of as minimizing an integral probability metric (IPM) between learner and expert behavior via an adversarial training procedure readers to their paper for a full list. Many other IRL approaches can instead be framed as mini- mizing an f-divergence via adversarial training (Ke et al., 2021), including GAIL (Ho & Ermon, 2016), DAC (Kostrikov et al., 2019), AIRL (Fu et al., 2017), FAIRL (Ghasemipour et al., 2020), f-GAIL (Zhang al., 2020), and f-IRI. (Ni et al., 2021). However, all of the preceding approaches involve training a discriminator, while our technique only requires fitting score functions and does
refer interested
This despite the fact that BC requires expert actions, while do not.
2

---

Published as a conference paper at ICLR 2025
not seek to approximate either an f-divergence or an IPM. Lai et al. (2024); Wang et al. (2024a) also explored the use of diffusion models for imitation learning. They insert the score matching loss into the f-divergence objective of the discriminator. Thus, their method also belongs to the f-divergence framework. Huang et al. (2024) also utilize diffusion models for imitation learning, proposing train conditional diffusion model discriminator for single-step state transitions between the expert and the current policy. Their approach remains within the classic f f-divergence framework well. Additionally, various authors have proposed using techniques either stabilize this training procedure (e.g. via boosting, Chang et al. (2024)) or reducing the amount of interaction required during the RL step (Swamy et al., 2023; Ren et al., 2024; Sapora et al., 2024) - as we focus on improving the reward function used in inverse RL, our approach is orthogonal to and could be naturally applied on top of these techniques.
Discriminator-Fres Inverse Reinforcement Learning. SQIL (Reddy et al., 2020) replaces training a discriminator with a fixed reward function (+1 for expert data, for learner data). Unfortunately, this means that performant learner is dis-incentivized to perform expert-like behavior, leading to dramatic drops in performance (Barde et al., 2020). The AdRIL algorithm of Swamy et al. (2021) uses techniques from functional gradients to address this issue, but requires the use off off- RL algorithm to implement, while our framework makes no such assumptions. ASAF (Barde et al., 2020) proposes using the prior policy to compute the optimal f-divergence discriminator in closed form, while we focus on a different class of divergences. IQ-L_errn (Garg et al., 2021) proposes to perform IRL in the space Q functions, but can therefore suffer from poor performance on problems with stochastic dynamics (Ren et al., 2024) or when the Q- Q-function is more complex than the reward function (e.g. navigating to a goal in a maze). Al-Hafez et al. (2023) propose to use a reward reg- ularization to overcome the instabilities of prior work that learns Q- Q-functions. Sikehi et al. (2024a) point out that prior methods rely on restrictive coverage assumptions and propose new method that learns to imitate from arbitrary off-policy data. Jain et al. (2024) propose measuring differences between learner and expert behavior terms of successor features, but are only able to optimize over deterministic policies. Sikchi et al. (2024b) suggest to learn multi-step utility function that quantifies the impact of actions on the agent's 's divergence from the expert's 's visitation distribution. DRIL (Brantley et al., 2019) relies ensemble methods and uses the disagreement among models in the ensemble as a surrogate cost function, making it challenging to prove similar theoretical guar- antees to our technique. Dadashi et al. (2021) frame IL. as minimizing the Wasserstein distance in its primal form rather than the dual commonly used in prior discriminator-based works. Liu et al. (2021) also consider score matching but they use energy-based models instead of diffusion models.
Diffusion Models for Decision Making, Diffusson models have been widely studied in the context of robot learning, where diffusion models have demonstrated a strong ability to capture complex and multi-modal data. Recent works have explored using diffusion models to perform behavioral cloning (Chi et al., 2023; Pearce al., 2023; Chen et al., 2024; Block et al., 2023; Team et al., 2024) and for reward guidance (Nuti et al., 2024) as well as using diffusion models to capture trajectory- level dynamics (Janner et al., 2022; Ajay et al., 2022). We instead focus on lifting insights from diffusion modeling and score matching to IRL, which uses both offline expert data and also online interaction with the environment. Unlike prior diffusion BC work, we use diffusion models for both the expert's and the learner's 's state distributions, not just the expert's s action conditionniii distributions. This means we do not require expert action labels, which can be challenging to acquire in practice.
3 PRELIMINARIES
3.1 MARKOV DECISION PROCESSES
We consider finite-horizon Markov decision processes (MDPs) with state space S C Rd, action space A. transition dynamics P. (unknown) cost function c* S - IR, and horizon H. The goal is to find a policy π. S + A(A) that minimizes the expected cost H Eendr c* (s) where d" denotes the average state distribution induced by the policy म, i.e., d (s) = Im= Pr(sh. = s)/H where 8i denotes the state at time h. The state value function of a policy
defined as (s)
Er~N Σi c² (8K) where 7 = (81, (81,...,81) 8n) denotes the trajectory induced by the policy π.
We assume the cost depends only on states. The extension to state-action costs straightforward.
3

---

Published as a conference paper at ICLR 2025
Expert
Score
Matching
P
g Vlogp"
Vlog P(s))
Score
Vlog 2,6s)
Replay Butfer
Matching
8" Vlogp"
N(O.I)
E
Vlog
Policy
V Vlog
(a) Illustration of DS Divergence (Defi-
nition 1).
(b) Illustration of SMILING (Algorithm 1).
Figure 1: Figure (a): The two curves represent the forward diffusion process of distributions P and Q. DS Divergence measures the squared difference between the diffusion score functions, V log (8.) and V log (s;). along the forward diffusion process of P. Figure (b): SMILING first pre-trains diffusion model from the expert's data. It then iteratively trains diffusion models on leamner's s data and performs RL optimize cost function formed by the learner's 's score function and the pre-trained expert score function. The cost function is designed to faithfully approximate the DS divergence (Definition 1) between the learner and the expert.
3.2 IMITATION LEARNING
In imitation learning, we are usually given expert demonstration in sille--cttonnnext-ssie tuples and aim to learn policy that mimics the expert policy æ² without access the ground-truth cost function c*. This paper considers a harder setting where only states are given. Specifically, given a dataset of state demonstrations D = {s()) Ji=1 sampled from expert, we aim to learn a policy i that minimizes the discrepancy between its state distributions and the expert's. In addition, can interact with the environment to collect more (reward-free) learner trajectories.
3.3 DIFFUSION MODELS
The forward process diffusion model adds noise to a sample from the data distribution Po- We can formalize it via a stochastic differential equation (SDE). For simplicity in this paper, we con- sider the Cemstein-Uhlenbed (OU) process: dx, = -x, dt + /2 dB,, Xo ~ Po. where B₁ is the standard Brownian motion in Idd. It known that the reverse process (yt)ce(o.T) satisfies the follow- ing reverse-time SDE (Anderson, 1982): dy, = (y: + 2V log pry()))) dt + v2 dB,, where (T) denotes the uniform distribution on [O, 2]. and Be now denotes reversed Brownian motion. The gradient of the log probability V log Pt is called the score function of the distribution Pt- When the score function known, we can use it to generate samples from po by simulating the reverse process. Estimating the score function V log Pt is called score matching, which is typi- cally done via minimizing the regression-bee
loss: ming Ex-pp ETT Exi Vx, log (xx x). We use (x; x) to denote the conditional distribution at time t the forward process conditioned on the initial state which has closed form qe (xpl x) N(xx (1 (1-e-1()
[]g( ()
Applying Diffusion State Distributions. For a policy म, we use Pē to denote the marginal distribution time obtained by applying the forward diffusion process d i.e., initial samples are drawn from p = d When presenting asymptotic results, we use O(-) to hide constants and (()) to hide constants and logarithmic factors.
4 ALGORITHM
We introduce a novel discrepancy measure between two distributions that our algorithm leverages: Definition (Diffusion Score Divergence). For two distributions P and Q, we define the Diffusion Score Divergence (DS Divergence)
Dps(P. (P((P =
E
E
P((s) - V Voo 2c(ss)
(~U(T) s.
4

---

Published as a conference paper at ICLR 2025
Algorithm SMILING (Score-Matching Imitation LearnING)
Require: state-only expert demonstration D = {
1: Estimate score function of expert state distribution:
[ll(stett v., log (s (+)))(
g argmin E 9Eg
E snDD (-(()) st-gil--s)
E
2: for = 1, ........ K do
3: Estimate the score function of learner state distributions:
R-1
g (k)
argmin gEg
Σ
II
E
E
[Ia((),)) - Vs, log qe [
(T) a se 18)
f=1
4: Update policy x (8) via RL (e.g., SAC) on cost c(H) (Eq. 3): (1)) < RI(((())
5: end for
Here ())= represents the conditional probability of the forward diffusion process at time t con- ditioned the initial state Pt and Qt denote the marginal distributions at time obtained by applying the forward diffusion process to and Q, respectively. We call V log Pe and V log Qe the Diffusion Score Function of P and Q.
Figure Ia illustrates the DS divergence. It measures the difference between two distributions by comparing their diffusion score functions within one diffusion process. It is analogous to Fisher divergence but differs by incorporating an expectation over the diffusion process. DS divergence is strong divergence in the sense that, whenever the DS divergence between the two distributions is small, the KI. divergence, Hellinger distance, and total variation distance are all small (see Lemma and also Chen al. (2023b); Oko et al. (2023)).
In our algorithmic framework Score-Matching Imitation LearnING - SMILING, we propose to frame imitation learning the minimization of the DS divergence between the expert's and the (history of) learner state distributions. We begin by discussing how to do so before discussing the theoretical benefits of doing so.
The first step is to pre-train a score function estimator (s, on expert's state distribution V. log pT (s). This can be done via standard least-squares regression-based score-matching, as shown in Line Here denotes the function class for score estimators. Then, we seek find policy π that minimizes the DS divergence between learner and expert state distributions:
E ((TT) anddd (-U()))) l19 (s1, V., log P. (st))12
where we have approximated the expert's diffusion score function by (s,,,, t). However, ((T) is not directly computable since do not know the learner's score function V, log (st). A naive approach would be to directly learn an estimator for V., log p7 (8x) score matching and substitute prove in Appendix B, this introduce significant errors due
into the equation. However, the unboundedness of a difference of score functions and variance-related concerns. We now derive method that does suffer from this issue.
Recall that we defined (s s) as the distribution of given the initial sample s, which is simple Gaussian distribution with an appropriately scaled variance. To develop our method, we first note that, luckily, V., log (s(s(s)) is an unbiased estimator of the score V., log pÝ (st), V., log (st) (Song et al., 2021). Given this, perhaps the most immediate strategy would be to simply replace V., log p. (s:) by its unbiased estimator, V., log ss) in Eq. 1. However, in contrast to linear objectives like an IPM, to approx- imate a squared objective like the DS Divergence accurately, we need to get both the expec- tation (i.e. have an unbiased estimator) as well as the variance correct. If we don't. t, then Ea-d" (t (x- (st, V., log qe (st [s)ss will differ from f(xx) by a term related to variance of the estimator v., log (s i.c
because Esj,, [.., log qe (s.
E and A (T
V log (st) V., log (s
(2)
5

---

Published as a conference paper at ICLR 2025
To faithfully approximate {TT), need to estimate the variance term and subtract it from Be-dd ( (st, - vs, log (s We can do this by adding a term to our objective that minimized at the variance and using another function network to optimize it. This leads us to the following estimator for {(T)):
(TT)
B smad t-- ( (st - V., (
minn 9E9 and (e
1lo 1) Vs, log q: (s(ls)
Minimized the variance of Eq.
where G is the function class for score estimators. Crucially, the highlighted term (orange) in the above expression estimates the variance in Eq. 2 since one of its minimizers will be the Bayes optimal Vs, s, log P (st), i.c. the conditional expectation Esjs, [V,, log (stls)]. Observe that mini- mizing this orange term just requires a simple regression-based score matching objective. So, while naive score matching causes issues due the variance of the estimator, more clever application of score matching can be used to fix this concern.
With ((T)) now serving as a valid approximation of ((TT), the II. problem reduces to searching for a policy minimize ((TT), i.e., min, {(T)) To facilitate this, we define payoff L(T. g) as:
E [(T, ardd (ee
(se, - Ve, log (st))) llg(s,, t) V., log q (srls))
Then, minimizing (TT) is equivalent solving the two-player
game min, max, [(T, 9)-
To solve this game, we propose following a no-regret strategy over 9 against a best response over i (i.e. dual algorithm (Swamy al., 2021)). Specifically, Algorithm applies Follow-the-Leader (Shalev-Shwartz et al., 2012) to optimize in Line 3, and performs best response computation over ı via RL (Line 4) under cost function c² (s)::
| - V. log (s s) - V., log q (5
(s),
(s) =
E
E
(T) senge(-
(3)
Remark Nloise-prediction Form of the Cost Function). The cost function in Eq. 3 involves the score function V log (s, s), which has a closed-form expression for most modern diffusion models including DDPM. Specifically, when the diffusion follows the OU process (described Section 3.3), the score function takes the form V log de (&1 (s)
82), which is exactly the noise added to the original sample s over diffusion (recalling that for the OÚ process, ge(s, |s)
e -
N(se [tseet,00 e 2)))) We denote this noise by Then, the cost function (Eq. 3) is equivalent
to c(k) (s) == (1) (e lllgg (se +,,t) El (kg( te,t) - This closely
resembles the noise prediction DDPM.
To implement FTL, in Line 3, we use the classic idea of Data Aggregation (DAgger) (Ross et al., 2011), which corresponds to aggregating all states collected from prior learned policies (1)) ,(.,,.., n (k-1), and perform a score- mooe-matching least square regression on the aggregated dataset. take advantage of any modern RI. optimizers. In experiments, SAC (Haarnoja et al., 2018) and DreamerV3 (Hafner et al., 2023), which serve as representative model-free and model- RL algorithms.
The RI procedure
THEORETICAL RESULTS
Our goal in this section is demonstrate that SMILING achieves instance-dependent regret bounds while at the same time avoid compounding errors when model misspecification, op- timization error, and statistical error exist. In practice,
only estimate the diffusion score
NNoe that similar ideas have been used in the offline RL literature for designing min-max based algorithms (2021)).
for estimating value functions (e.g., Chen & Jiang (2019); Uchara
"Note that L(T, g) is square loss functional with respect to g. Since square loss is strongly convex. FTL is no-regret. In contrast, when optimizing an IPM, has to use wew.thh-Reeullrizzede Leader (FTRL) to have the no-regret property (e.g., Sun et al. (2019); Swamy al. (2021)), making implementation more complicated.

---

Published as a conference paper at ICLR 2025
function up to some statistical error optimization eπror due to finite samples. For similar reasons, the RL procedure (Line 4) can only find a near-optimal policy especially when the policy class is not rich enough to capture the expert's policy. To demonstrate that algorithm can tolerate these errors, we explicitly study these errors in our theoretical analysis, particually how our regret bound scales with respect these errors. We provide the following assumptions to formalize these errors.
Assumption 1. We have the following error bounds, corresponding respectively to Lines 1, and of Algorithm 1:
(a) The estimator 9" is accurate up to some error Cucore: Egend" EE-U(() Eaconed (s) (illl
s. log (s))]2] I 2 Ecarro
(b) There exists a function Regret(K) sublinear in K such that the sequence of score function estimators {g ]k=1 has regret bounded by Regret(K),
(10))))))))) Ve, log log 1s) 27
E
E (TU
E (
R= (tr
[ls((e:t) Vo, ls s)1B + (egret(K);
min
E k= tr (0) (TT) senge(t)s)
E
E
(c) There exists ERI. > such that, for all k = I,.... K, the RL procedure finds an ERI. PRt.ootiiml policy within some function class II: E Ex (s) mingen Ee c( S <RL- Note that we do not assume HO E II.
Item (a) is a standard assumption in diffusion process and score matching (Chen et al., 2023b;a; Lee et al., 2022). When the function class & (hypothesis space) is finite, Secore typically scales at rate of Ô (In()))//N) in terms of the number of samples N. For infinite function class, more advanced bounds can be established (e.g., see Theorem 4.3 in Oko et al. (2023)). We emphasize that the discretization error arising from approximating the diffusion SDE using Markov chains is included in éscore as well. Item (b) is similar to Item (a) but for the sequence of score function estimators. It can be satisfied by applying
sello-the-Leaade (FTL) since the loss functional for g is (FTRL) or online
a square loss. Other no-regret algorithms, such as Follow-the-ree sether-ggularzee-leaddee gradient descent, can also used. Under similar conditions as in Item (a), one typically achieves Regret(K) K) = Ö(v K). Item (c) can be satisfied as long as an efficient RI. algorithm is applied at each iteration in Line 4. The rate of ERL has been well-studied in the RL theory literature for various of MDPs. For instance, in tabular MDPs, one can achieve ERI. Ó A///M) where S is the number of states, A is the number of actions, and M is the number of RL rollout samples. We emphasize that CRL this case does not scale with as it is defined as the upper bound on the averaged return instead of the cumulative return over H steps.
In the theoretical analysis, we will assume any misspecification of score functions. Specifically, for any policy π, we assume that its score function lies in the function class G (i.c., V log E G). However, we allow for misspecification in the RL procedure (i.e., x may not be in II). This enables us to focus more on the misspecification of the RL procedure under our defined framework.
Next, we introduce the regularity assumptions on the diffusion process, which are standard in the literature diffusion models.
Assumption 2. For all π, the following two conditions hold: (1) the diffusion score function V log Pe is Lipschitz continuous with finite constant for all t E [0, T), and (2) the second mo- ment of d= is bounded: Es-ar [||s|[]] S m for some O.
The first condition ensures that the score function behaves well so we can transfer the DS divergence bound to a KL divergence bound. The second condition is needed for the exponential convergence of the forward diffusion process (i.e., the convergence to the standard Gaussian in terms of KL divergence). We note that the second condition is readily satisfied in certain simple cases such as when the state space is bounded. Notably, if we disregard discretization errors, the Lipschitz constant will not appear in our theoretical results as long as it is finite, even if it is arbitrarily large. Similarly, the constant m only appears within logarithmic factors so it can be exponentially large.
7

---

Published as a conference paper at ICLR 2025
However, when discretization errors are considered, both the Lipschitz constant and will appear in Eacore and (egret(T)) (see, c.g., Chen et al. (2023b;a); Oko et al. (2023)).
Now we are ready to present our main theoretical result. Let (1:2) denote the mixture policy of {x(1),, ...... ...).. Particularly, ((1:K) is executed by first choosing policy (8) uniformly at random and then executing the chosen policy. For any policy we define the variance of its return
as Var = Variance 1x=1 (s%)). Below is our main theoretical result with proof in Appendix C.
Theorem 1. Under Assumptions I and 2, Algorithm / achieves these instance-dependee bounds:
(Second-order)
V - -V* =6
|mmnn (Var",, Var -EE . eH
(First-order)
Vallik)
- V= =0 =Ō=== V min (VT*, Vxlli))) cH + +cH
if E I1, = & II. specification Emis == miD ((T)) where we recall that E(-) is defined in Eq. 1.
Eecore + ERL + Regret (
where we define total error c =
and the mis-
Eecore + ERL + Regret /) K + Emis
Here the misspecification error mis measures the minimum possible DS divergence to the pre- trained expert score 9². We note that Cmis is algorithmic- pathi independent once given g", i..., it does not depend on what the algorithm does- - is quantity that is pre-determined when the IL problem is formalized and the expert score of is pre-trained. If we increase the expressiveness of TT, fmis will decrease. In contrast, interactive II. algorithms DAgger (Ross et al., 2011) and Aggre & Bagnell, 2014; Sun et al., 2017), which claim to have no compounding errors, actually have algorithmic-paat dependent misspecification errors. This means their misspecification error does not necessarily decrease and in fact can increase when one increases the capacity of (since affects the algorithm's 's behavior).
(Ross
is known that the second-order bound is tighter and subsumes the first-order bound (Wang al., 2024c). Importantly, previous results on second-orde bounds for sequential decision socision-making have relied exclusively on the Maximum Likelihood Estimator (MLE) (Wang et al., 2023; 2024c; Foster et al., 2024; Wang et al., 2024b; 2025), which in general is not computationally tractable even for simple exponential family distributions (Pabbaraju et al., 2024). In this work, we show for the first time that a computationally tractable alternative diffusion score matching can achieve these instance-dependen bounds in the context of IRL. In particular, our second-order bounds (or the first-order bound) scale with the minimum of Var** and Var (1:8k (or minimum of V** and (x.0.)) Hence, our bounds are sharper whenever one of the two is small. In contrast, bounds in these prior RL work do not scale with the min of variances associated with the learned policies and the comparator policy T². This subtlety is likely due to the difference between the IRL setting and the general RL setting instead of techniques in the analysis.
Improved sample efficiency with respect to the size of expert data. Let's now focus on sample complexity with respect to expert data, which is typically the most expensive data to collect in IL, to highlight the benefits of our method over previous approaches. For conciseness, we ignore all errors that are purely related to computation. Specifically, spent to run the RL procedure, let the number of iterations K - oc so Regret K) K)/K + 0, and as- sume nh ПT. Under these conditions, our bound solely depends éxcore For simplicity, we further assume finite score function class G. neglect misspecification errors (i.c., V log p. (x) G for all I)) and assume the magnitude of the score loss is (1). Now standard regression analysis (Agarwal et al., 2019; Oko et al., 2023) yields éscore = O( O ((Vooo(())//)) where N is the number of expert samples. Plugging this back into our second-order bound, we obtain
set EgL - 0, as it only depends
the time
1 KL
(xll)) V+
=0 10='=0
-o(inin (Var"", (o
log( GG)
H (oog(((9))
N +
N
When the expert's cost the learned policy's cost has low variance, i.c., Var* 0 or Var= (1))) + 0, our bound simplifies to O(H log(Ig))/N). This shows a significant improvement over prior sample complexity bounds for IRL. with general function approximation, which are
8

---

Published as a conference paper at ICLR 2025
typically O(H / ()))
(e.g., IPM-based methods (Kidambi et al., 2021; Chang al., 2021)). This demonstrates the benefit of using diffusion processes and score matching over IPMs.
In addition to the sample benefit above, show our approach require less expressive function approximators than discriminator-basse methods. Please refer to Appendix A for more details.
6 EXPERIMENTS
We evaluate our method, SMILING, on a set continuous control tasks from the Deepmind Control Suite (Tassa et al., 2018) and HumanoidBench (Sferrazza al., 2024). These tasks vary in difficulty, as summarized in Table of Appendix D. While we will compare to multiple baselines including non-adversarial baselines, we focus testing our hypothesis that score-matching based approach should outperform (-ddvvrgeecce-aaseeee approach. Additional results and details are in Appendix D.
Baselines. We include Discriminator Actor-Critic (DAC), IQ-Leam,, DiffAlL,, and Behavior Cloning (BC) as baselines. Our DAC implementation is based on the original design by Kostrikov et al. (2019) that uses JS-divergence GAN-style objective. We use the official implementation of IQ- Learn. We tried several configurations provided in their implementation for the MuloCo tasks and reported the best one. DiffAIL also follows official implementation. The BC performance is reported as the maximum episode reward obtained across all checkpoints during training. We found MLE outperforms squared loss for BC, so all BC results below use MLE.
Diffusion Models. We employ the Denoising Diffusion Probabilistic Model (DDPM) as the diffu- sion model. The score function is modeled via an MLP with one hidden layer of 256 units, and we use the same architecture for the discriminator in DAC. The diffusion process is discretized into 5K steps and we use a learnable time embedding of dimensions. We approximate the cost function (Eq. 3) via empirical mean using 500 samples. Since the scale of the score matching loss (and con- sequently our cost function) is sensitive to the data scale (in this context, the scale of state vector), normalize the cost of each batch have zero mean and standard deviation of 0.1.
Solvers. For DMC tasks, we use Soft Actor-Critic (SAC) (Haarnoja et al., 2018) as the solver for both DAC and our method, based on the implementation provided by Yarats & Kostrikov (2020). For HumanoidBench tasks, we adopt D3eamerV3 (Hafner et al., 2023) using the implementation from the HumanoidBench repository (Sferrazza al., 2024). Additionally, we maintain separate "state buffer" that operates the same as the replay buffer but only stores states. This buffer is used to sample states update the score function in our method and the discriminator in DAC.
Overall, we keep the implementation of SMILING and the baseline DAC as close as possible (e.g., using the same RL solver) so they only differ in objective functions (i.e., SMILING learns score functions while DAC learns discriminators to approximate JS divergence). This allows us to show the benefits of our acore-matching approach to the discriminator-ban based GAN-style ones.
Expert Policy and Demonstrations. We use the aforementioned RL solvers to train expert policies and collect demonstrations by running the expert policy for five episodes. Each task has a fixed 1K-step horizon without early termination, and thus the resulting dataset has 5K steps per task.
6.1 LEARNING FROM STATE-ONIL DEMONSTRATIONS
Figure 2 illustrates the results across six control tasks when the expert demonstrations are state-only. We trained for 0.4M steps on Ball-in-cup-cate matches expert performance and outperforms all baselines on five of them. Specifically, our method outperforms all baselines by a large margin on four. For humanoid-sha performance, ours converges the fastest. For cheetah -rnn although DAC initially shows faster convergence, it becomes unstable soon and oscillates significantly after 2M steps; in contrast, our algorithm remains much more stable and surpasses DAC in 3M steps. Notably, we observed DiffAIL. collapses at the beginning of training on all four humanoid tasks due to numerical issues (neural network weights diverging to infinity). Hence, note that, in the plot, we include BC as a reference though it is not directly comparable since it uses state-action demonstrations, Here BC serves as a performance upper bound for Behavioral Cloning from Observation (BCO) (Torabi et al., 2018), which infers the expert's 's actions from state-only data before training. Nevertheless, we observe that both method and DAC consistently outperform
and for 3M steps on all other tasks. Our method
while none achieves expert
were unable to report the those results. Also
9

---

Published as a conference paper at ICLR 2025
Ball in Cup Catch
Cheetah Run
Humanoid Walk
1.00-
1 (nornalized)
0.75
0.50
0.00-
Steps
x10
Steps
x10%
Steps
x10
Humanoid Sit
Humanoid Crawl
Humanoid Pole
1.00-
V (normalized)
0.75
0.50-
0.25-
0.00
6
2
3 x10
2
Steps
x 10
Steps
Steps
x 10
SMILING (ours)
DAC
IQ-eaam
DifAiL
BC
Figure 2: Learning curves for learning from state-only data across five random seeds. The x-axis corresponds to the number of environment steps (also the number policy updates). The y-axis is normalized such that the expert performance is one and the random policy is zero. Our method clearly outperforms all baselines in five tasks out of six.
BC (and thus, BCO). We provide video demos of SMILING and DAC on the humanoid-cree and momenoid-pole tasks in Figure 3 to give clear visual comparison of their performance differences.
6.2 ADDITIONAL RESULTS
Learning from State-Action Demonstrations. Our method can be easily extended to learning from state-action demonstrations by appending the action vector to the state vector for both training and computing rewards. Hence, we also explore its performance in this setting. The results are consistent with the state-only setting and can be found in Appendix D.I.
Number of Expert Demonstrations. We also investigate the impact of the number of expert demon- strations on the performance of our method and basclines in order to support theoretical results in Section 5. The results can be found in Appendix D.2 and show that our method is clearly more sample-efficient and performs well when the number of expert demonstrations is small.
Expressive of Diffusion Models. In Appendix A, we argued that score matching is more expres- sive than diseriminator-basee methods, which may explain why approach generally outperforms DAC. To support it, we conducted an ablation study on the cheetah- task where ours converged slower than DAC previously. We removed the activation functions in both the discriminator of DAC and the score function of our method to make them purely linear. We found that DAC's performance degrades notably while our method remains more effective and outperforms DAC clearly. Detailed results are in Appendix D.3. This reinforces our claim about the expressiveness of score matching.
7 CONCLUSION
We propose a new IL framework, Score-Matching Imitation Learning (SMILING), that leverages the diffusion score function to learn from expert demonstrations. Unlike previous methods, SMILING is not formulated from any f-divergence or IPM perspective. Instead, it directly matches the score of the learned policy to that of the expert. Theoretically, we show that score function is more expressive than learning discriminator on f-divergence. Additionally, our method achieves first- and second- order bounds in sample complexity, which are instance- -deendee and tighter than previous results. In practice, our method outperforms existing algorithms on of continuous control tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
GKS was supported in part by a STTR grant. WS acknowledges support from NSF IIS-2154711, NSF CAREER 2339395, and DARPA LANCER: LeArning Network CybERagents. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp. 1, 2004.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- fional Conference on Learning Representations, 2022.
Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. In /CLR, 2023.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017. URL https://
openreview wwwtteeuun/aieeeeeeeeeeee HorrrmmiddHHk...... _
Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Chris Pal, and Derek Nowrouzezahrai, Adver- sarial soft advantage fitting: Imitation learning without policy optimization. Advances in Neural Information Processing Systems, 33: 1331443-11344, 2020.
Matt Barnes, Matthew Abueg. Oliver F Lange, Matt Deeds, Jason Trader, Denali Molitor, Markus Wulfmeier, and Shawn O' Banion. Massively scalable inverse reinforcement learning in google maps. In The Twelfth International Conference on Learning Representations, 2023.
Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake. Provable guar- antees for generative behavior cloning: Bridging low-level stability and high-level behavior. Ad- vances in Neural Information Processing Systems, 36:48534 48547, 2023.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreemen--egulaaiizee imitation learning. In International Conference Learning Representations, 2019.
Jonathan Chang, Masatoshi Uchara, Dhruv Sreenivas. Rahul Kidambi. and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
Jonathan Daniel Chang, Dhruv Sreenivas, Yingbing Huang, Kianté Brantley, and Wen Sun. Ad- versarial imitation learning via boosting. In The Twelfth International Conference on Learning Representations, 2024.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learming, pp. 4735-4763, PMLR, 2023a.
Jinglin Chen and Nan Jiang. Information -hoomattion-theortiic theoretic considerations in batch reinforcement learning.
In International Conference on Machine Learning, pp. 1042-1051. PMLR, 2019.
Shang- Fu Chen, Hsiang-Chir Wang, Ming- Hao Hsu, Chun-Mao Lai, and Chao-Hua Sun. Diffusion model-ane auggeeeee behavioral cloning. In Proceedings of the 41st International Conference on Machine Learning, pp. 7486-7510, 2024.
11

---

Published as a conference paper at ICLR 2025
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions. International Conference on Learning Representations, 2023b.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273668. 2023.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. In ICLR 2-2I-Ninth International Conference Learning Representations, 2021.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy- bbaegy models. arXiv preprint arXiv:1611.03852 2016.
Dylan Foster, Adam Block, and Dipendra Misra. behavior cloning all you need? understanding horizon in imitation learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv: 1224, 2017.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 4028-4039, 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. divergence minimization perspective on imitation learning methods. In Conference on robot learning, pp. 1259-1277. PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-1L) 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 6851, 2020.
Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, Dai-Jie Wu, and Shao-Hua Sun. Diffusion imita- tion from observation. arXiv preprint arXiv:2410.05429, 2024.
Arnay Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, and Sanjiban Choud- hury. Revisiting successor features for inverse reinforeement learning. ICML 2024 Workshop on Models of Human Feedback for Al Alignment, 2024.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09)) 09999999.99........ 2022.
Oliver Johnson.
Information theory and the central limit theorem. World Scientific, 2004.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113. springer, 2014.
12

---

Published as a conference paper at ICLR 2025
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop the Algorithmic Foundations of Robotics 14, pp- 313-329. Springer, 2021.
Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances Neural Information Processing Systems, 34:28598-28611 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference Learning Representations, 2021.
lyy Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-actor--iitn Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
Chun-Mao Lai, Hsiang Chun Wang. Ping-Chun Hsieh, Yu-Chiang Frank Wang, Min-Hung Chen, and Shao-Hua Sun. arXiv:2405. 161994 2024.
Isiang-Chun
Diffusion-reward adversarial imitation learning. arXiv preprint
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882. 2022.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In Pro- ceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 809 817, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probubility, 29(2):429-443, 1997.
Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching, In Conference Robot Learning, pp. 529- 551. PMLR, 2021.
Felipe Nuti, Tim Franzmeyer, and João F Henriques. Extracting reward functions from diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distri- bution estimators. In International Conference Machine Learning, pp. 26517-26582. PMLR, 2023.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ Robotics, 7(1-2): 1-179, 2018.
Chirag Pabbaraju, Dhruv Rohatgi, Anish Prasad Sevekari, Holden Lee, Ankur Moitra, and Andrej Risteski. Provable benefits of score matching. Advances in Neural Information Processing Sys- tems, 36, 2024.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcareel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 10677, 2023.
Dean A Pomerlcau. Alvinn:
An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 224::2004006225, 1(2):3, 2022.
13

---

Published as a conference paper at ICLR 2025
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pp. 729-736, 2006.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning. arXiv preprint arXiv:2402.08844 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Silvia Sapora, Gokul Swamy, Chris Lu, Yee Whye Teh, and Jakob Nicolaus Foerster. Evil: Evolution strategies for generalisable imitation learning. arXiv preprint arXiv:2406. 1905, 2024.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
Shai Shaley-Shwartz et al. Online learning and online convex optimization. Foundations and Trends@ in Machine Learning, 4(2):077-14,, 2012.
Sikehi, A Zhang, and S Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In International Conference on Learning Representations. International Con- ference on Learning Representations, 2024a.
Harshit Sikchi, Caleb Chuck, Amy Zhang, and Scott Nickum. A dual approach imitation learning from observations with offline datasets. In 8th Annual Conference on Robot Learning. 2024b.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Yuda Song, Drew Bagnell, and Aarti Singh. Hybrid reinforcement learning from offline observation alone. In International Conference on Machine Learning, pp. 46019-46049. PMLR, 2024.
Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply ag- grevated: Differentiable imitation learning for sequential prediction. International conference machine learning, pp. 1309-3318. PMLR, 2017.
Wen Sun, Anirudh Vemula. Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036-6045 PMLR, 2019.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning. pp. 10022-10032. PMLR, 2021.
Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforce- ment learning without reinforcement learning. In International Conference on Machine Learning, pp. 33299-33318, PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801 000900 2018.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213. 2024.
Flemming Topsoe. Some inequalities for information divergence and related measures of discrimi- nation. IEEE Transactions information theory,
2000.
Faraz. Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the Twenty-Sevente International Joint Conference on Artificial Intelligence. International Joint Conferences Artificial Intelligence Organization, 2018.
Masatoshi Uchara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie. Finite sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.02981, 2021.
Bingzheng Wang, Guoqiang Wu, Teng Pang, Yan Zhang, and Yilong Yin. Diffail: Diffusion ad- versarial imitation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15447-15455, 2024a.
Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
Kaiwen Wang. Nathan Kallus, and Wen Sun. The central role of the loss function in reinforcement learning. arXiv preprint arXiv:2409. 12799, 2024b.
Kaiwen Wang, Owen Oertell, Alekh Agarwal, Nathan Kallus, and Wen Sun. More benefits of being distributional: second-order bounds for reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pp. 51192-51213, 2024c.
Zhiyong Wang,
Dongruo Zhou, John CS Lui, and Wen Sun. Model-based rl as a minimalist approach to horizon-free and second-order bounds. In The Thirteenth International Conference on Learning Representatiinss, 2025.
Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, et al. Imitating language via scalable inverse reinforcement learning, arXiv preprint arXiv:2409.0136) 2024.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// github. www.coo//////e wttmm//enisaaattsspyytaaa
sac, 2020. 2020.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for gen- crative adversarial imitation learning. Advances neural information processing systems, 33: 12805-12815, 2020.
Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
15

---

Published as a conference paper at ICLR 2025
A COMPARISON TO DISCRIMINAATR-BBASED (F-DIVERGENCE AND IPM) METHODS
In this section, we compare DS divergence with f-divergence, particularly its discriminator-base approximation (the key metric used by IL algorithms such as GAIL and DAC):
max TEJ
E snup
log f())] + E [log(1- (os((])())]]
where F C (0,1)$ is a restricted function class, It is known that the above is a lower bound of the -haseonShannon (JS) divergence between p and q. In particular, when p + q) E F, it is exactly equal to the JS divergence. However, representing p/(p + q) may be challenging, Below we show why score-matching is more preferred than discriminator-batt f-divergence via the example of exponential family distributions,
On the expressiveness of score functions and discriminators. Consider the distributions p and q to be from the exponential family, p(s) = exp(w" ((s)//Z, (and similar for q), where is a quadratic feature map, and Zp is the partition function. In this case, the score function V T.s)) is simply linear in In contrast, the optimal discriminator for JS divergence of p and q has the form of p/ (p++g) q). which is inherently nonlinear and cannot be captured by a linear function As result. if use linear functions to model discriminators, the discriminator -basdd objective cannot even faithfully serve as a tight lower bound for the JS divergence. Thus minimizing loose lower bound of the IS-divergence does not imply minimizing the JS-divergence itself.
V. log p(s)
Now consider integral probability metric (IPM), i.e., maxfer (Eang (s) - Ex-p (s)). When and q are from the exponential family, perfectly distinguish q and p. one needs design discrim- inator in the form of 0 6(x) since ø is the sufficient statistics of the exponential family distribution. In our example, á quadratic feature mapping. meaning that we need non-linear discriminator for IPM as well.
Hence, score matching only needs relatively weaker score function class represent complicated distributions to control certain powerful divergence including KL divergence and Hellinger distance. However, for f f-divergence- based or IPM-based methods, we need a more expressive discriminator function class to make them serve tight lower bound of the divergences. This perhaps explains why in practice for generative models, score matching based approach in general outperforms more traditional GAN based approach. In our experiment, we conduct an ablation study in Appendix D.3 to compare our approach to a GAN-based baseline under linear function approximation for score functions and discriminators, and we show that SMILING still learn well even when the score functions are linear with respect to the state s.
Can f-divergence IPMeb approaches achieve second-order bounds? Wang al. (2024b) has demonstrated that simply matching expectations (first moment) is insufficient to achieve second-order bounds in simpler supervised learning setting. Mapping this the II. setting, if the true cost is linear in feature ((s), i.e., (s) 0* 0*))(,,, and we design discriminator g(s) = ((s), then IL algorithm that learns a policy to only match policy's expected feature Eawr ((s) to the expert's Essre Ø(s) cannot achieve a second-order regret bound. For f-divergence such as JS- divergence, unless the discriminator class is rich enough to include the optimal discriminator, the discriminator-base objective is only loose lower bound of the true f-divergence. On the other hand, making the discriminator class rich enough to capture the optimal discriminator can increase both statistical and computational complexity for optimizing the discriminator. Taking the exponen- tial family distribution as an example again, there the optimal discriminator relies on the partition functions Z, and Z, which can make optimization intractable (Pabbaraju et al., 2024).
B ERROR ANALYSIS OF OBJECTIVE
In this section, we show the following two things: (1) why directly replacing the diffusion score function of æ with an estimator g" in {(T)) (Eq. 1) incurs large error; (2) why the error is small in derived objective (T))
16

---

Published as a conference paper at ICLR 2025
On the one hand, replacing the score function of with g" incurs the following enror:
E E
E (st, -ace))-g"(ae g0(()) (s:" V. log lo
(NU(T
Se (e)
(V,,, logpi - (s:)) (20 (st, (8t) V., log pi (.:))
0"
soud (NU(T)
(a)
Here the scale of term (a) is bounded by the statistical error from score matching. However, the scale of term (b) unbounded since ge can arbitrarily deviate from both g" and v. log pf (5t). Hence, the error is potentially unbounded.
On the other hand, our derived objective bounded. To see this, we note that our objective incur the following error:
| logpp (st) Ve, log (s s) (st) V., log ( 18)]
E E
E
amidd tN Be Renige
V., logr, (s.) 9"' (5x)) (s))
E
E
sw-dd (T)
a (-)))
Here the equality is by Lemma 4. We observe that it is exactly bounded by the statistical error from score matching.
C PROOF OF THEOREM
C.1 SUPPORTING LEMMAS
The following lemma is from Wang et al. (2024c) and the fact that triangular discrimination is equiv- alent to the squared Hellinger distance up to a multiplicative constant: 2D, I DA S (Topsoc, 2000).
Lemma 1. (Wang et al., 2024c, Lemma 4.3) For two distributions p,q E [0, 1] over some random variable, denote their respective means by P and q. Then, it holds that
p -- - - 8// Var(p) (p. 20D
where Var(-) denotes the variance, and D2 (p.4) = [(vp(=) - vq(z))2 dx/2 is the squared Hellinger distance.
The following lemma is adapted from some known results from diffusion processes and score match- ing. shows that as long as the DS divergence (Definition is small, the Hellinger distance is also small under some mild conditions. Similar results can be found in some theoretical papers on diffu- sion models such as Chen al. (2023b); Oko et al. (2023).
Lemma 2. Let P E A(R2) and g Rd [0, + Rd. Define Q A(R)) as the distribution obtained through the reverse process of diffusion starting from (0, I) by treating as the "score function" " of the process. Specifically, Q is the distribution of zr of the following SDE:
dz (z; + 2g(zz, t)) dt v2 dB, Zo N(O
We assume the following:
(a) For all I. 2 0, the score V log Pt is Lipschitz continuous with finite constant;
(b) The second moment of P is upper bounded: Ex~P [||[[]] m for some >> 0.
Then, if
E [lx()x,)) - Vx. log Pr(())) 5
E x (T) xemaee-11
B
for some e > 0, we have
(
(m + d) exp((-T)
17

---

Published as a conference paper at ICLR 2025
Proof of Lemma 2. We consider the following three stochastic processes specified by SDEs:
dt v2 dB,,, Xo Pri dx; =(x; + 2V log (x))(x))) (r-(xx)
dy: = (y: + 2, dt + v2 dB, Yo ~ Pr;
dz, (2, + 2g( ()) dt 2 dB,, Zo (0,
To clarify the notation: the variables X,, Vz, above are defined in the reverse time order, whereas in the lemma statement, x follow forward time order.
By triangle inequality for Hellinger distance:
D (P.Q) (xT, S (Du(XT,yr) + Du(yT, 2T))
I 2DD (XT. - (yr,
We will bound the terms separately. Since the squared Hellinger distance is upper bounded by KL divergence (i.e., D I DKL). we seek to establish the KL divergence bounds instead.
First, by Item (a) and Girsanov's Theorem (Karatzas Shreve, 2014) (also see, c.g., Chen et al. (2023b); Oko et al. (2023)), we have
(
t) (1²) = (Te)).
DJ (xr,Yr) < DKL (xr ll yr) O T-
E
V log Pr-((x)) - 900-g(( g T T
(2)(()
Next, by Item (b) and the convergence of OU process (e.g., Lemma 9 in Chen et al. (2023a)), we have:
DKL(Pr [1, I)) ( d) exp(-T)) By data- paaacesrocesinggg incquality for f-divergence, we have
D2 (yr, s (P,, I))
I DKL (Pr (1, I)) = 0( (m + d) exp(-T))
Plugging them back, we complete the proof.
The following lemmas standard results from diffusion processes and score matching. We show them here for completeness. Lemma 3. Given any 9. we have
(g(x),(), b), Vx. log (xxxxxx)))))>))(
E (g(x), f), Vxe (xee)) |xo)
E XerDD
E
Xomoo (Xm Xo)
where ( denotes the inner product.
Proof of Lemma 3. It basically follows from integration by parts. We start from the left-hand side (LHS):
LHS = g(xi,t) Vx, (xx.) (a(e dx,,
g(xe,1) (x.Pe(x.) dx:
Vx, g(xx))))) pe(x)) dxt
(integration by parts)
Vxi g(xe, 1) px( xo) (x0) dx, dxo
No Xx
0 +
g(xe,t) Vx.Pi xo) (x(x dx, dxy (integration by parts again)
g(x1,() Xx, (xe xo) px(xe xx) po(xo) dx; dxo
Xo x.
RHS.
Hence, the lemma is proved.
18

---

Published as a conference paper at ICLR 2025
Lemma 4. Given any 9. have
xtmpi | xx, log 1x)p( xomppoxxirpple xx)) o 1) Vx, logpe(Xx |x0)
=B Vx log (xx)
| Dogg:((x) (xx xx) 12
Xo.x;
where we note that the right-hand side of the equation is independent of g.
Proof of Lemma 4. We prove the lemma by expanding the first term the left-hand side:
g((x t) Xx, logp((xi)
Vx, log log |
E xp
(x(xx.
-2 2 E
(g(xe,)) (), Vx, logg:(xe))
+
xp-Di
xpmpp
[xxx,, t) |12
E x0,xt
E
(g(x,,t) t), Vx. log xn))
E
V Xx, log
lo pt (x+
(Lemma 3)
[(xx,, 1) 1
Ux. lx xoo l|
-2 (g(xeit), t), Xx, log xo)
E
Xo.Xt
XaiXe
Xa.Nt
Vx, log (xe| xo) ||_
?
E
+
Vx, log log (x,)
Xo.X1
x
E
((x,,t) Vx, log xo)
E
Vx, log (x: (x0)
+ E Vx, log Pr
This completes the proof.
C.2 STATISTICAL RESULTS
For the ease of presentation, we will leverage the following quantities:
C(T.
E sndd (~)(() s
(s:,t) v. log (sc >) |lg(s,, t) V., log (s |s)
By Lemma 4, the above is equivalent to the following by replacing the conditional score functions with the marginal ones for both terms in the expectation:
L(T. g) = E
E
E [se, - Vs, logp? (sx) -
() - V., log (s))
andd (~U(T) (-
(4)
One can show that, fixing a policy n, we have
max C(T, g)
E
E
E
(s,, V log Pl (s))1 ]1]] = {(x)
(5)
(T)
sorueel- (8)
where the first equality is by observing that the second term Eq, (4) can minimized to zero by choosing V., lo (st) (recalling that we do not assume misspecification for score functions),
and the second equality is by definition.
Bounding the difference between π (1:K) and n is what we aim to do the following lemma. Lemma S. Under Assumptions I and 2, assuming misspecification of RL (i.e., n² may not be in IT), we have
d==)=== D4 (u d= =00 =0(r 2 + T Cmis + -
ERL + T (-Regre((K)) (m d) exp( exp(-T))
Setting T = log( x tERL-Emui
mtd
yields the following bound:
D (4:8)) d δ 2 + Cmis + CnL +
-
Regret (K)
K
where Ô hides logarithmic factors.
19

---

Published as a conference paper at ICLR 2025
Proof of Lemma S. Items (b) and (c) in Assumption 1 implies the following:
K
max
L n (k)
9 -
(k) .g(h)) + Regret K))
(6)
#=1
L( (())) min [(x,, g (b)) I ena.
xell
Combining Equation (6) and Equation (7), we have
max
C(
Rk)
min [(T,, (k), + K <RL KRLL (K).
1=1
Now the right- hand side can be upper bounded by replacing all (*)) with maximization over g:
K
k
max
(k)
9)
min max L(T, 9) + K ERL + Regret( K)
xell
1=1
1=1
min (TT) + K ERL + Regret(K)
(by Eq. 5)
rell
K Emis K CRL Regret(K)
(8)
Note that the left-hand side is equivalent the following
K
max Σ (() .9) = K max L(m (1:K) (1:K) .9) = K (((x(1:)))
where the first equality is by the definition of C. Inserting it back to the previous inequality yields upper bound (1:K3)
{( (1:K) S Emis + ERI. + Regret(K)
K
Now define new distribution facilitate the analysis: let d= denote the distribution induced by the reverse diffusion process starting from (00,N(0,.) I) by treating g" as the "score function". Specif- ically, dxt is the distribution of 8T of the following SDE:
d3, (3, + 2g (3,, - t)) dt + v2 dB,, 30 ~ N(0,1)
where we recall that B, is the standard Brownian motion. We can see that d= is approximating d= Then, we invoke Lemma 2 and get
= ('mii Emis + -ERL
T R.RRegert() (m + d) exp(-I))
Dig Da
K
Now it remains bound D2, (dtT ,Tt') so that we can apply the triangle inequality get upper bound on D3 (txx) dd)). Since ((TT) I e (Item (a) in Assumption 1), we invoke Lemma
again and immediately get
D2 (d .F') s o 22o0e + (m + d) exp( (T)(-T))
We conclude the proof by putting them together via triangle inequality: ,d"') +DD d=))
Di (d (0) .d=)) s 2D,, (d (d (110)
-0 Emia + T 2 + +T T ERL
ERL + T Regret(K) (m d) e)exp(-1))
Lemma 6. Under the same condition Lemma 5 but assuming no misspecification (i.e., H² E II), we have
D2 (00)) .d"') ) + ERL + Regret(K)
K
Proof of Lemma 6. The proof is almost identical to that of Lemma 5 except that, when there is misspecification, min ((TT) in Eq. 8 can be simply upper bounded by {(TT), which is bounded by 2 Score (Item (a) in Assumption 1). The rest of the proof is the same.
20

---

Published as a conference paper at ICLR 2025
C.3 MAIN PROOF
In this section, we provide the main proof of Theorem 1. First, the following is by definition of value functions:
VT
H
(((s,))
E
(8,
By Lemma I, it is bounded by
V=(0)) V
S H. $ Var
E
c((,a) Dijlee d=lli)) , , ~ + 20D (c" ootinng d=
e,anddre
= =8/h Var" Diplet di , d=a) + 20HD (
where d (and d²') denotes the distribution cost rather than the distribution of d
(and d= itself. Now we apply the data processing inequality and get
( V" S / D2 (d ddr)) ) + 20HD (a (l.k
Plugging in the bound from Lemma 5 if there is misspecification error Lemma otherwise, get
V=(0)) V p="==ō =ō 0 Var" .e+ c+eHl
Similarly, we can derive the second half of the second-order bound (with (r((k)) replaced with Vo* inside the square root) by applying Lemma proof of the second-order bound.
Eq. 9 in the other direction. These conclude the
The first-order bound a direct consequence of the second-order bound as shown Theorem 2.1 in Wang et al. (2023).
D ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results: in Appendix D.I, we show the results of learning from state-action demonstrations; Appendix D.2, we analyze the impact the number of expert demonstrations on performance; in Appendix D.3, we study the expressiveness of diffu- sion models over other methods; in Appendix D.4, we include plots of unnormalized returns for reference; finally, in Appendix D.5, we provide detailed implementation details.
addition, visual demonstrations of the experiments (such as time-lapse images) are provided in Figure for better illustration.
D.1 LEARNING FROM STATE- ACTION DEMONSTRATIONS
Our method can be easily extended to learning from state-action demonstrations by appending the action vector to the state vector for both training and computing rewards. Hence, also explore its performance in such a setting. The experimental setup is identical to that of learning from state-only data. The training curves are presented in Figure 4. The results are highly consistent with those from the state-only setting, indicating that the performance gap is stable whether training on state data alone or state-action pairs. This also demonstrates that SMILING is robust different data types.
D.2 ON THE NUMBER OF EXPERT DEMONSTRATIONS
As we established Section 5, our algorithm can have better sample complexity in terms of ex- pert demonstrations. To empirically verify this, we conduct experiment with varying numbers of
21

---

Published as a conference paper at ICLR 2025
Crawl
Time
V
Expert
Expert
0.73
SMILING
0.5
(ours)
0.25
DAC
SMIT ING DAC
BC
(eurs)
Pole
Time
V
Expert
Expert
0.75
SMILING
0.5
(ours)
0
DAC
(ours)
Figure 3: Whole-body humanoid control via IL from state alone. The two panels illustrate the crawl and pole tasks, respectively. both tasks, we show the time-lapse frames of the expert policy and the policies learned by method (SMILING) and DAC after 3M training steps. In the crawl task, the goal is to crawl through grey tunnel, where both the expert and ours succeed and the crawling movements task, the goal is to travel through dense forest of poles. Ours successfully navigates through the poles, though with less stability than the expert, while DAC collapses to the ground and cannot move. The bar graphs on the right show normalized policy performance, where SMILING significantly outperforms DAC and Behavioral Cloning (BC) both tasks, approaching expert performance in crawl. Note that BC uses expert actions, while DAC and SMILING learn from states alone.
similar. However, DAC collapses and fails to complete the task. In the pole
expert demonstrations. We train SMILING and DAC with IK, 2K, 5K, 10K, 25K, and 125K expert states on humanoid-walk and compare the results. The relationship between the number of expert demonstrations and the final performance (after 3M training steps) is shown in Figure 5 and the training curves of both algorithms are shown in Figure 6. From Figure we observe that SMILING dominates DAC across all expert demonstration sizes. In particular, SMILING is able to approach expert-level performance with 5K expert states, while DAC requires 25K to achieve similar perfor- mance. This suggests that SMILING is more efficient than DAC in utilizing expert demonstrations.
D.3 EXPRESSIVENESS OF DIFFUSION MODELS
In Appendix A, argued that score matching is more expressive than discriminator-based meth- ods, which may explain why our approach generally outperforms DAC in previous experiments. conducted ablation study the cheetah-run task where
To further support this argument, our method previously showed a slower convergence compared to DAC. Now we remove the ac- tivation functions in both the discriminator of DAC and the score function of our method, making
22

---

Published as a conference paper at ICLR 2025
Ball in Cup Catch
Cheetah Run
Humanoid Walk
1.00-
1 (normalized)
0.75
m
0.50
0.00-
Steps
x100
Steps
x106
Steps
x 10
Humanoid Sit
Humanood Crawl
Humanoid Pole
1.00
V (normalized)
0.75
0.50-
m
0.25-
0.00
0
2
Steps
x 10
Steps
x10
Steps
X10
SMILING (ours)
DAC
IQ-Laam
DiffiL
BC
Figure 4: Learning curves for learning from state-action data across five random seeds. The x-axis corresponds to the number of environment steps (also the number of policy updates). The y-axis is normalized such that the expert performance one and the random policy is zero. The results consistent with those from the state-only setting. Humanoid Walk
1.00
1 (normalized)
0.75
0.50
0.25
Ours
0.00
DAC
10 25
125
Number of Demos
x10
Figure 5: Relationship between the number of expert demonstrations and the final performance. The x-axis corresponds to the number of expert states. The y-axis is the final performance that is computed the average of the last 100 training epochs. Each point the average of five random seeds.
them purely linear, and then re-run the experiments. The results are in Figure 7. Different from the previous experiments where DAC outperformed our method, in this case, DAC's performance degrades notably by showing slower convergence and significantly higher variance. In comparison, our method remains more effective and outperforms DAC clearly.
We attribute the performance degradation of DAC to the limited expressiveness of the discriminator, which results in mode collapse- well-known issue in discriminator-based methods, originally identified in GANs. In such methods, if the discriminator's performance falls significantly behind the generator (e.g., due to insufficient training the case here), the generator will exploit this gap and produce single-mode data that can easily fool the discriminator. In this case, the generator falling into a local minima. As the discriminator gradually catches up and learns to identify this data, the generator will discover new data to fool the updated discriminator, falling into another local minimum. This perpetual cycle of chasing
lack of discriminator expressiveness, which is
23

---

Published as a conference paper at ICLR 2025
Humanoid Walk (SMILING)
Humanoid Walk (DAC)
1.0
0.8
tn W i chooo M
1/28 (nommalized) (azzd
0.6
0.4
0.2
0.0
Steps
x10*
Steps
x10
1K
2K
5K
1K
2K
5K
10K
25K
125K
10K
25K
125K
Figure 6: The training curves of SMILING (left) and DAC (right) with varying numbers of expert demonstrations. The x-axis corresponds to the number of environment steps (also the number of policy updates). The y-axis the normalized return.
Cheetah Run (Linear Function)
1.00
1/at (normalized)
0.75
0.50
Ours
DAC
0.25
0.00
BC
Steps
x10
Figure 7: Learning curves with linear functions. Compared to using MLPs, DAC shows signif- icant decline in performance of slower convergence and higher variance, while ours remains more effective.
leads to continuous oscillations in the training process. We conjecture this the exact reason for DAC's instability in this experiments-Wiite the discriminator reduced to purely linear function, its expressiveness significantly weakened, allowing the RL policy to exploit it easily, and thus a perpetual chasing emerges. However, method remain more effective even with linear score functions, which aligns with our hypothesis in Appendix A.
D.4 PLOTS WITH UNNORMALIZED RETURNS
We provide the unnornalized-retum versions (i.e., with the y-axis not normalized) of Figures 2 and in Figures 8 and 9 for reference.
D.5 IMPLEMENTATION DETAILS
We summarize the difficulties of the tasks evaluated in experiments in Table 1.
24

---

Published as a conference paper at ICLR 2025
Ball in Cup Catch
Cheetah Run
Humanoid Walk
1000
750
500
250
0-
2
Steps
x 100
Steps
x10F
Steps
x
10
Humanoid Sit
Humanoid Crawl
Humanoid Pole
1000
750
500
250
0
2
Steps
x10
Steps
x10
Steps
SMILING (ours)
IQ-Leam
DifAIL
Figure 8: Same curves as in Figure but with unnormalized return V on the y-axis.
Ball Cup Catch
Cheetah Run
Humanoid Walk
1000
750
500
250
0
3
3
2
Steps
x10
Steps
x10
Steps
x10
Humanoid Sit
Humanoid Crawl
Humanoid Pole
1000
750-
250-
0:
-6
Steps
x10
Steps
x10
Steps x²
x10
SMILING (ours)
DAC
IQ-Learn
DiffAL
BC
Figure 9: Same curves as Figure but with unnormalized return V on the y-axis.
Implementation of RL Algorithms. We use the SAC implementation from Yarats & Kostrikov (2020) and the DreamerV3 implementation from Sferrazza et al. (2024) without modifying their hy- perparameters. The only change we made is substituting their ground-truth cost (reward) functions with cost function. For example, whenever the SAC agent samples batch of transitions from
25

---

Published as a conference paper at ICLR 2025
the replay buffer, we replace the corresponding cost values with those defined in our algorithm. We do the same for DreamerV3.
Expert Training and Demonstration Collection. We the aforementioned RL algorithms (SAC for DMC tasks and DreamerV3 for HumanoidBench tasks) to train the expert policies. We did not change the hyperparameters for these algorithms, making them the same those provided in their original implementations Yarats Kostrikov (2020); Sferrazza et al. (2024). In other words, these algorithms were directly applied their respective benchmarks without modification.
The RL algorithms were run until the performance plateaued, with the required number of train- ing steps varying across tasks: hall-in-cup-catee (0. steps), cheetah-run (4.8M steps), humanoid-walk (3.4M steps), humanoid-sit (2.7M steps), humanoid- crawl (5.4M steps), and humanoid d-pole (16M steps). The resulting expert policies achieve the following average total return: 972.85 for 794.12 for humanoid-sit, 914.31 for humanoid-cran and 879.48 for humanoid-pole.
915.16 for cheetah-run 875.61 61 for humanold-walk,
Once the expert training is complete, we generate expert demonstrations by running the learned expert policy for five episodes for each task. Since each episode has a fixed horizon of 1K steps, the resulting dataset contains 5 x 1, 000 5, 000 samples per task.
Implementation of Our Method and DAC. We made efforts to align the implementation details of both algorithms to ensure fair comparison. A comprehensive list of hyperparameters is provided in Table 2. We perform one update of the score function in our method and the discriminator DAC per IK RL steps. The expert diffusion model is trained for 4K epochs using 5K expert states (and actions, if learning from state-action pairs).
Implementation of IQ-Learn. We used the official implementation from Garg et al. (2021). We note that, unlike SMILING and DAC that can be built on any RL algorithm, IQ-Learn is exclusively tied SAC. This makes it unclear how to implement it top of DreamerV3, as we did for SMILING and DAC. Therefore, we adhered to their SAC implementation to run all experiments including the HumanoidBench tasks.
Implementation of DiffAIL. We used the official implementation provided by Wang et al. (2024a) and made their neural networks have same size ours. However, when applying their code to the tasks involving humanoids, we encountered numerical issues: the loss and weights of the neural networks diverged to infinity. We suspect this is due numerical instability in their implementation, which becomes pronounced in the high-dimensional feature space of humanoid tasks.
Table 1: List of environments ("DMC" DeepMind Control Suite, "HB" = HumanoidBench),
Task
Difficulty
Benchmark
Ball in Cup Catch
Easy
DMC
Cheetah Run
Medium
DMC
Humanoid Walk
Hard
DMC
Humanoid Sit
Hard
HB
Humanoid Crawl
Hard
HB
Humanoid Pole
Hard
HB
Table 2: Hyperparameters of our method and DAC. Shared columns indicate same hyperparameters.
Hyperparameter
Ours
DAC
Neural Net
MLP (1 hidden layer w/ 256 units)
Learning rate
5 x 10-3
Samples used per update
100,000
Batch size
1,024
Discretization steps
5,000 16
N/A
Time embedding size
N/A
26


Reviews:
[{"review_id": "ayZLcFU13l", "reviewer": "Reviewer_TxHS", "summary": "This work focuses on imitation learning from observations through the introduction of SMILING (Score-Matching Imitation LearnING). SMILING proposes to match the state distributions of experts and imitators by leveraging a diffusion model. \nThis study presents theoretical results related to SMILING, specifically regarding first and second-order approximation bounds of suboptimality. Additionally, the authors theoretically demonstrate that score-matching methods can achieve superior data efficiency compared to traditional f-divergence or Integral Probability Metric (IPM) matching methods, particularly under deterministic transition scenarios.", "questions": "Q1. Is the theoretical result specific to state-matching only? or can it also be applied to (state-action)-pair score matching?\n\nQ2. LobsDICE [G] highlights a limitation of the state-matching objective, specifically that $D_f (d_E(s) | d_\\pi (s))$ fails when it ignores the directionality of expert trajectories, and they suggest using $D_f (d_E(s,s’) | d_\\pi (s,s’))$ instead (see the appendix of [G]). \nCan the score-matching objective $D_{DS}$ circumvent these issues?\n\n**Reference** \n\n[G] Kim et al., “LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation”, NeurIPS 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work presents a straightforward yet powerful objective: to match the score functions of expert and imitator state distributions. Despite its simplicity, this objective is strongly supported by theoretical results grounded in plausible assumptions. It is expected that the proposed method, SMILING, will show enhanced data efficiency relative to traditional imitation learning (IL) methods where the expert's (cumulative) cost is nearly deterministic as supported by the analysis in line 368.", "weaknesses": "I have main concerns in this study as follows:\n\n**1. Insufficient comparative analysis with other diffusion-based IL methods:**\n\nWhile this work introduces new theoretical contributions to the score-matching IL objective, the method itself lacks novelty. \nPrevious studies [A, B, C, D] have already explored integrating diffusion models with imitation learning. \nHowever, the manuscript does not provide a detailed comparison between SMILING and these existing approaches, both conceptually and empirically. \nFor instance, what are the key distinctions or what is the novelty compared to these methods? Under what conditions does the proposed method offer advantages over similar methods? Such discussions should be provided to clarify the novelty of this work.\n\nThis work appears to attempt to distinguish itself from other approaches by addressing the more specific problem of LfO, rather than standard IL. However, the manuscript does not show specific considerations for LfO; the stated objective could readily apply to state-action matching, as demonstrated in experiments (Figure 4).\n\n**2. Weak Experimental Evaluations:**\n\nThe experimental evaluation presented also raises concerns. In the manuscript, SMILING is compared only against two obsolete baselines: BC and DAC. It is recommended to include comparisons with recent state-of-the-art IL methods such as ValueDICE [E], IQlearn [F], etc.\nMoreover, the absence of comparisons with similar diffusion-based IL methods (e.g. DiffAIL [A], Diffusion-KDE [B], DBC [C], DRAIL [D] ...) further weakens the empirical validation. \nSuch comparisons are insufficient to establish SMILING as a competitive method. \n\nIn addition, the lack of experimental validation on theoretical results makes the paper less compelling. \nGiven the theoretical emphasis on data efficiency in Section 5, experiments quantifying the relationship between the number of expert trajectories and the converged normalized score are essential for substantiating these claims. \nAdditionally, providing experimental outcomes in both stochastic and deterministic cost scenarios would enhance the reliability of the theoretical results as well.\n\n**References**\n\n[A] Wang et al., “DiffAIL: Diffusion Adversarial Imitation Learning”, AAAI 2024.\n\n[B] Pearce et al., “Imitating Human Behaviour with Diffusion Models”, ICLR 2024.\n\n[C] Chen et al., “Diffusion Model-Augmented Behavioral Cloning”, ICML 2024.\n\n[D] Lat et al., \"Diffusion-Reward Adversarial Imitation Learning\", NeurIPS 2024.\n\n[E] Kostrikov et al., \"Imitation Learning via Off-Policy Distribution Matching\", ICLR 2020.\n\n[F] Garg et al., \"IQlearn: Inverse soft-Q Learning for Imitation\", NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9QuSZB1Mkj", "reviewer": "Reviewer_jpw4", "summary": "This paper addresses the problem of online imitation learning. To this end, the paper proposes an algorithm that matches the learner's distributions to the expert's using a cost function computed by a diffusion model. Theoretical derivations demonstrate the advantages of minimizing the proposed Diffusion Score Divergence. However, the experiment results are currently insufficient to fully support the effectiveness of the proposed approach.", "questions": "As noted in the weaknesses section, I have the following questions:\n\n- Are there any results that compare the proposed method to recent adversarial imitation learning baselines, such as DiffAIL?\n- Are there any results that compare the proposed method to offline imitation methods that utilize diffusion models?\n- Could the authors provide the performance and learning details of both the expert policy and the random policy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Motivation and intuition**\n- The motivation behind the proposed Diffusion Score Divergence is convincing, as diffusion models have demonstrated a strong capability for distribution modeling.\n\n**Theoretical contribution**\n- This paper presents solid theoretical contributions, including well-defined assumptions, lemmas, and derivations. Additionally, it discusses related topics such as sample efficiency, second-order bounds and comparisons between score functions and discriminators.\n\n**Clarity**\n- The writing is clear overall, providing both theoretical and intuitive explanations. The notations, formulations, and theorems are well-explained and easy to follow.", "weaknesses": "**Experiment setup**\n\nThe experiments are not sufficient. While the claims look promising, the proposed method only compares two baselines, behavior cloning (BC) and discriminator actor-critic (DAC), which is not sufficiently convincing.\n\nFirst, recent adversarial imitation learning baselines, such as DiffAIL [1], should be considered. \nAlthough these methods do not explicitly learn reward functions but instead train discriminators, I am not convinced that this work is orthogonal to these previous works. They are feasible in the experimental setup in this paper, which includes limited expert demonstrations and online learner demonstrations.\n\nSecond, prior works [2,3] demonstrate that diffusion models are effective policies for offline imitation learning. Although offline methods can suffer from compounding error issues, it is still necessary to compare to one of the above methods to show that the proposed score-matching approach is beneficial when online learner demonstrations are accessible. \n\n**Experiment details**\n- The details of the expert policy are missing. Since all experiments are normalized by the performance of the expert and random policies, the authors should report the specific performance values of these policies as reference points; otherwise, the reported results remain unclear.\n- The number of environment steps used to train the expert policy is also missing, which makes it difficult to draw conclusions about sample efficiency from the results in Figures 3, 4, and 5. Specifically, online imitation learning aims to leverage expert demonstrations to improve sample efficiency. If the number of training steps for the expert policy is on a similar scale (e.g., 10^6), directly applying reinforcement learning to learn the expert policy could be a simpler solution.\n\n[1] Wang, Bingzheng, et al. \"DiffAIL: Diffusion Adversarial Imitation Learning.\" Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2024\n\n[2] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" Robotics: Science and Systems (RSS). 2023\n\n[3] Pearce, Tim, et al. \"Imitating human behaviour with diffusion models.\"  International Conference on Learning Representations (ICLR). 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jPkXPpAnj", "reviewer": "Reviewer_fJoG", "summary": "This paper proposes a new imitation learning method using score-based diffusion models. Specifically, the authors perform score-matching to minimize the difference between the expert’s and learners’  state distributions. In this way, they can perform IL using states only and also state-action pairs. They provide theoretical bounds on sample complexity when score functions are used, and show that they provide tighter bounds than when IPMs or discriminators are used. The empirical results show that their method SMILING performs favorably compared to other baselines.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper's contributions seem fair. They apply previously established score matching using diffusion models to IL. There have been many methods that used diffusion models in imitation learning (Section 2) and thus I rate the novelty as low. However, the authors provide new theoretical bounds and comparisons to discriminator-based methods. \n\n- The paper is fairly well-written and straightforward. I did not check for the correctness of the proofs and thus cannot comment.", "weaknesses": "- The framing of the method in terms of inverse reinforcement learning confused me. As far as I know, inverse reinforcement learning infers a reward function from the state-action-next state tuples and uses that to learn a policy. However, this method seems to be pure imitation learning where actions are learned directly from states or state-action pairs. Section 2 seems to imply that SMILING does IRL but I  see this method as an imitation learning method (without a discriminator). Can the authors clarify if a reward function is inferred or clarify on how SMILING is indeed an IRL method?\n- Some parts of the paper are disconnected with the rest of the paper and might possibly be left over from previous drafts  (e.g. connection to IRL, SAC/DreamerV3, etc.).\n   - Lines 267-269: Were SAC and DreamerV3 used in the experiments? The experiments compared against DAC and BC. \n- It's unclear how Section 6.3 helps to show that score functions are more expressive than discriminators. The results just seem to exploit a known weakness with adversarial training (mode collapse). Does the expressivity here mean that score matching does not suffer from mode collapse?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6v0yz21cnT", "reviewer": "Reviewer_yQfZ", "summary": "Traditionally in Inverse Reinforcement Learning (IRL), when expert demonstrations are available, a generative and adversarial setup is often used. In this setup, the discriminator’s goal is to learn the distribution of the desired behavior from expert demonstrations, while the generator’s goal is to produce behaviors that fit the distribution implicitly defined by the discriminator. Methods such as GAIL have proven powerful but come with a drawback: the optimization process. In these settings with a generator and discriminator, the optimal solution is a saddle point; however, it is often difficult to achieve due to issues like mode collapse and instability in training dynamics. In this work, the authors propose using diffusion models as an alternative. They achieve this by learning the score of the state distribution from expert demonstrations and then applying the DS divergence, which measures the difference in scores throughout the diffusion process. This metric is then used as the reward function to train the policy. The authors demonstrate impressive results on challenging humanoid tasks, particularly the Humanoid Walk task.", "questions": "The paragraph at line 081 could be clarified for readability and comprehension.\n\nCould the authors elaborate on the sentence: \"We first fit a score function to the expert demonstrations. Then, at each iteration of our algorithm, we fit a score function to the state distribution of the mixture of the preceding policies, before using the combination of these score functions to define a reward function for the policy search step.\" Specifically, what does \"mixture of the preceding policies\" mean?\n\nYour explanation of the algorithm here is helpful, but this part could be more precise. For example, does \"mixture of preceding policies\" refer to averaging the state distributions from previous iterations, or does it imply some other form of combining past policies? It would also improve clarity if you could provide a concrete example or a diagram illustrating how this mixture is constructed and applied in the algorithm. These additions would address potential confusion and make the process more accessible to readers.\n\nI think there can be additional clarifications in the method: Algorithm 1, would be good to define $c^{(k)}$ explicitly either in the algorithm pseudocode or in the accompanying text. Additionally, adding how $c^{(k)}$ relates to the score functions mentioned earlier and if it's connected to the cost function in Eq. 3 would provide a more comprehensive understanding of the algorithm.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Impressive experimental results, clearly demonstrating the benefit from not having a GAN approach for IRL. \nClear motivation and formulation and writing is of good quality.", "weaknesses": "I think there are prior work with a similar motivation.  Can you elaborate on how your is novel compared to the following:\n\nhttps://openreview.net/pdf?id=NN60HKTur2\nhttps://arxiv.org/pdf/2004.09395\nhttps://arxiv.org/abs/2407.00626", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VI0wrkKnIK", "reviewer": "Reviewer_H2TW", "summary": "This paper introduces an Inverse Reinforcement Learning (IRL) algorithm called SMILING, which replaces the commonly used f-divergence with a novel approach called Diffusion Score Divergence. This new method measures the discrepancy between the expert's and the learner's state distributions along the same forward diffusion process. The proposed algorithm achieves significantly tighter first- and second-order instance-dependent bounds, supported by thorough error analysis throughout the entire algorithm. Experimental results demonstrate the effectiveness of SMILING across various standard control benchmarks.", "questions": "The same with the weaknesses I mentioned above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the numerous contributions and the excellent presentation of this paper:\n\n- The idea is an innovative application of score matching techniques to model the state distributions of the policy. Unlike previous approaches that rely on training a discriminator between expert and learner data, the theoretical results demonstrate that the score-based method is more effective and sample-efficient, avoiding issues such as unstable training and mode collapse.\n- The presentation is clear and highly informative. I particularly enjoyed the writing in Section 5.1, which intuitively explains why the score-based approach is more expressive than discriminators, especially in the context of linear models. The analysis in Section 5.1 is further validated by the experiments in Section 6.3, which is a great reinforcement of the theoretical insights.\n- Across various benchmarks, the method provides a significant improvement over baseline techniques in the experiments, showing strong empirical results.\n\nOverall, the paper's theoretical and experimental results convincingly demonstrate the novelty and effectiveness of the proposed method. This approach could potentially inspire other fields that rely on training discriminators to explore score-based techniques.", "weaknesses": "Although this paper presents promising results, there are three primary weaknesses that should be addressed to some extent. I will consider any responses to these concerns when recalibrating my final score.\n\n- **Time Complexity of Evaluating the Cost Function $c^k(s)$ in Eq.3**:\n  - The main concern is the time complexity associated with evaluating the cost function. In the experimental sections, the authors employ a DDPM framework with over 5,000 steps. This implies that evaluating the cost function for a given state requires 5,000 evaluations of the model, which could be prohibitive for more complex scenarios, particularly with higher-dimensional state spaces.\n  - If the authors are using a single randomly sampled time step to estimate Eq.3, the variance of this estimation across time should be incorporated into the final theoretical results. However, this issue is not addressed in the current analysis, raising questions about the method's scalability to high-dimensional state spaces.\n- **Choice of Data Aggregation**:\n  - In Line 3 of the algorithm, the score matching target aims to force $g(s_t, t)$ to model the marginal score of the state distributions from a mixture of policies $\\pi^{(1:k-1)}$. However, the paper lacks a clear justification for using this mixture-based approach instead of the more intuitive choice of selecting the prior learned policy $\\pi^{k-1}$. While the theoretical results focus on the mixture of policies, it would be valuable to demonstrate the necessity of this choice.\n  - Providing more empirical or theoretical results to verify the effectiveness of this data aggregation technique during training would help clarify its benefits. For instance, comparing the proposed approach with a baseline that only uses $\\pi^{k-1}$ could offer insights into the significance of data aggregation.\n- **Assumption (c) and the Rate of $\\epsilon_{RL}$ in Continuous State-Space Settings**:\n  - The rate of $\\epsilon_{RL}$ should be discussed in more detail, especially considering that score matching is typically applied in continuous spaces. Although some recent algorithms extend score matching to discrete spaces [1, 2], incorporating them into the current framework may not be straightforward.\n  - It would be beneficial to discuss how the proposed framework might be adapted or fit into established theoretical results in discrete spaces. This could include exploring potential modifications or providing insights into the limitations of the current approach when dealing with discrete settings.\n\nAddressing these concerns would significantly strengthen the paper by enhancing its scalability, clarifying the choice of data aggregation, and broadening the discussion on the applicability of the proposed framework in various state-space settings.\n\n[1] Meng, Chenlin, et al. \"Concrete score matching: Generalized score matching for discrete data.\" *Advances in Neural Information Processing Systems* 35 (2022): 34532-34545.\n\n[2] Lou, Aaron, Chenlin Meng, and Stefano  Ermon. \"Discrete diffusion language modeling by estimating the ratios of the data distribution.\" *arXiv preprint arXiv:2310.16834* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)