Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation
Authors: Anish Abhijit Diwan
Julen Urain
Jens Kober
Jan Peters
Venue: ICLR 2025 Poster (2025)
Keywords: imitation learning
energy based generative models
reinforcement learning
imitation from observation
Primary area: reinforcement learning
TL;DR: Proposes a new algorithm for imitation learning for observation that uses score-based models to learn the expert distribution's energy function and uses these learnt energy functions as reward functions.

Abstract: This paper introduces a new imitation learning framework based on energy-based generative models capable of learning complex, physics-dependent, robot motion policies through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert's motion data distribution and learns smooth, and well-defined representations of the data distribution's energy function using denoising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined in the manifold of policy-generated samples. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algorithms like Adversarial Motion Priors (AMP). Our framework sidesteps the optimisation challenges of adversarial imitation learning techniques and produces results comparable to AMP in several quantitative metrics across multiple imitation settings.

Published as a conference paper at ICLR 2025
NOISE-CONDITIONED ENERGY-BASED ANNEALED REWARDS (NEAR): A GENERATIVE FRAMEWORK FOR IMITATION LEARNING FROM OBSERVATION
Anish Abhijit Diwan' Julen Urain 2.5 Jens Kober't Jan Peters2.3.4.57
1 DDeaarmmett of Cognitive Robotics, TU Delft, Netherlands
Department of Computer Science, TU Darmstadr, Germany
"Hessian Center for Artificial Intelligence (Hessian.ai), Germany
'Center for Cognitive Science TU Darmstadt, Germany German Research Center for AI (DFKI)
"Equal supervision
ABSTRACT
This paper introduces new imitation learning framework based on energy-based generative models capable of learning complex, physics-dependent, robot mo- tion policies through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert's motion data distribution and learns smooth, and well-defined representations of the data distribution's 's energy function using de- noising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined the manifold of policy-generater sam- ples. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algo- rithms like Adversarial Motion Priors (AMP). Our framework sidesteps the opti- misation challenges of adversarial imitation learning techniques and produces re- sults comparable to AMP in several quantitative metrics across multiple imitation settings. Code and videos available at hwwph//wwwanggthu..oooooo sogthuubioonoiss-ccndttonneedd based-annealedd
INTRODUCTION
Learning skills through imitation is probably the most cardinal form of learning for human beings. Whether is a child learning to tie their shoelaces, a dancer learning a new pose, or gymnast learning fast and complex manoeuvre, acquiring new motor skills for humans typically involves guidance from another skilled human in the form of demonstrations. Acquiring skills from these demonstrations typically boils down to interpreting the individual features of the demonstration mo- tion - for example, the relative positions of the limbs in a dance pose - and subsequently attempting to recreate the same features via repeated trial and error. Imitation learning (IL. is an algorithmic in- terpretation this simple strategy of learning skills by matching the features of one's 's own motions with the features of the expert's 's demonstrations.
Such a problem can be solved by various means, with techniques like behavioural cloning (BC), inverse reinforcement learning (IRL), and their variants being popular choices (Osa et al. 2018
The imitation learning problem can also be formulated in various subtly differing ways, leading to different constraints on the types of algorithms that solve the problem. One notably challenging version of the problem is Imitation from Observation (IfO) (Torabi et al. 2018 2019 Zare et al. 2024], where the expert trajectories are only comprised of state features and no information about the expert's is actions is available to the imitator. This means that learning policy is not straight- forward as capturing the distribution of the expert's state-action pairs. Instead, the imitator must also
"Coresponding author; Anish Abhijit Diwan (ani shhdi Shhdiwan@gra wan@ a com

---

Published as a conference paper at ICLR 2025
learn to capture the dynamics of its environment. From practical perspective, the IfO problem is quite relevant as obtaining action-rich data for weal-world tasks - across several agent embodiments and at large scales is rather challenging. In most tasks, the expert only has an implicit represen- tation of the policy. Imagine how a dancer cannot realistically convey their low-level actions - like muscle activations positional targets in dance routine. Further, collecting action-rich data by telcoperating the agent requires significant human effort and often offers limited motion complexity. Imitation from observation hence closely depicts the data-limited reality of applying IL in the real world. Unfortunately. because BC relies on the expert's actions, a large fraction of BC techniques (including state-of-the-an diffusion- based algorithms like (Chi et al. 2023)) are inapplicable the problem of imitating from observation. Inverse reinforcement learning, on the other hand, can still be applied to such problems.
In this work, we mainly focus on observation- inverse reinforcement learning, where the agent recovers a sealar reward signal from the demonstrations that when maximised by updating the agent's policy, provides the agent with the "correct" motivation to imitate the expert. While reward learning in itself is a broad field of study, recent works that leverage generative adversarial techniques for this task have shown markedly good results Tessler et al.. 2023 Peng et al. 2021 Ho & Ermon| 2016 Torabi et al., 2018). The fundamental idea in adversarial imitation learning (AIL) to taneously learn and optimise the return from the reward function implied in the expert demonstrations through an optimisation objective derived from Generative Adversarial Networks (GANs) (Goodfellow et al. 2014). This GAN-inspired min-max optimisation procedure considers the agent's 's closed-loop policy as generator and simultancously trains a discriminator to differen- tiate between the motions in the expert data distribution and the motions produced by the agent's policy. The discriminator aims to correctly label samples from both distributions while the gener- ator aims to return an action that when applied to the environment, leads to features that resemble those in the expert's 's data distribution. The discriminator's 's prediction is used as a reward signal in reinforcement learning (RL) and the policy and the discriminator are updated iteratively until con- vergence. Although methods like AMP (Peng et al. (021). GAIL (Ho & Ermon 2016), GAIfO (Torabi et al. 2018)) (purely adversarial IL), and DiffAIL (Wang et al. (2024). DIFO (Huang et al.) 2024) (diffusion enhanced adversarial IL.) have achieved impressive results in wide variety of imi- tation tasks, they are prone to challenges intrinsic to their theoretical formulation. The simultaneous min-max optimisation used learn the reward function in these techniques is highly sensitive to hy- perparameter values. This causes adversarial learning techniques to have unstable training dynamics and learn non-smooth probability densities (Saxena & Cao 202T Kodali et al. 2017 Arjovsky & Bottou 2017: Goodfellow et al., 2014). These limitations ultimately lead to instability and poor reinforcement learning when using adversarial techniques to learn reward functions.
This paper explores non-adversarial generative framework for reward learning that completely sidesteps the limitations of previous generative imitation learning techniques. Our primary contri- bution is to use energy- mergy-based generative models as the backbone of the reward learning framework to learn smooth and accurate representations of the data distribution. We propose to use the learnt energy functions reward functions and present a new imitation learning algorithm called Noise- conditioned Energy-based Annealed Rewards (NEAR) that has better, more stable learning dynam- ics, and learns smooth and unambiguous reward signals. NEAR produces motions comparable to state-of of-tte-o--hh-arr adversarial imitation learning methods like AMP|Peng et al. (202T). Before diving into our proposed framework (Sections and we first briefly discuss the challenges of adversarial II. in Section 2 and score-base based generative modelling in Section 3
2 BACKGROUND: ADVERSARIAL IMITATION LEARNING
Given an expert motion dataset M containing i.i.d. data samples x = ss, E X implying a distribution PD. where X the space of state transitions [] adversarial II. methods aim to learn a differentiable generator (policy) Toe (s) S + A where S is the state space, A is the action space, and s E S is a sample drawn from the occupancy measure of the policy Pa- Similarly to a standard GAN, the idea here is to learn a differentiable discriminator (x) : X + [0, 1] that returns a scalar value representing the probability that the sample x was derived from pr. However,
In this paper define all expressions for the partially observable case, however, the same results also apply to the fully observable cases.
2

---

Published as a conference paper at ICLR 2025
Smooth Reward Landscape
Constant Energy Reward
Agent
#poch
opochh
99 +
Changiig
Non- Sen-Smooth Adversorial Landscape
Figure A comparison of reward functions (probability density approximations) Iearnt 2D target-reaching imitation task (left). In this task, agent aims to reach goal and expert demon- strations (pp) pass through an L-shaped maze. The learnt reward function is expected to encourage the agent to pass through the maze. In the middle, show rew(s' [s) for all reachable states around a state 8 (green circle) at different training epochs. On the right, we show an illustration of the non- smooth reward landscape of adversarial IL. The energy-basee reward is smooth (with continuous gradients), accurate representation of PD and is constant regardless of the distribution of policy- generated motions (pc). contrast, the adversarial reward non-smooth and prone instability. Additionally, it changes depending pG (the discriminator tends to minimise policy predictions) and can provide non-stationary reward signals (additional details in Appendix B.
14c6 Training Samples
16e6 Training Samples
18e6 Training Samples
20e6 Training Samples
Figure 2: Degradation of an adversarially learnt policy (AMP) in stylised walking imitation task. With sufficient training, the policy does learn to complete the task, however, performance fluctuates substantially throughout training (with degradation
16e6 training samples).
now there exists additional function ((Teg (s)) - X that maps the output of the policy
to the discriminator's input space. The discriminator is learnt assuming that W (T (s)) ) is an i.i.d. sample X and the generator learnt via policy gradient methods by using log Dø (W (mea (s))) as reward function (Peng et al. 2021 Torabi et al. (201).. The AIL optimisation procedure is as follows (J is the performance measure as per the policy gradient theorem Sutton et (999)..
min En- [log Dap (1)) + Ee-piog [log(1 Dap (W (s)))
(1)
max J where Ve (TTC) = Exog IQ (s, Ve log Tee (8, 6g
where Q (8, = Ereeg [log Do (W (TTa (s)))
Similarly standard sumple-generation-fouuuu GANs, AlL algorithms also suffer from having perfect discriminator (Arjovsky & Bottou, 2017). This leads to low discriminator predictions on the policy-generated samples, causing zero or constant rewards for the RL policy and ultimately leading to poor training dynamics. Moreover, the iterative nature of the AIL problem leads to constantly changing manifold of policy-generate samples and an arbitrarily changing discriminator decision boundary. This causes unpredictability in the rewards and introduces drastic non-stationarity in the RL problem. Lastly, the rewards learnt via adversarial techniques are non-smooth and do not always provide unambiguous signal for improvement. Here, smoothness refers to the ability of reward
3

---

Published as a conference paper at ICLR 2025
function to convey informative gradients in the sample space These challenges compound to cause fluctuations in the learnt policy and convergence to local minima (elaborated in Appendix E).
2
To conclude this section, we point to Figures| ! and that show visual example of the non-smooth nature of the discriminator and qualitative results demonstrating instability of an adversarially learnt policy. Having discussed several issues with adversarial II. the next section introduces an altema- tive method of learning the expert's 's data distribution using which we subsequently propose a new imitation learning algorithm.
NOISE-CONDITIONED SCORE NETWORKS (NCSN)
Score-based generative models are family of techniques recently popularised for generating realis- tic images and video samples. They model the unknown data distribution as Boltzmann distribution and generate samples by an iterative denoising process by traversing along the gradient of the data distribution's 's log probability (Song & Kingma 2021). Score-based models approximate the gradient of the log probability (called the score function) through a procedure called denoising score match- ing (Vincent) 2011). Similarly to GANs, the aim here is to learn probability distribution pe that closely resembles the data distribution pD. with pa _ <-"(m) (Boltzmann distribution) and E(x)) called the energy function of the distribution. Intuitively, the energy function a measure of the closeness of a sample to PD while the score (V x lo (z)) is vector pointing towards the steep- est increase in the likelihood of pp. Learning the score function implicitly also learns the energy function V log (x( v log(e- /2) = - F(F(). In this paper, we propose to explicitly learn the energy function to then use the energy of a sample to guide reinforcement learning. To do so, we make modifications
score-based framework called Noise Conditioned Score Networks
(NCSN) (Song& & Ermon 2019 2020
The underlying idea of NCSN is learn the score function by a process of artificial perturbation and denoising. Data samples are first perturbed by adding variable amounts of Gaussian noise. The score function is then learnt by estimating the denoising vectors that point from the perturbed data samples to the original ones. Given i.i.d. data samples {ur PD E RD), NCSN (Song & Ermon, 2019) formulates a perturbation process that adds Gaussian noise N (1x σ) to each sample x. where is the standard deviation representing a diagonal covariance matrix and is sampled uniformly from a geometric sequence {ai, 02 ..., J.J. Following this perturbation process, we obtain a conditional distribution yo (x']xx = =N(x'|aa [z. oI) from which marginal distribution yo (x') can be obtained as I 9o (2' (2)pp(r)dd dr. Given this perturbed marginal distribution. NCSN attempts to learn a score function (z, o): IR + I2" that points from the perturbed samples back to the original ones. The idea is to learn conditional function to jointly estimate the scores of all perturbed data distributions, ie., Yo E [o s0(2 σ) ~ Vz log 9- (xx). The score network is learnt via denoising score matching (DSM) Vincent 201 on samples drawn from the conditional distribution go (x'[1x)) P
The final DSM loss is averaged over the various values assigned to data samples in the training batch.
By perturbing individual data samples with Gaussian noise, NCSN essentially creates perturbed distribution that is a smooth and dilated version of PD (illustrated in Figure D with the standard deviation a controlling the level of dilation. This perturbation strategy ensures that pp is supported in the whole sample space and not just a low-dimensional manifold in IR. D (manifold hypothesis (Fefferman et al. 20166] Cayton et al.) 2008)), ensuring well-defined gradients and allowing better score function approximation. It also ensures that the score function is accurately approximated in
data-sparse regions in PD by increasing sample density in such regions (Song & Ermon 2019).
NOISE-CONDITIONED ENERGY-BASED ANNEALED REWARDS (NEAR)
In NCSN, the perturbed conditional distribution g(ø (z)) is formulated as a Boltzmann distribution such that q(>" (x) 019i(a' / where DIST() is a function that defines some distance measure
2 While the gradient of the reward function is direct part of the policy update in policy gradient methods, a smooth reward function is necessary for "sensible" policy updates. We leverage the fact that LPsM (go (x')) LosM(ga (=f(=))) + const. (Song & Kingma 2021 Vincent
2011
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Noise-conditioned Energy- Annealed Rewards
Data: M = ((sss')) {or)
Initialise energy network co. policy Rea. and rollout horizon: : Initialise replay buffer B t- 0, annealing buffer A < 0, and annealing threshold
// Appendix B.2.2
Subroutine EnergyNCSN ():
while not done do
bMM + sample batch of transitions from M
Isigma t- sample batch of noise levels dı uniformly from [m:)/=1
Update Eg according to Equation 2 using pairing bM beigme
end
Subroutine RL ():
Initialise 0% = 01
while not done do
for ; = 0. do
T. - {18, a, s', rew rre--tf(e(s, ox)) borizon Joed
ox))
// Sect ion 5.1
Store T; in B & ((s, horizon] } in A
end
eo(A.or
progress =
mean enengy on switching to d
Switch 0% 0k+ if progress >> a and ox 0k-1 1 progress <a; Annealing
A + @ & update Tea using B
end
Algorithm:
Call EnergyNCSN
Call RL
between sample and its perturbed form. In our case, DIST, is the energy funetion of a Gaussian distribution with a standard deviation and is a smooth, dilated representation of the energy landscape of the expert data distribution pn- Our approach leverages the fact that for sample x X, DISTa (x) is essentially just scalar-valued measure of the closeness of x to pD. meaning that can be used as a reward signal to guide a policy to generate motions that gradually resemble those in pp-
This approach sidesteps several of the shortcomings of adversarially learnt rewards. Since DIST. is learnt via score matching on samples arbitrarily far away from the pp. it is both well-defined and continuous the relevant parts of the sample space X. Further, since DIST~() is a dilated version of the energy function of pn. it is also not prone to being constant valued. Continuity and informativeness the whole space indeed require an infinitely large a, however, realistically a a that sufficiently large to cover the worst-case policy would guarantee that the rewards are both smooth and non-constant in the parts of X where the policy-gencce -enengeeerreddd samples are realistically expected to lie (proof: Appendix A.Ip. Moreover, DIST,() is learnt using perturbed samples from pD and hence does not rely on the policy-generated samples. This means that is disconnected from the policy and is not prone to issues of high variance that come with simultaneous training, Finally, propose to train NCSN before training the reinforcement learning policy, eliminating any concerns of non-stationariiy. The following sections discuss the procedure learn these energy functions and other algorithmic details of our approach (Algorithm
4.1 LEARNING ENERGY FUNCTIONS
Given D-dimensional i.i.d. data samples {~~~ PD E RD} where PD is the distribution of state-transition features in the expert's trajectories, NEAR lears parameterised energy function c=(x', D + E that approximates the energy samples x' in a perturbed data distribution obtained by the local addition of Gaussian noise (2z, to each sample x. The idea here is to jointly estimate the energy functions of several perturbed distributions, co(x',
DIST,(x'), The sample's score is computed by taking the gradient of the predicted energy w.r.t. the perturbed sample, 5(2',o) = Vzreo(z',0) .). The en- ergy network is learnt via denoising score matching (DSM) (Vincent 2011
ie., Y o E {o,)/_
using this computed
5

---

Published as a conference paper at ICLR 2025
Annealing
After Annealing
(focused energy-based rewards for improved policy)
SUppYRA
ce(s
Po
supp
Fo
40 dloua
Figure 3: Annealing (during RL) ensures that the agent always receives a focused and well-defined reward signal, thereby motivating the policy produce motions similar pD- Here, PD is dis- tribution of the expert's state transitions in a 2D target-reaching task (introduced in Figure [ The learnt energy functions eø(-, o%) are illustrated dilated versions (L-shaped boundaries) of pp that are well-defined only inside their respective perturbed manifold ("inner" regions of the L-shaped boundaries). The manifold of policy-generated motions indicated by supp(Tea). The policy is shown to have improved from left to right since supp(Tee) ) for the improved policy is closer to pp- The reward function currently maximised by the agent highlighted in red. During RL, the agent starts at the energy function (reward) of lower noise level (co(:. oi.)) and switches to a higher one (eo(:, 0x++1) )) upon receiving a sufficiently high average return. Arrows indicate the gradient of the rewards in supp(Tee) (avg. score).
score and the final DSM loss in a training batch is computed as an average over the various values assigned data samples in the training batch.
1 losM(O)= 2 Ex
x'-
0²
(osm(tor)(is) e los((ot)
4.2 ANNEALING
We modify the definition of the perturbed conditional distribution <(>'(z) by flipping the sign of the energy function such that higher energies indicate closeness Pp- This is done simplify the downstream reinforcement learning such that the predicted energy can be maximised directly. Following the improvements introduced in Song & Ermon (2020), define es(2', .0) ca(')) where (xt) ( is an unconditional energy network This allows us to learn the energy function of large number of noise scales with a very small sized dataset.
The appropriate selection of the noise scale (fa)) is highly important for the success of this framework. σr, must be small enough that the perturbed distribution qo, () is nearly identical to pp- This ensures that the policy aims to truly generate samples that resemble those in pn. In contrast, T1 must be sufficiently large such that o0()'o0) well-defined, continuous, and non-zero for any sample that is generated by the worst-possible policy. This ensures that the agent always receives an informative signal for improvement. Assuming that policy degradation unlikely, σ must be such that supp(go, ()) effectively contains the support of the distribution induced by randomly initialised policy network. In practice, these are dataset-dependent hyperparameters.
'The score is the gradient of the energy function and the norm of the score scales inversely with a. The hence he approximated by rescaling the energy with 1 (Song & Ermon 2020).

---

Published as a conference paper at ICLR 2025
The trained energy network cg d) can directly be used as reward function to train policy network Tée (say initialised at fcu) using some fixed noise level σg. But how do we decide on an appropriate og? Assuming that during training the noise scale was set appropriately, 9r. () is On the other hand Yo, () is extremely dilated version of pD but but supp(1o, () is likely to have a high intersection with supp(TTaau) ). This means that any chosen noise-level d% offers tradeoff' between sample quality and e0(x,, .kk) being continuous and well-defined in the manifold of the samples generated by the current policy.
nearly identical to pD but supp( () has a low intersection with supp( ) [
3
We propose an annealing framework inspired by annealed Langevin dynamics and its predecessors (Song & Ermon Kirkpatrck et al. 1983; Neal| (201) to ensure that the learnt reward function is always well-defined and continuous while also gradually changing to motivate the policy to get closer to pD (illustrated in Figure B) Instead of focusing on sample generation, annealing in the context of reinforcement learning focuses on making gradual changes to the agent's s reward function. Our annealing framework hence depends on the agent's progress from imitation perspective. Training is initialised with the energy function of the lowest noise level in the geometric noise scale. Then, at every new noise level in the scale, the agent tracks the average return of the first few policy updates. The noise level of the energy function is increased if the average return of the last few policy updates is higher than some percentage of the initial return. We note that changing the reward function introduces non-stationarity in the reinforcement learning problem, meaning that the learnt policy is susceptible to degradation. To account for this, our framework also lowers the noise level if the return drops below some percentage of the initial return. This means that if the policy gets worse, the noise level decreases, thereby increasing the intersection between supp(g=() ()) and supp(Toa) and ensuring that the degraded policy still has an informative reward signal for improvement.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We evaluate NEAR (Algorithm D on complex, physics-dependent, contact-rich humanoid motions such as stylised walking, running, and martial arts. The chosen task set demands an understanding of physical quantities such as gravity and the mass/moments of inertia of the character and contains a variety of fast, periodic, and high acgeeeccelrrnnoon motions. The expert's ' motions are obtained from the CMU and SFU motion capture datasets and contain trajectories of several motions. For each motion, dataset of state transitions MM = {(s s')} is created to learn imitation policy. Rewarding the agent for producing similar state transitions as the expert, incentivises the agent to also replicate the expert's 's unknown actions.
F(s, a. (')) = task, tash (s, a, + w energy es
(3)
To understand the impact of motion data availability on the algorithm, we also train NEAR in single-clip setting using a single expert motion for training on challenging motions like mummy- style walking and spin-kick. Further, understand the composability of the learnt rewards, we train NEAR with both environment-supplics rewards (such as a target reaching reward) and ensrgy-based rewards learnt from different motion styles (to perform hybrid stylised motions). To incorporate the environment-suppliee task reward pask a, g) E [0. 1]. we use the same strategy from |Pen et al. (22TI) and formulate learning as a goal-conditioned reinforcement learning problem. where the policy' is now conditioned on goal y and maximises a reward r(s, a, s', .g) (Equation (3) Details of the tasks and goals can be found in Appendix B.I We also apply an additional reward transformation of tanh (f+ - 710/ where r' is the mean k = 3 policy iterations, r' = m(ean((Re-./hooizoo
return received by the agent in the last (/z)) This bounds the unnormalised energy reward fixed interval so that changes between the noise levels a are smoother. Additionally, it grounds the agent's 's current progress in relation to its average progress in the last few iterations. The policy is trained using Proximal Policy Optimisation (Schulman et al. 2017) and we use the following quantitative metrics to measure the performance of our algorithm.
As shorthand, we abbreviate the support of the distribution of state transitions induced by rolling out policy ).
7

---

Published as a conference paper at ICLR 2025
Table 1: A comparison of the avg. pose error (shaded, lower is better) and spectral are length (non- shaded, closer to expert is better) at the end of training. Stdev. across independent runs is shown as an error value (+))
Algorithm
Walking (74 clipe)
Rumning (26 clips)
Left Punch (19 clips)
Crune Pose (3
Mummy Walk clipp))
Kick clip)
NEAR AMP
0.51
-7.22 1.4
0.62
-7.24 1.x
0.37 sors
-687
0.94 gois
0.66 sow
0.78 2:
-5.90
8.78
0.65
0.32
9.95
0.82
13.84
0.58
-3.16
Expert
-3.99
-1.73
12.28
3.39
Table 2: A comparison of the avg. pose error (shaded, lower is better) and task return (non-shaded, higher is better) at the end of training with composed reward functions. Stdev. across independent runs is shown as an error value (+)).
Algorithm
Turget Reaching (walking)
Target Reaching (rumning)
Target Reaching & Punching
NEAR
0.94
2.75
am
L18 am
3.6 2M4
AMP
1.09
2.23
1,77
-0.15
3,85
This is the mean dynamic time warping (DTW) error (Sakoe & Chiba 1978) between trajectories of the agent's and the expert's poses averaged across all expert motions in the dataset. The DTW error is computed using || 2ell2 as the cost function, where Et and Kt are the Cartesian positions of the reference character and agent's bodies time step l. To ensure that the pose error is only in terms of the character's local pose and its global position in the world, we transform each Cartesian position to be relative to the character's s root body position at that timestep (&t f -, a-root and 24 f Zz
Average Dynamic Time Warping Pose Error:
root)
Spectral Arc Length: Spectral Are Length (SAL) (Beck et al. 2018 Balasubramanian et al., 2015) is measure of the smoothness of a trajectory and is an interesting metric determine the policy's ability to perform periodic motions in a controlled manner. SAL relies on the assumption that smoother motions are comprised of fewer and low-valued frequency domain components while jerkier motions have more complex frequency domain signature. It is computed by adding up the lengths of discrete segments (ares) of the normalised frequency-domat map of a motion (in this case, do not transform the positions to the agent's 's local coordinate system).
2011
5.2 RESULTS
We compare Noise-conditioned Energy-based Annealed Rewards (NEAR) with Adversarial Motion Priors (AMP) (Peng et al. 2021) and in both cases only use the learnt rewards to train the policy. AMP is used as a baseline since it is an improved formulation of previous state-of-the-ar techniques (Torabi et al. 2018 Ho & Ermon 2016) and has shown superior results in the state-only adversarial combination is trained 5 times independently and the mean per- formance metrics across 20 episodes of each run are compared. Both algorithms are trained for a fixed number of maximum iterations.
II. literature Each algorithm-
Figure shows snapshots of the policies trained using NEAR. We find that NEAR achieves very close imitation performance with the expert's 's trajectory and learns policies that are visually smoother and more natural. For the quantitative metrics, we use the average performance at the end of training for fair comparison and find that both NEAR and AMP are roughly similar across all metrics (Table T). In most experiments, NEAR is closer to the expert in terms of the spectral are length while AMP has better pose error. NEAR also outperforms AMP in stylised goal-conditioned tasks, producing motions that both imitate the expert's style while simultaneously achieving the desired global goal (Table 2 and Figure 4 bottom). From the experiments on spatially composed learnt rewards, we find that NEAR can also learn hybrid policies such as waking while waving. Finally, we notice that NEAR performs poorly in single-clip imitation tasks, highlighting the challenges of accurately capturing the expert's data distribution in data-limited conditions. Conversely, AMP is less affected by data unavailability since the discriminator in AMP is simply a classifier and does not explicitly capture the expert's 's distribution.
5.3 ABLATIONS
We also conduct ablation experiments (Table 3 that help identify the crucial components of NEAR. The main focus of these experiments is understand the contributions of annealing and the effects of
8

---

Published as a conference paper at ICLR 2025
Walking
Running
Crane Pose
Left Punch
Mummy Style Walking
Spin
larget Reaching & Punching
larget Reaching (walking)
Figure 4: Snapshots of the policies trained with NEAR. Mummy-style walking and spin-kick are single-clip imitation tasks. The bottom row shows goal-conditioned RL policies that also optimise an environment prrironment-provideed task reward.
Table 3: Avg. pose eтror (shaded, lower better) and spectral are length (non-shaded, closer to expert is better) in several ablated configurations of NEAR. Stdev. across independent runs is shown as an eπror value (1.).
Effect of task rewards
Effect of annealing
Cunfig
Walking
Running
Crane Pise
Confii
Walking
Rumning
Crans Powr
0.69 4.2
-7.1 LMM
0.52 2011
-6.09 188
1.38 asa
.6.3 1.29
sancal
0.5 aa
7.52 123
0A62 AuT
-7.24
0.4 waa
2g Expert
am
1:17
0.49
600--1
Espori
using cevirronment-proviiin task reward (without goal-conditioning) For walking and running, task reward favoured forward motion and episode length while for the crane pose task only favoured episode length. Given these parameters of interest, train ablated configurations of NEAR with either annealing or a reward function conditioned on a fixed noise level (az Rs 9.21) and with either only learnt reward (ea) or composed reward function (f: Equation
can be seen that the addition of the task reward leads to an improvement in the pose error. This is especially apparent in tasks like walking and running where the task reward is closely aligned with the imitation objective. It must however be noted that the addition of the task reward does mean that the agent has reduced closeness to specific characteristies of the expert's motion like spectral length, velocity, and jerk. Ultimately the closeness the imitation under combined reward still highly depends on the harmony between the two reward functions.
Annealing does not have a significant impact on performance in walking and running, however, leads an improvement in more complex, non-periodic cases like the crane-pose task. is possible that for complex tasks, the expert distribution is more densely concentrated. In this case, a higher noise level for complex task might be more informative than one for simpler task for which the expert distribution is more spread out (examples in Appendix (A2)). The increased information available from annealing might hence be the reason for better results with annealing in complex tasks.
LIMITATIONS & CONCLUSIONS
While NEAR is capable of generating high-quality, life-like motions and also outperforms AMP in several tasks, it is still prone to some limitations. The annealing strategy discussed in Section 4.2 does lead to progressively improving rewards, however, annealing at high noise levels also tends cause unpredictability in the received rewards. We attribute this unpredictability to the static

---

Published as a conference paper at ICLR 2025
nature of our annealing strategy. While the geometric nature of the noise scale indeed maximises the intersection between supp(go, () and supp(1oz4, ()) (Song & Ermon 2019). at higher values of k. a fixed percentage increase in the return does not always ensure that supp (T) is outside supp(q,, 0)), supp(gr.:1) ) (refer to Figure 3). This means that a change in noise level suddenly causes the energy function to be ill-defined on portion of the manifold of policy-generated motions, leading poor rewards for these transitions and the introduction of non-stationarity in the problem. This can be verified with the energy return plot, where the return often drops at noise level changes indicating that the changed noise level is suddenly low-rewarding (Figure [ Degradation at higher noise levels highlights the sensitivity of NEAR to the noise scale and is a limitation of this framework.
Improvements can perhaps be made by experimenting with linear noise scale, even more noise levels, or a more dynamic form of annealing-
Interestingly, ablation studies show that the addition of a task reward reduced the overall unpredictability of NEAR at the later stages of training. A reason for this could be the reduction in non-stationarity by the addition of a fixed component to the reward function. Finally, state-only reward learning techniques like NEAR and AMP are also quite sensitive to the motion dataset. We find that the policy is prone to converging locally optimal behaviour if the dataset contains bi- directional state transitions (such that (x, and (s', 8) are equally likely to occur).
Humanoid Crane Pose
Mesdaach Every Returt
Traiving Samples
x00
Humanoid Crone Pose
Despite these limitations, the fundamental idea of using energy func- tions as reward functions still seems quite promising. Improvements can be made in several directions in the future. For example, it might be interesting to explore the effects of different noise distributions (instead of Gaussian noise) in the NCSN component of NEAR. Im- provements to the annealing strategy could perhaps also be made by introducing softer noise-level changing criteria. One interesting idea could be to only use the updated reward function ppost-annealing) for the portion of the transitions that are within the support of the new energy function and maintain the old reward function for other transitions. This might greatly reduce the negative effects of anneal- ing shown in Figure [5] Other reinforcement learning improvements could also be made through ideas like targeted exploration, domain randomisation, or by changing the action sampling distribution from being Gaussian to a different probability distribution (Eberhard et al. 2023).
Petalalle Laee j di
NEAR
Transing Samples
<
Figure 5: higher noise levels causes drop in the energy reward's return.
Annealing a
To conclude, this paper proposes an energy- based framework for imitation learning in partially ob- servable conditions. Our framework builds on Noise-conditioned Score Networks (Song & Ermon 2019) to explicitly learn a series of smooth energy functions from dataset of expert demonstration motions. We propose to use these energy functions as reward functions to learn imitation policies via reinforcement learning. Further, we propose annealing framework to gradually change the learnt reward functions, thereby providing a more focused and well-defined reward signal to the agent. Our proposed imitation learning algorithm called Noise-conditioned Energy- Annealed Rewards (NEAR) outperforms state-of-the-art methods like Adversarial Motion Priors (AMP) in several quantitative metrics as well as qualitative evaluation across a series of complex contact-rich human imitation tasks.
ACKNOWLEDGMENTS
The authors acknowledge the use of computational resources of the DelftBlue supercomputer pro- vided by Delft High Performance Computing Centre this project was obtained from (i) mocap.cs.cmm www.ssscc..e.iu. (created with funding from NSF EIA-0196217) and (ii) mocap.ccs.sf..u (created with funding from NUS AcRF R- R-255-000--499-13 133--33-----1 and SFU Pres-
The data used in
ident's 's Research Start-up Grant).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative ad adversarial networks. arXiv preprint arXiv: 04862, 2017.
Sivakumar Balasubramanian, Alejandro Melendez-Calderon, and Etienne Burdet. A robust and sensitive metric for quantifying movement smoothness. IEEE transactions on biomedical engi- neering. 59(8):21262213, 2011.
Sivakumar Balasubramanian. Alejandro Melendez-Callderon Agnes Roby-Brani and Etienne Bur- det. On the analysis of movement smoothness. Journal of neuroengineering and rehabilitation, 12:1-11, 2015.
Yoav Beck, Talia Herman, Marina Brozgol, Nir Giladi, Anat Mirelman, and Jeffrey M Hausdorff. Spare: a new approach to quantifying gait smoothness in patients with parkinson's 's disease. Jour- nal of neuroengineering and rehabilitation, 15:1-9, 2018.
Lawrence Cayton al. Algorithms for manifold learning. eScholarship. University of California, 2008.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2):29-35, 2021. doi: 10. 10.1109/ 109/ 1MM.022130611344,
George Cybenko.
Approximation by superpositions a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5): -366, 1989.
Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, Dai-Jie Wu, and Shao-Hua Sun. Diffuusion imita- tion from observation. arXiv preprint arXiv:2410.05422 2024.
Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671-6800 1983.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. convergence and stability of gans. arXiv preprint arXiv: arXiv: 1705.0725, 2017.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey. Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
11

---

Published as a conference paper at ICLR 2025
Radford M Neal. Annealed importance sampling, Statistics and computing, 11:125-1333 139, 2001.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-basee based character control. ACM Transactions on Graphics (ToG), 40(4))1-20, 2021.
Hiroaki Sakoe and Scibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions acoustics, speech, and signal processing, 26(1):43-49, 1978.
Divya Saxena and Jiannong Cao. Generative adversarial networks (gans) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), 54(3):1-42, 2021.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 13::124881222 12448, 2020.
Yang Song and Diederik P Kingma. How to train your energy-basee models. arXiv preprint arXiv:2101.03288, 2021.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3:9-44, 1988.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-9, 2023.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 008.800700668....8 2018
Faraz. Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. arXiv preprint urXiv:/905. /900./3566 /3566, 2019.
Pascal Vincent. connection between tation, 23(7):1661-1674 2011.
matching and denoising autoencoders. Neural compu-
Bingzheng Wang, Guoqiang Wu, Teng Pang, Yan Zhang, and Yilong Yin. Diffail: Diffusion ad- versarial imitation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15447-15455, 2024.
Larry Wasserman. All of statistics: a concise course in statistical inference. Springer Science & Business Media, 2013.
Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions Cybernetics, 2024.
12

---

Published as a conference paper at ICLR 2025
A PROOFS & EXTENDED EXPLANATIONS
A.1 PROOF OF ENERGY FUNCTION SMOOTIINESS
In this section, we prove that an energy function learnt via denoising score matching (Vincent 2011 is smooth and well well-defined in the manifold of perturbed samples.
Lemma A.1. Let PD be distribution with support contained in closed manifold M C R We assume that PD is continuous in this manifold. Let 9o be a distribution supported in a closed manifold P C Id obtained by the addition of Gaussian noise N(x σ) - (exp-Bo(x))/2 to each sample x in pD (s.1. (x(x) N [x, (z, Then qa continuous in P and V. log (2) - Ex (I) Eo
is smooth in P.
Proof. The convolution of function with Gaussian kernel results in smooth function. By the same reasoning, F log do (a:) is differentiable in because fo continuous P.
Lemma A. shows that a perturbed distribution and its score function are both smooth in the mani- fold of perturbed samples. Given this perturbed distribution, denoising score matching aims to learn score function (xx σ) We σ) where eo (x, σ) ~ (z). From the universal approximation theorem (Cybenko 1989 Hornik et al. 1989), follows that a sufficiently large neural network can approximate any continuous function (score function in our case) on compact domain to arbitrary precision, meaning that Vzeo(x, or) a smooth function.
Theorem A.2. Given a distribution Oa that is supported in a closed manifold P and is also con- tinuous in this manifold, parameterised energy function learnt via denoising score matching on samples drawn from 9a is smooth in P.
Proof. Lemma A.1 implies that the gradient of the score function is smooth in P and can be approx- imated smoothly by a neural network V, (x,, o). Since a function with a continuous gradient in a domain is itself continuous in that domain, it follows that cø(x, o) is also continuous in P.
Continuity es(x, o)) in the whole space requires P to be equivalent to Rd. However, the anneal- ing strategy introduced in this paper and sufficiently large ensure that the manifold of policy- generated samples always lies in P.
A.2
ANNEALING DISCUSSION
A.2.1 WHY ANNEAL IF THE LEARNT ENERGY FUNCTION IS SMOOTH?
If the learnt energy function (. σR) is smooth, then simply maximising should still provide an unambiguous improvement signal to the agent. Why then must we anneal the energy functions?
Annealing not only ensures that the reward signal is well-defined, but it also progressively provides more "focused" rewards to the agent. Given score function s(x', .) "zre(("', = (a' =)/o², for any fixed sample x'. eerea 0++1) > Vere (x)) This means that any given point in training, the increase in received rewards for making positive progress is much higher for the (I 1hth energy function. This can greatly incentivise the agent to move closer PD.
A.2.2 WHY Do SOME TASKS SUBSTANTIALLY BENEFIT FROM ANNEALING?
Ablation experiments from Section 5.3 show that annealing has a significant positive impact on more challenging tasks like crane pose. We hypothesise that the expert data distribution pD is more densely distributed for some tasks while sparsely for others (Figure 6). Assuming that a newly initialised policy always starts in the same manifold, the agent would start at more informative reward function for a sparsely distributed pn than for a densely distributed pn- This means that a noise level change for densely distributed pp provides more informative rewards.
13

---

Published as a conference paper at ICLR 2025
Dense
Sparse
Figure 6: An illustration of a dense and sparse PD- In the case of the sparse distribution, annealing would have lower impact as the newly initialised policy would start out receiving much higher rewards.
B EXPERIMENT DETAILS
B. TASKS
The task reward and goal features for each imitation task are described below.
Target Reaching
In this task, the agent's objective is to navigate towards a randomly placed target. The agent's 's state augmented include goal g: x² where is the current timestep, and 2 the target's position in the agent's local coordinate frame. During training, the target is randomly initialised in a 240 are around the agent within a radius in the range [2.0, 6.0] meters. The agent is rewarded for minimizing its positional error norm and heading error to the target. Here, 2a is the agent's position, o is the agent's velocity vector, d' unit vector pointing from the agent's root to the target, and 8* is scalar desired velocity set to 2.0 for walking and 4.0 for runninng.
task 0.6 (exp || sell3) + 0.3
1
0.1 (1 -[1lvll )²)
1 exp (5 ())
(4)
Target Reaching & Punching
In this task, the agent's objective is to both reach a target and then strike it with left-handed punch. Here, the goal is a vector of the target's position in the agent's local frame and a boolean variable indicating whether the target has been punched, 9e target initialisation strategy as before with an are of 45 and an are radius in [1.0, 5.0] meters. The agent is rewarded using the target location reward when it is farther than threshold distance from the target and with a target striking reward when it within this threshold. The striking reward aims minimise the pose simultancously aiming to achieve certain end effector velocity and height. The complete reward function is shown below where o is the end effector position, of/1 is the end effector velocity vector, f. is the end effector height, and refS 4.0 and heefs 1.4 are scalar desired punch speed and height.
æt, punch state
We use the same
and heading
between the agent's end effector and the target while
1.0 near far
target has been hit ||,, - 2 1.2 otherwise
tosk
(5)
14

---

Published as a conference paper at ICLR 2025
far Equation
-2.0 2ē - 2525 -=ggg(( 1))) 0.4
2
phear
0.3 + 0.3
0.1
exp
1
5 (uiiladr,)
1 1 exp
|,511
(1 (110//11 - - + 0.2 (1 (1 - - h=)fs)2)
1
+ 0.3
(6)
Unconditioned Rewards
These reward functions were used in our ablation experiments. In this case, we do not use any additional goal-conditionigg
The agent is rewarded positively for every time step in the episode, en- couraging longer episode lengths. Further, the agent is also rewarded for relative positive displace- ment, with the clipping at 0.5 meters. The final task reward is a weighted combination of both. paak 0.5 0.5 F......5
Walking Running:
0.5
Crane Pose & Punching: The agent is rewarded positively for every time step in the episode, encouraging longer episode lengths. r.ack 1.
B.2 TRAINING & EVALUATION DETAILS
B.2.1 ARCHITECTURES
For both NEAR and AMP, the policy is a simple forwarorward neural network that maps the agent's state Gaussian distribution over actions, n0, (u(s), Σ) with the mean ((s) being returned by the neural network and a fixed diagonal covariance matrix Σ. In our experiments, the neural network is fully-connected network with (1024, 512) neurons and ReLU activations. Σ is set to have values of e- -299 and stays fixed throughout training. The critic (value function) is also modelled by similar network. The value function is updated with TD(A) (Sutton) 1988) and advantages are computed using generalised advantage estimation (Schulman et al.) 2015). When using the mnvironment-supplic task reward, we set task energy 0.5.
The NCSN neural network is fully-connected network with an auto-encoder style architecture. Here, the encoder has (512, 1024) neurons and maps the input to 2048-dimensional latent space. The decoder has (1024, 512, 128) neurons with the output being the unconditional energy of sample. We use ELU activations between all layers of the auto-encoder and use Xavier uniform weight initialisation (Glorot& & Bengio 2010) to improve consistency across different independent training runs. Further, we standardise samples before passing them to the network. The NCSN noise scale defined geometric sequence with σ 20, a 0.01, and L 50 following the advice
& Ermon (2020). Following(Song & Ermon (020 we also track the exponentially moving average (EMA) of the weights of the energy network during training and use the EMA weight during inference, as it has been shown to further reduce the instability in the sample quality. All models in this paper were trained the Nvidia-A100 GPU (Choquette et 2021
B.2.2 REINFORCEMENT LEARNING
We borrow the experimental setup from Peng et al. (2221) where the agent's state is a 105- dimensional vector consisting of the relative position each link with respect to the root body and the rotation of each link (represented as a 6-dimensional normal-tangent vector of the link's linear and angular velocities). All features are in the agent's local coordinate system. Similarly to Peng et al. (2021). we do not add additional features to encode information like the feature's 's phase in the motion, or the target pose. Further, the character is not trained to replicate the phase-wa wise features of the motion and the learnt rewards are generally only representation of the closeness of the agent's 's motion to the expert's s data distribution. The agent's actions specify a positional target that is then tracked via PD controllers at each joint.
15

---

Published as a conference paper at ICLR 2025
We use asynchronous parallel training in the IsaacGym simulator (Makoviychuk et al. 2021 for all experiments and analyses in this paper and spawn 4096 independent, parallel environments during both training and evaluation. Given initial policy, training is carried out by first rolling out the policy in all environments for a rollout horizon (16 in our experiments). During rollouts, an environment is reset if it happens reach the done state. A replay buffer is then populated with all agents' transitions obtained from the rollouts. Then, with rollouts paused, the policy and value function are updated with the data from the replay buffer. After the update, the rollouts are restarted with the updated policy. We use multiple mini epochs to update the policy and value function at every update step. In each mini epoch, several replay buffer. The number of mini-batches depends on the relative size of each mini-batch and the replay buffer.
of data samples are drawn from the
During policy evaluation, the same rollout procedure is used, however this time, the rollout horizon set to 300 and the networks are of course not updated. Performance metrics are recorded every main training epoch as mean across the k- 20 most rewarding environments. Finally, we use reference state initialisation to initialise all environments at a random state in the expert's 's motion dataset and use early termination to reset when the agent falls over. For certain tasks like spin-kick, we find that it is especially challenging to learn a policy starting from certain initial states (such as the jumping-off point). Additional exploration is required to learn the optimal actions starting from these states. In these cases, reference states are drawn from a beta distribution instead of uniform distribution (with 3 3.0 and punching, we use both reference motions during initialisation. In this case, both reference motions are used with probability of 0.5 and the target is initialised in the range [1.0, 1.2] when the agent is initialised with the punching reference.
1.0). For temporally composed tasks like target reaching and
B.2.3 EVALUATION METRICS
This is the mean dynamic time warping (DTW) error (Sakoe & Chiba 1978) between trajectories of the agent's and the expert's poses averaged across all expert motions in the dataset. Given a set of j expert motion trajectories i, of arbi- trary length Lj where each trajectory series containing the Cartesian positions of the reference character's joints f;,, D { terministically
Average Dynamic Time Warping Pose Error:
Naros where i, {#1 V/r first, we roll out the trained policy de- | across several thousand random starting-pose initialisations - Then, the k 20
most rewarding trajectories are selected to form a set of policy trajectories D = {Tm } J= where T.m and each trajectory has an arbitrary length Lm- The average dynamic time warping pose error is then computed as the average DTW score of all Tm across all expert trajectories ř, with | æ; ll2 as the cost function. To ensure that the pose error is only in terms of the character's local pose and not its global position in the world, we transform each Cartesian position be relative to rfoot).
Tm {=;}/<m
the character's root body position at that timestep (ă; +- 2, - 2foot and x f Z4
Spectral Are Length: Spectral Are Length (SAL) (Beck et al.. 2018 Balasubramanian et al., 2011 (2155) is a measure of the smoothness of a motion. The smoothness of the character's 's trajectory is an interesting metric to determine the policy's 's ability to perform periodic motions in a controlled manner. The underlying idea behind SAL is that smoother motions typically change slowly over time and are comprised of fewer and low-valued frequency domain components. In contrast, jerkier motions have more complex frequency domain signature that consists of lot of high-frequency components. The length of the frequency domain signature of a motion is hence an appropriate indi- cation of motion's smoothness (with low values indicating smoother motions). SAL is computed by adding up the lengths of discrete segments (arcs) of the normalised frequency-domain map of motion. In our experiments, we use SPARC et al. (0188). more robust version of the spectral are length that is invariant to the temporal scaling of the motion. We track the average SAL of the k 20 most rewarding trajectories generated by the policy at different training intervals and use the root body Cartesian coordinates to compute the SAL. Note that in this case, we do not transform the positions to the agent's 's local coordinate system.
6 Deterministic here means that we use the predicted mean as the action instead of sampling from N ((((z))) E). This is done ensure that the computed performance not biased to any single initial state.
16

---

Published as a conference paper at ICLR 2025
B.2.4 HYPERPARAMETERS & TRAINING ITERATIONS
Table 4: Hyperparameters used in our experiments. pendix B.2
Architectural details are mentioned in Ap-
Hyperparam
Value
Reinforcement Learning
Discount Factor Y
GAE A
TD
Learning Rate
c
PPO clip threshold
0.2
Training horizon
Mini-batch size
2048
NEAR (Energy NCSN)
Batch size
128
(T)
20
0.01
Num noise levels
50
Exponentially moving avg. rate
0.999
Learning rate
c-5
Adam 3
0.9
AMP (Discriminator)
Batch size
512
Gradient penalty
5
Demo observations butfer size
2 e5
Discriminator loss coefficient
5
Discriminator output regularisation
0.05
Table 5: Details of NCSN training iterations (NEAR only) and reinforcement learning environment interactions (same for NEAR & AMP). 1; clips include turning motions. 1 clips include turning and punching motions.
Task
Num. Motion Clips
NCSN Iters.
Env. Interactions
Walking
1.5 e5
60 e6
Running
26
1.5 c5
60 c6
Crane Pose
1.0
60 e6
Left Punch
1.2 c5
80 c6
Mummy Walk
0.8 c5
80 e6
Spin Kick
1,2
100
Target Renching (walking)
22
e,5
100 e6
Target Reuching (running)
81
1,2
100
Target Reaching Punching
1.2
100 c6
B.2.5 REPEATABILITY & DETERMINISM
Each algorithm trained 5 times independently every task with separate random number gen- erator seeds for each run. However, using a fixed seed value will only potentially allow for determin- istic behaviour in the IsaacGym simulator. Due GPU work scheduling, it possible that runtime changes to simulation parameters can alter the order in which operations take place, as environment updates can happen while the GPU doing other work. Because of the nature of floating point nu- meric storage, any alteration of execution ordering can cause small changes in the least significant bits of output data, leading to divergent execution over the simulation of thousands of environments and simulation frames. This means that experiments from the IsaacGym simulator (including the original work on AMP) are not perfectly reproducible on a different system. However, parallel sim- ulation is a major factor in achieving the results in this paper and minor non-determinism between independent runs hence just an unfortunate limitation. More information on this can be found in the Envss benchmarks package, Note that this is only characteristic of the reinforcement learning side our algorithm. The pretrained energy functions are also seeded and these training runs perfectly reproducible.
17

---

Published as a conference paper at ICLR 2025
B.3 MAZE DOMAIN DETAILS
This section provides additional details the experiments and procedures used to generate Figurel) T In this experiment, the agent is initialised randomly in small window at the top portion of the L- shaped maze. The agent aims to reach the goal position at the bottom right (the episode ends when the agent's position is within some threshold of the goal). Expert demonstrations were collected, so the expert's trajectory did not reach the target directly but first passed through an L-shaped maze. The agent is expected to learn imitate this by passing through the maze. We train AMP and NEAR in this domain and visualise the learnt reward functions. In the case of NEAR, we visualise the energy function (eg(:, 07 with 20.0) and in the of AMP visualise the discriminator. The energy function is trained by training our modified NCSN on the expert state transitions in the maze domain. The discriminator trained while training the AMP policy. Figure [ shows two comparisons. We compare rew(s'(s) at fixed state in the maze at different training iterations. The energy-based reward function stationary throughout training and rew(s'|s only changes with s' for fixed In contrast, the adversarial reward depends on the agent's policy and keeps changing to minimise prediction for the samples in the policy's ss distribution pc-
There is no environment-proviee reward function in this domain and the visualisations are obtained only by using the learnt rewards. Apart from visualisation and comain-related differences, the train- ing regime for both NEAR and AMP in this task is identical to the training regime used in all other experiments in this paper. Please refer to Appendix B.2 for training details.
C EXTENDED RESULTS
Table 6: A comparison of the mean performance of NEAR and AMP at the end of training (Avg. pose error: lower is better. Others: closer to expert is better). Stdev. across independent runs is shown as error value (=).
Task
Algorithm
Avg. Pose Error (m)
Spectral Are Length
Root Body Velocity (
Root Body Jerk (2))
Walking
NEAR
0,51
-7.52
1.25
360.89 18436
AMP
0.51 108
-8.78
1.87 1.01
736.32 7825
Expert
1.31
130.11
Running
NEAR
0.62
-7.24 ::
3,52 0.37
1298.42 21542
AMP
0.65 0.01
-9.71 +: 1.54
3.79 0.14
1560. 14 à 87.18
Expert
-3.79
Crane Pose
NEAR AMP
0.94 = 0.15
-6.6 à: 1.97
0.12 0.22
46.77 +: 77.72
1: 0.80
1.18
0.01
19.29 à: 5.17
Expert
-12.28
0.03
49.05
Punch
NEAR
0.37 =e 0.05
-6.87 +: 1.47
0.01 #
11.34 +: 3.56
AMP
0.32
-9.93
0.06 0.02
29.6
Expert
-1.73
0.16
72.49
Mummy Walk
NEAR
0.30
.4.72
18941
AMP
0.41
-13.84 +
0.98
354.49 33.02
Expert
-4.11
0.73
79.63
Spin Kick
NEAR
0.78
-5.59
0.53
286.63
AMP
0.58 0.1
33.66 1: 6:33
0.5 d 0.1.
278.25 29.2
Expert
-3.39
1.05
273.61
18

---

Published as a conference paper at ICLR 2025
Table 7: comparison of wall-clock times of NEAR and AMP. Stdev. across independent runs is shown as an error value (+). We find that overall NEAR requires slightly more training time than AMP. Across all tasks, NCSN contributes less than 30% of the total computational time of NEAR, with RL accounting for the majority of the computation load and wall-clock time. Interestingly, AMP always has a larger reinforcement learning computational load. This expected as AMP learns both the policy and the reward function simultaneously.
Task
Algorithm
Avg. NCSN Wall Time (min.)
Avg. RL Wall Time (min.)
Total Wall Time (min.))
Walking
NEAR
10.19
15.07
25.26 1:052
AMP
18:46 0:5.
18.46 0.52
Running
NEAR
8.11 6:
15.15 :: 0.29
23,26 0.42
AMP
17.97 1:
17.97 0.44
Crane Pose
NEAR
6.61 6.16
13,92 =:
20.53 =:
AMP
17.16
17.16 0.
Left Punch
NEAR AMP
7,83
201.99
28.22 0:045
22,42 a: 061
22.42 n.6
Mummy Walk
NEAR AMP
5.07 6.
18,11 0:33
23,18 0.034
21.13 + 0:33
21.13 0.033
Spin Kick
NEAR AMP
7.73 0.22
19.94 1: 0.46
27.67 1.08
24.47
24.47
Tuget Reaching (walking)
NEAR
10.52
28.93 =
39.45
AMP
36.65 0.22
36.65 0.22
Target Reaching (running)
NEAR
8.08
28.71
36.79 0.87
AMP
34,95 J: 1:99
34.95 1.99
Target Reaching & Punching
NEAR
7.93 a 0.09
28.68 =: 0203
36.61 0.00
AMP
37.54
37.54
D EXTENDED ABLATIONS
Table A comparison of ablated configurations of NEAR (Avg. pose error: lower is better. Others: closer to expert is better). Stdev. across independent runs is shown as an error value (+).
Task
Config
Avg. Pose Error (m)
Spectral Are Length
Root Body Velocity ("))
Root Body Jerk (57
Walking
anneal & c9
0.51 : 0.15
-7.52 : 1322
1.25 0:188
360.89 :: 184,36
anneal & F
0.42 002
201
0.88 1.0.
249.97 :: 196.56
oa & ag & F Expert
0.49
7.11
1.58
653.
0.42 002
8.7 12
1,25 +.0.5
360.86 154.68
-5.4
1.31
130.11
Running
anneal &
0.62
-7.24
3.52 0.37
1298.42 42 215.42
anneal & F
0.59 0.5
8.02 0ST
4.86 0.16
1875.44 1288
ag & eg
0.62 =: 0.18
-6.69 + 1.85
3.48 0.52
1334.1 #: 244.91
as & F Expert
0.57 0.04
8.61 1 0.8
4,76 1:0.1
1826.98
3.79
3.55
$13.68
Crane Pose
anneal & eg
0.94 0.15
66.6 0.6:100
0.12 :
46.77 4: 77.72
anncal & F
1.33 0.1
-7.24 #251
0.14
72.24 ::
as & at & F Expert
1.38 08
-6.03
0.07 0.1
29.5 40.22
1.23 0007
.4.4 + 100
0.02 0.00
16,3 4.455
-12.28
0.03
49.05
E ADVERSARIAL IL CHALLENGES & AMP DISCRIMINATOR EXPERIMENTS
In this section, we elaborate on the challenges of adversarial imitation learning and provide addi- tional empirical results demonstrating instability and non-smoothness in the AMP discriminator. As briefly highlighted Section 2 the root causes for the challenges of adversarial techniques are the
Foor NEAR, the total wall time is calculated by assuming that the wall times of NCSN and RI runs normally distributed, Total wall time NCSN avg. + RL. avg. + v NCSN and + RI. sdd.
19

---

Published as a conference paper at ICLR 2025
simultaneous min-max optimisation in their training procedure and the formulation of the discrim- inator as a classifier. Throughout training, the policy is updated to bring PG closer to PD, meaning that the support of Pc the manifold of the samples generated by the policy keeps changing. Simultaneously, the discriminator's 's decision boundary is also constantly changing to distinguish between the samples in supp(pd) and sup)(pe). This iteratively changing adversarial procedure leads to high-variance discriminator predictions and causes performance instability.
At any point in training. the discriminator is trained to discriminate pD from pC - and not pp from all that is not pn meeninng that it is quite accurate on samples in supp(pp) and supp(pa) but is often arbitrarily defined in other regions of the sample space (Arjovsky & Bottou 2017) 7). The changing nature of supp(pc) means that after policy update, some of the samples passed on to the discriminator could potentially have come from a region outside these two supports. Since the discriminator is arbitrarily defined here, it is likely to return misleading predictions, leading to misleading policy updates. This variance potentially further heightened by the stochastic nature of reinforcement learning techniques like Proximal Policy Optimisation (PPO) (Schulman et al. 2017 meaning that the agent's ss exploration is typically met with poor rewards We empirically verily this high-variance hypothesis through experiments AMP (Appendix E.I.I
Further, reward smoothness is indeed an important criterion for faster convergence and sensible pol- icy improvements in RL methods like PPO. An ideal data-driven reward function is both smooth in the sample space as well as consistent throughout training (stationarity). Unfortunately, be- cause the discriminator Dop is non-smooth in the whole space X and is arbitrarily defined in (supp(pd) U supp(pa)) - parts of the sample space that are unexplored and not in the demon- stration dataset M -, the rewards in this region are also non-smooth. Moreover, the iteratively changing nature of the discriminator's decision boundary means that the reward function is also non- non-stationary. These issues compound such that the agent often receives constant or arbitrarily changing rewards and hence the policy receives uninformative updates. Appendix E.1.2 discusses experiments that highlight this non-smoothness and non stationnatn in AMP.
Finally, adversarial learning techniques are also prone to poor performance due to perfect discrimi- nation. AArjovsy&& & Bottou (2017) introduce the perfect discriminator theorems that state that if pp and PC have disjoint supports or have supports that lie in low-dimensional manifolds (lower than the dimension of the sample space X), then there exists an optimal discriminator D* X - [0. that has accuracy and V (zr)
O'vx supp(pD) U supp(pc) (Theorems 2.1 and 2.2 in (Ar- jovsky & Bottou| 2017). ). They also prove that under these conditions PG is non-continuous in X and it is increasingly unlikely that supp(pp) and supp(pc) perfectly align (Lemmas 2 and 3 in (Arjovsky& & Bottou 2017)). In the case of All., the generator is indeed a neural network mapping low-dimensional samples (features 8 S) to the discriminator's high dimensional input space. Fur- ther, PG and pD are potentially disjoint and it is at least unlikely that their supports perfectly align. This means that the initial stages of training, the discriminator very quickly learns perfectly dis- tinguish between the samples in the expert dataset M and those in Pc. assigning a prediction of 0 to any sample in the agent's trajectory. When used as a reward function, log Da,, (TT,, (s)) = log O, instantly leads to arbitrary policy updates. Even when using modified reward formulation, say D() instead of log D(). the agent would receive a nearly constant reward, say c. Under such a constant reward function, the gradient of the performance measure quickly goes down to zero since the expectation of the gradient of the log probability of a parameterised distribution (Fisher score) is zero at the parameter (Wasserman| 2013). We verify these claims by replicating the experiments from Arjovsky& & Bottou (2017) on adversarial motion priors (AMP) (Appendix E.1.3).
To conclude this section we highlight that NEAR does not rely on such simultaneous optimisation and instead has a fixed, stationary reward function. Hence, NEAR is not prone to such instability and non-smoothness.
'W agree that the KI. diverge constraint in PPO might somewhat reduce the negative impacts of this, however, penalising policy change with worse rewards is still non-ideal.
20

---

Published as a conference paper at ICLR 2025
Discrimintorr Predictions
Discriminator Reward
130
Two (D7,,.PPPeee (e00))
sumplas
samples
Alter Se5 samples
After Sel samples
samples
Mter 10c8 samples
-10
-00
60
150
500
750
1000
1000
1950
Training
Trinning
Figure 7: Discriminator variance experiments. The policy was first trained for 2e6, 5e6, and 10c6 data samples. Then, with the policy updates paused, discriminator training was continued. We plot the rewards received across several independent policy rollouts and observe high variance. Given that the policy is unchanging, a high variance in the reward indicates poor reinforcement learning.
E.1
AMP DISCRIMINATOR EXPERIMENTS
E.1.1 HIGH DISCRIMINATOR VARIANCE
We conduct additional experiments to very the high variance in the adversarial motion priors (Peng al. 202T discriminator predictions. We train AMP humanoid walking task using the loss function from Equation (T) for the discriminator and Proximal Policy Optimisation (PPO) (Schul- man et al. 2017) to train the policy, The discriminator is slightly modified by removing the sigmoid activation at the output layer and instead computing the loss on s sigmo Ammoid(DD) (D))) 10 (same setup as the main experiments in this paper). Training is continued normally until some cut-off point. The cut-off point is varied across runs to obtain varying levels of intersection between supp(pn) and supp(per). Then, with the policy updates paused, we continue training the discriminator to maintain the learnt decision boundary and visualise the variance in the trained discriminator's predictions the motions generated by an unchanging policy. We hypothesise that as training continues and the supports of the two distributions get closer, the discriminator is less likely to samples from region outside supp(pp) U supp(pa). meaning that its variance reduces as training progresses. We observe that the discriminator's predictions indeed have quite high variance and the range of the predictions varies vastly across training levels (Figure Z Further, the variance indeed reduces over the training level, indicating gradually increasing intersection between supp(pp) and supp(pa). The adversarial optimisation is likely to get stabilised as the policy gets closer to optimality, however, training for the most part is still rather unstable because of the high variance in the discriminator's predictions.
E.1.2 DISCRIMINATOR NON-SMOOTHNER
We also conduct experiments understand the smoothness of the learnt reward function and its changes over training iterations B). To do this, we again train AMP on a humanoid walking task. This time we do not modify the algorithm and simply evaluate the discriminator at gradually increasing distances from the true data manifold at various points in training. We find that the discriminator's 's predictions on average, decline as we move farther away from the true data manifold. However, again, the predictions are quite noisy and have fairly large standard deviation.
1.1 PERFECT DISCRIMINATION
Finally, we replicate the experiments from Arjovsky Bottou (0117) on adversarial IL (Figure 9 We use the same experimental setup Appendix E.I.I but now compute the accuracy with the output of the final Sigmoid layer. Here, instead of continuing the discriminator's training, we retrain the discriminator to distinguish between samples in the expert dataset pp and samples in pG (same
This is done to allow the network to predict any arbitrary value and to allow more flexibility in the reward function transformatio.. The same is done in the original AMP procedure |eng et al. (202T) and we make no additional modifications their code.
21

---

Published as a conference paper at ICLR 2025
Moan Discriminttor Prediction
Discrininator Prediction Sid.
Mik semppis
samplo
1.70
4e ammple
Alter 33-5 sarplo
After 10el samples
Alte 40ed somples
Afw 10et sarples
After 40-6 somples
Klt 70e argll
Ahe one corple.
1.30
Afee 702 comple
Afte 5940 comple
2,00
0:00
105
Dusteces Ru /where sntt|-r.1) 10
2
Dastress Fan to (when
Figure 8: Discriminator non-smoothness experiments. Plots show the mean and std. discriminator prediction over large batch of perturbed expert data samples at varying levels of perturbation (where a distance of 0 the x-axis corresponds to unperturbed expert data in pD). Notice the high value the std. compared to the mean at any given distance from pD.
Discriminator Error
Discriminator Accuracy (on both pa and
10a
Altur 5 sample
1.0
After 2et samples
0.9
10-1
After bet somples
Accrrcy
0.7
After 0.5e6 samples
10-1
After 206 samples
After samples
0.4
0
250
200
750
1000
1200
1000
200
300
750
1000
1250
1500
Training Iterations
Training Itorations
Grad. Disc()
Discriminator Predictions
After 0e6 samples
0.5e5 samples
After 266 samples
311
Miter 266 samples
Alter sumples
samples
b
Phy Coa (-1)))
1
21
101
13
11
500
1000
15.0
750
1000
1300)
Training Iterations
Training Iterations
Figure 9: Perfect discriminator experiments on the walking task (multi-clip dataset with 74 motions clips). The policy was first trained for 0.5e6, 2e6, and 5e6 data samples. Then, with the policy updates paused, the discriminator was retrained. We find that the discriminator very quickly learns to perfectly distinguish between pp and PG (notice the logarithmic scale).
procedure as Arjovsky & Bottou 2017 Our experiments reproduce the results from Arjovsky
& Bottou (2017) adversarial IL (Figure 9.. The discriminator loss from Equation (1) rapidly declines indicating a near-perfect discriminator prediction and highlighting the fact that even after sufficient training, PG and pD are non-continuous. The accuracy of the discriminator reaches value of 1.0 in at most 75 iterations and V (x)()) rapidly declines to be 0, further corroborating the theoretical results. Finally, we also find that the discriminator's predictions on the motions generated by the policy rapidly drop down to zero, meaning that the policy receives unhelpful updates.
Despite these core issues of AIL, is still unclear why these techniques (and traditional GANs) still function comparably to basre-based alternatives. One reason could be that the changing discrimina- tor inputs mitigate the challenges of perfect discrimination. However, this still an open question that needs future work and deeper analysis.
22


Reviews:
[{"review_id": "HN3uV1mHbf", "reviewer": "Reviewer_Xdsd", "summary": "This paper presents a method to use generative adversarial networks in an imitation learning framework. The general idea is to create a reward function using the concept of energy from a generative network which is a metric for how close a sample is to being from a distribution. They directly optimize on this energy metric as a reward and perform experiments in high dimensional locomotion settings. They find their method is able to outperform baselines in some situations but ends up struggling in situations with data scarcity.", "questions": "iI there a reason you only compare to one baseline? Is that the only state of the art method and nothing can generally do better than it? I see line 364 says that it achieves superior results but is there truly no other baseline that even comes close? If so then that is fine. \n\nIs this the only/first work that uses energy-based diffusion models? If it is I would appreciate you explicitly saying so.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The research problem for this work is good. Imitation learning is a reasonable method for many control tasks where reward is difficult to specify and expert data is available. Aiming to improve the imitation policy from samples is an interesting a relevant problem.\n\nThe novelty of this work seems good but it is slightly difficult to tell (see weaknesses).\n\nThe algorithm details seem good. It is unclear to me what “until horizon” means in line 230 and 231. Other than that the algorithm description is clear.\n\nAt a high level the experimental section is good. I think the presentation could be improved slightly and have questions about the baselines and statistical rigor. The discussion is interesting and provides insight into the method.\n\nAblation Studies are good. The discussion on the importance of annealing is interesting and relevant.\n\nFailure Analysis is great. I really like the analysis and discussion of the reasons for failure. I think that leads to better future work and adds to the significance of this work.\n\nConclusion is good. I wish there was a future work section. Maybe more work into your annealing strategy?", "weaknesses": "I feel like as well the sentences from 71-77 are pretty vague and I would appreciate you explicitly stating the challenges clearly and then comparing your work even if is just “our method has smooth distributions and…”. The related work also seems to be wrapped in the intro which is fine but I would appreciate a more explicit comparison of other methods. Currently it simply says “this is what other methods do” instead of “here is how ours is different” which would make it easier to tell novelty.\n\nThe significance of this work seems good but it would be nice if the contribution was more explicitly stated. My interpretation was that the contribution is the use of energy based diffusion to train imitation learning but it would be better not to leave it up to the reader.\n\nBaseline comparisons are ok (see questions).\n\nThe presentation of results is ok but I wish the tables were bigger sized, I’m pretty sure the page limit is 10 so you should be able to simply increase the size.\n\nThe statistical rigor is ok. You say each is trained 5 times and it is the results are averaged across 20 trials. 5 seems low to me here. Especially since the confidence intervals seem to be sort of wide and overlapping. Doing maybe 20 or 30 would be much better unless this is prohibitively expensive.\n\nThe clarity is ok. I didn’t find this paper very easy to read. Generally, I think more intuition could have been used in the paragraphs that starts at 167, 189 and 154. I’m admittedly not extremely versed in GANs but I think it should be written to provide more intuition to non GAN experts and more RL experts (as that is the target audience in my opinion). As well, I don’t feel the figures are helpful at all. I don’t understand what figure 1 is trying to convey nor do I understand even what it means. As well figure 3 also doesn’t make sense to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLwh2RqDwK", "reviewer": "Reviewer_PK39", "summary": "This paper introduces Noise-Conditioned Energy-based Annealed Rewards (NEAR), a novel framework for imitation learning from observation using energy-based generative models. NEAR leverages denoising score matching to learn smooth representations of the expert's motion distribution and uses these energy functions as rewards. Unlike adversarial imitation learning approaches, NEAR avoids unstable min-max optimization, achieving smoother and more stable reward signals. Additionally, an annealing strategy progressively transitions between energy functions to provide more refined guidance for the agent’s policy. NEAR is evaluated on complex humanoid tasks, showing promising results when compared to state-only adversarial imitation learning baselines like Adversarial Motion Priors (AMP) in terms of motion quality, stability, and imitation accuracy.", "questions": "Could you provide a comparison of NEAR’s computational requirements (e.g., training time, GPU hours) relative to other baselines like AMP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. NEAR performs well on a range of complex motion tasks, including stylized walking, running, and martial arts. The results demonstrate competitive imitation accuracy and smoothness compared to AMP, particularly in complex tasks where AMP struggles with stability.\n2. The paper includes ablation studies to explore the impact of key components.", "weaknesses": "1. NEAR’s effectiveness is primarily evaluated in humanoid tasks, which are continuous and physics-driven. The framework’s applicability in other types of imitation learning tasks, especially those with discrete actions or diverse goal-oriented, is not fully explored.\n2. NEAR requires training a noise-conditioned energy model, which can be computationally intensive. A detailed comparison of training costs relative to other methods, particularly in terms of time and resources, would be beneficial.\n3. NEAR is only compared against one baseline - AMP and it doesn't seem to always be the winner (but has higher variance in most cases) despite the additional complexity of learning an energy network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RTdqAg5AwP", "reviewer": "Reviewer_Tb6J", "summary": "This paper proposes a method to perform imitation learning in absence of the expert's actions (i.e., only having access to the state trajectories). The proposed algorithm, NEAR, uses noise-conditioned score networks to model the probability distribution of the trajectories in the dataset. In this way, NEAR obtains a reward signal for imitation learning that does not depend on an adversarial network, as it is the case in current state of the art methods like AMP. In this way, known issues with learning in an adversarial setting are avoided. NEAR successfully learns to imitate reference trajectories, with similar performance to AMP and with smooth motion.", "questions": "* The authors mention that they use 20 test episodes to obtain the average performance, which sounds low compared to other RL papers. How large is the variability in performance across episodes? The confidence levels are provided across random seeds, so the variability of the performance within a seed is not evident.\n\n* You mention that AMP is less affected by data availability than NEAR. Shouldn't it also be a problem for AMP when data is scarce, since the task of the discriminator might become too easy when it can perfectly remember all the ground truth trajectories?\n\n* I did not fully understand why the energy function works well as a reward signal. If the energy is high when a sample is likely to be generated by the probability distribution of the ground truth data, why does the policy follow a trajectory instead of just reaching a high probability state? I thought that one reason can be that the energy function depends on the current state, so it will assign high energy only to the states that, according to the dataset, follow the current one with high probability. While this concepts are likely trivial for the authors, they should be more clearly explained in the paper for the less familiar reader to fully understand why the algorithm works. I would propose to use the example from Figure 1 to qualitatively describe why the displayed energy function is a good reward signal, if the energy changes as the agent moves, and other high-level considerations.\n\nI commit to increase the score if my questions and doubts highlighted in the \"weaknesses\" section are carefully addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is very well written. The algorithm presented in this paper (NEAR) is explained in detail. The authors clearly explain how this work is positioned within the field of motion imitation, providing helpful context information about adversarial imitation learning and noise-conditioned score networks.", "weaknesses": "It is unclear how Figure 1 has been generated. It is used for illustration purpose, however some more information about the energy function and the adversarial reward are necessary. I also do not understand the choice of using different scales for the two rewards, and why the energy is high around the agent's trajectory while the advesarial reward is low. I would ask the authors to detail how the energy function and the adversarial reward have been learnt (also in the supplementary material if it does not fit the main text).\n\nThe comparison with adversarial imitation learning methods is clear and well done. However, another possibility for imitation learning would  be, e.g., to just provide a reward signal in the form of the L2 distance between the states visited by the policy and the ones to imitate. NEAR seems, in fact, a more sophisticated version of this simple method. It would be important to explain why other non-adversarial imitation learning methods are excluded from the evaluation, and their differences and similarities with NEAR.\n\nThe algorithm presented in this paper does not seem to be a clear improvement over the baseline method (AMP). While the evaluation metric \"spectral arc length\" seems to favor NEAR, motions generated with AMP are generally closer to the ground truth. The authors should motivate why NEAR is better or at least a good alternative to AMP, e.g., proving lower sensitivity to the hyperparamenter choice, faster learning, less variance, more sample efficiency ... I encourage the authors to also compare the learning curves of the two methods, possibly including the number of interactions with the environment and the wall time necessary for convergence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUcSTgB5t1", "reviewer": "Reviewer_hjSn", "summary": "This paper introduces Noise-conditioned Energy-based Annealed Rewards, a new framework for inverse reinforcement learning that leverages diffusion models. Instead of using unstable and non-smooth adversarial learning to approximate reward functions, NEAR learns an energy function via score matching. NEAR provides smooth and accurate reward signals for training policies through reinforcement learning.", "questions": "1. **Comparison with DiffAIL [1]:** [1] also applies inverse reinforcement learning using diffusion models. Could the authors explain NEAR's major advantages compared with other works that integrate diffusion models into IRL? For instance, as the authors mention that GAIL can work well with single-clip data, how does NEAR compare in such settings, especially regarding data efficiency and performance? Further justification or emperial evaluation would be appreciated. \n    \n2. **Comparison with SMILING [2]:** [2] introduces a non-adversarial framework using diffusion models for imitation from observation and provides theoretical analysis. Could the authors elaborate on the differences and advantages of NEAR compared with [2]? Including further empirical results comparing NEAR with [2] would strengthen the paper and clarify NEAR's contributions relative to existing methods.\n    \n\n[1] B. Wang, G. Wu, T. Pang, Y. Zhang, and Y. Yin, “DiffAIL: Diffusion Adversarial Imitation Learning,” Dec. 12, 2023, arXiv: arXiv:2312.06348. Accessed: Oct. 19, 2024. [Online]. Available: http://arxiv.org/abs/2312.06348\n\n[2] R. Wu, Y. Chen, G. Swamy, K. Brantley, and W. Sun, “Diffusing States and Matching Scores: A New Framework for Imitation Learning,” Oct. 17, 2024, arXiv: arXiv:2410.13855. Accessed: Oct. 19, 2024. [Online]. Available: http://arxiv.org/abs/2410.138", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "NEAR replaces adversarial learning with energy-based modeling using diffusion models and score matching. This results in a stable and smooth reward representation, addressing issues of instability and non-smooth reward landscapes inherent in adversarial methods.", "weaknesses": "**Lack of Comparison with Related Work:** While the paper acknowledges limitations like stability with large noise levels and data requirements, a significant concern is the absence of direct comparison with other works that apply diffusion models to imitation learning from observation. Specifically, works like [1] and [2] also leverage diffusion models in inverse reinforcement learning. Since the authors claim that their work is the first to apply diffusion models for reward learning, providing further clarification (for instance, [2] is a general diffusion-based IRL algorithm, it could be helpful if the author could highlight the difference in the paper) and direct comparisons in the experiment section with these methods would greatly enhance the paper's quality and situate it within the existing literature.\n\n[1] B. Wang, G. Wu, T. Pang, Y. Zhang, and Y. Yin, “DiffAIL: Diffusion Adversarial Imitation Learning,” Dec. 12, 2023, arXiv: arXiv:2312.06348. Accessed: Oct. 19, 2024. [Online]. Available: http://arxiv.org/abs/2312.06348\n\n[2] R. Wu, Y. Chen, G. Swamy, K. Brantley, and W. Sun, “Diffusing States and Matching Scores: A New Framework for Imitation Learning,” Oct. 17, 2024, arXiv: arXiv:2410.13855. Accessed: Oct. 19, 2024. [Online]. Available: http://arxiv.org/abs/2410.138", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)