Generalized Behavior Learning from Diverse Demonstrations
Authors: Varshith Sreeramdass
Rohan R Paleja
Letian Chen
Sanne van Waveren
Matthew Gombolay
Venue: ICLR 2025 Poster (2025)
Keywords: Behavior Discovery
Demonstrator Heterogeneity
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose an imitation learning approach that utilizes a new diversity formulation to generate novel behaviors that generalize over demonstrators' latent preferences.

Abstract: Diverse behavior policies are valuable in domains requiring quick test-time adaptation or personalized human-robot interaction. Human demonstrations provide rich information regarding task objectives and factors that govern individual behavior variations, which can be used to characterize \textit{useful} diversity and learn diverse performant policies.
However, we show that prior work that builds naive representations of demonstration heterogeneity fails in generating successful novel behaviors that generalize over behavior factors.
We propose Guided Strategy Discovery (GSD), which introduces a novel diversity formulation based on a learned task-relevance measure that prioritizes behaviors exploring modeled latent factors.
We empirically validate across three continuous control benchmarks for generalizing to in-distribution (interpolation) and out-of-distribution (extrapolation) factors that GSD outperforms baselines in novel behavior discovery by $\sim$21\%.
Finally, we demonstrate that GSD can generalize striking behaviors for table tennis in a virtual testbed while leveraging human demonstrations collected in the real world.
Code is available at https://github.com/CORE-Robotics-Lab/GSD.

Published as a conference paper at ICLR 2025
GENERALIZED BEHAVIOR LEARNING FROM DIVERSE DEMONSTRATIONS
Varshith Sreeramdass, Rohan Paleja, Letian Chen, Sanne van Waveren, Matthew Gombolay Georgia Institute of Technology
(va (vareeramdass, jaa, letian .chn,, sanne (anne)@gatech.com ma matthew. matthew.gombolayy..... goombolayecc.....mm ay@cc. gatech. edu
expert
Guided Strategy Discovery task-relevant discovery
Perform ways generalized limited
demonstrations lask
State action apace
dintes
CSD lask relevance constraint f(ave)) ((bie)- Petie Pator.
Mtatos
relevant rogion
State-action
((xo)) (aaa-/
space
space
pieature
Refine relevance mussure
Encourage
relevant through GSD
stiles
back-immooeant behanrioss l.o. outside tadi relevaee region)
Novel anyogpplittingg
Colaped rectenggees the Intrord cocmtnuuuu Intent toctir: plasement
colordd-on onded
Labest coddes
Figure I: The figure overviews our framework: Guided Strategy Discovery. (a) Given diverse task demonstrations with underlying latent factors, (b) framework optimizes task-relevance guided diversity objective, (c) to discover behaviors that generalize to unseen latent factor values.
ABSTRACT
Diverse behavior policies are valuable in domains requiring quick test-time adap- tation or personalized human-robot interaction. Human demonstrations provide rich information regarding task objectives and factors that govern individual be- havior variations, which can be used to characterize usefil diversity and learn di- verse performant policies. However, we show that prior work that builds naive rep- resentations of demonstration heterogeneity fails in generating successful novel behaviors that generalize over behavior factors. We propose Guided Strategy Dis- covery (GSD), which introduces novel diversity formulation based learned task-relevance measure that prioritizes behaviors exploring modeled latent factors. We empirically validate across three continuous control benchmarks for generaliz- ing to in in-distribution (interpolation) and -itttf--dittibution (extrapolation) factors that GSD outperforms baselines in novel behavior discovery by ~21 %. Finally, we demonstrate that GSD can generalize striking behaviors for table tennis in a virtual testbed while leveraging human demonstrations collected the real world. Code is available at CithuulcommCORE--Rooooc-llaaCC
INTRODUCTION
Intelligent agents that encounter novel variation of learned task should be able to adapt their default decision making to suit the variation at hand. To adapt on-the-fly, agents must learn concise set of variations quickly tune their behaviors. Such adaptability is valuable in applications such as few-shot learning (Duan et al.) (017) and personalized human-robot interaction (Wang et al. (022) where limited examples or interaction must inform compatible approaches for task completion.
Behaviors that adaptable agents learn must be meaningfully diverse such that novel task variations can be addressed sufficiently. In the past, unsupervised reinforcement learning (RL) (Laskin et al.) 2021) has been used to learn diverse behaviors or "skills". However, learned behaviors intended to explore the agent's environment may not directly be useful towards task completion. Such ap- proaches further limited by their inability to identify and exhibit meaningful variations that useful during deployment. While supervised RL can employed, reward specification to align diverse behaviors with user expectations be cumbersome (Soares & Fallenstein 2014).

---

Published as a conference paper at ICLR 2025
In contrast to RL, learning from demonstration (LfD) methods enable agents to learn decision- making policies directly from human examples. The distinct variations that individuals often ex- hibit, even when pursuing the same task objectives (Sanderson (989). reflect creative ways of task completion. We assume that these variations are governed by distinct but latent behavior factors. These factors impart useful diversity in human behaviors that adaptable agents can exploit.
Prior work in heterogeneous Imitation Learning (IL) (Li et al.) 2017 Chi et al. 2023) largely focuses
on generating behaviors corresponding to modes training datasets or inferring representations of test behaviors. In this work, we address the challenge of generating behaviors with novel varia- tions. We specifically study the ability to inter- and extrapolate from demonstrations effectively produce new behaviors, that correspond to behavior factor values not seen in the training dataset, while also accomplishing the task. For example, consider robot quadruped that runs at differ- ent speeds. We seek policies that run at 2m/s or 4m/s from demonstrations with speeds of 1m/s and 3m/s. Such a generalization ability can provide task aass-accomplihhig behaviors with desirable charac- teristies directly through latent prior sampling. However, generalization with latent behavior factors is challenging. as we need to accurately identify the latent dimensions along which demonstrations vary and locate individual behaviors in the corresponding space before extending novel behaviors.
We focus on novel behavior generation in the setting of online IL (Ho & Ermon, 2016) due to its satals efficiency. We show that prior approaches that utilize mutual information (MI)-based diversity objectives (Li et al. 2017) fail to produce novel behaviors. We draw inspiration from recent work in unsupervised RL (Park et al. 2022] 2024 2023) that modify MI-based objectives and structure latent representations in order to induce specific behavioral traits (e.g., high Euclidean or temporal distances between states, controllability, etc). We propose to modify representation learning by restricting the latent space from capturing state-action space regions irrelevant to the task, identified through distillation of demonstration -rtpnsttttiio-ppcciiiie occupancy measures. We find that our formulation encourages diversity specifically along traits that vary across demonstrations. We refer to this objective as task-relevant diversity as it produces behaviors that retain pakk-peformmace.
We present a novel approach to learn diverse task cook-accomllishngg behaviors from demonstrations that generalize over latent behavior factors. Our contributions are four-fold:
We show the need for a novel formulation of diversity for generalization in IL from diverse demon- strations through experiments in 2D Point Maze domain (Sec. 4
We formulate task-relevant diversity, an objective to encourage diversity along factors of variation
among demonstrations by restricting representations from capturing irrelevant regions. We pro- pose Guided Strategy Discovery (GSD), an algorithm that optimizes diversity alongside imitation discover novel accoomllllliinnngg behaviors (Sec.5)) 5)).
We demonstrate that GSD generalizes to novel behaviors with 21% average eттor reduction in behavior factors (known during evaluation) over four baselines across two splits (interpolation and extrapolation) in three domains spanning robot control, driving, and manipulation (Sec. 6.1]
We demonstrate that GSD generalizes from physical human demonstrations to capture diverse stroke styles in a simulated Table Tennis domain (Sec. 6.3)
2 RELATED WORK
Generalization in Behavior Learning Prior works have studied generalization when agents
faced with task specifications from test distributions (Benjamins et al.]
2023]
Silva et al., 2021
Padalkar et al., 22033 Nair et al. 2022; Shridhar et al.) 2023] Xu et al. 2022
Driess et al. 2020
or deployment settings different from training environments (uu et al.
2018 Kumar et al. Xie al., 2023 Cobbe et al.
2020
Packer al.
2018 Kumar et al. 2021
2019)]
Osa et al. 2022
Zahavy et al.,
2022). ). In IL, generalization has been considered when demonstrators operate with
diverse conditions
QQu et al.
2023
Tangkaratt et al.) 2020; Chen et al.
2021 Paleja et al. 2020)
Schrum et al.) 2023b Li et al.
2017 Chen et al.) 2020 Wang et al. 2017 Li et al. 2017 Hausman
et al., 2017 Peng et al. 2022a).
Our work focuses on the latter, where we study heterogeneous
demonstrators with latent behavior factors. Among these, prior works either simply imitate multiple behaviors (Wang et al. 2017]] Li ct al.] (aa7smaa et al.) 2017: Peng ct al. (022a)), characterize
heterogeneity through latent representations (Paleja et al. 2020 Schrum et al. 2023b Li et al.
2017 Chen et al. 2020) or learn performant behaviors from diverse demonstrators (Qu et al. 2023
Demonstration diversity can also be attributed to varying sub-optimal ways of performing task Raa-
machandran & Amir 2007 Our work, however, focuses solely on task-optimal demonstrations.
2

---

Published as a conference paper at ICLR 2025
Tangkaratt et al.)
2021). Our work is the first to address learning both behavior representations and performant behaviors that generalize over behavior factors.
2020
Chen al.,
Learning from demonstrations Prior works in LID utilize demonstrations learn rewards
(Abeeel & Ng 2004 Ziebart et al., 2008 Fu et al., 2018 Chen et al.] 2020 Ross et al., 2011
or task-performmnnt polictes (Rosss et al. 201 Ho Ermon 2016 Qiu al. 2003] Tangkaratt al,
2020 Chen et al. (02T). Our work seeks diverse policies. particularly from expert demonstrations
with continuous latent factors (Wang et al. 2017 [i et al. 2017] Hausman et al.
2017
|017 CCen al.
2020 Peng ct al. 2022a). While several works address heterogeneous IL with large datasets (Chi ctt al. 2023), we use environment interaction to tackle covariate shift in low data regimes (Ho &
Ermon 2016 Kostrikov et al.) 2019 Reddy et al., 2020; Garg et al.) (02I). Our work belongs to the class of adversarial IL (AIL) methods (Orsini et al. 2021) which model imitation as adversarial
game between policies and discriminator that captures expert occupancy. We study and address
the limitations of MI-based diversity objectives used alongside AlL methods (Li et al. 2017 Haus-
man et al. 2017 Peng et al. 2022a) in capturing latent factors specific to demonstrations.
Diverse behavior learning Diverse behavior learning has been employed for exploration, pre- training, and generalization to novel environments. Quality Diversity (Batra et al., (024) assumes the availability of task performance metrics and functions for measuring behavior factors. In con- trast, our work is related to unsupervised RL (Laskin et al. (202)) that learn behaviors without such privileged information. Among them, are similar to competence-basee methods (Sharma al.)
2020 Hansen et al.
2019: Park et al. 2022; 2023 2024) that learn latent spaces to represent hetero- geneous behaviors. Works focus different aspects of diversity, such state coverage (Eysenbach)
cet al.
2018 Park et al.
2022 Laskin et al. 2022 Mendonca et al. (021), dynamics (Sharma et al.)
2020
controllability (Park et al. 2023 2024), etc. Our work adopts ideas of regularization (Park
et al.
2022
2023
2024) for designing diversity objectives to improve heterogeneous IL.
Structured methods for heterogeneous IL CASSI (Li et al. 2023) uses MI-based objectives to leamn novel locomotion behaviors from unlabeled data but relies additional rewards for task com- pletion, unlike our approach which does not depend on rewards. FLD (Li et al. 2024) structures latent spaces using differentiable fast Fourier transforms for periodic motions. In contrast, our ap- proach is domain-agnostic, making it applicable across a broader range of tasks. ASE (Peng et al,) 20226) applies latent sequence modeling combined with hyperspherical priors for smooth motion transitions, and CALM (Tessler et al.) (023) builds on ASE with latent-conditiondd discriminators. However, both methods suffer from limitations of naive MI formulations, which our work addresses.
PROBLEM STATEMENT AND PRELIMINARIES
We consider an infinite horizon, discounted, and reward- free Markov Decision Process (MDP)R) (S, A, P. Po: 7), where and A represent state and action spaces, P: S' A x S - E. the transition probabilities, Po: S > I. the initial state distribution, and 7, the discount factor. An optimal expert policy 2 is governed by a continuous factor, w E Q. The factor space, Ω2 is split into disjoint train and test regions, Tr(12) and Te(2), respectively. Given a demonstrations set, 2, of trajectories r = (50, 00, $1, ".., a₂ (-1s,, wi), 81-1 P(-(s,,ar) and wi E (T))), we aim learn policy ı that captures the expert behavior T over the entire factor space without access w; Ω.
We ground our approach in InfoGAlL (Li et al. 2017). built upon Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) to imitate demonstrations: JGAIL == E [log ((s, a)] + En [log(1 - Dss (ss,0 a]]]. where D a discriminator that distinguishes between the learned policy, n, and the expert policy, 75, InfoGAIL additionally introduces a latent variable, z Z, capture factors underlying expert demonstrations. InfoGAIL optimizes MI by variational lower bound (Barber Agakov| 2004), q(2((,,)) a). We refer g as the decoder as it infers from the state action pair. The InfoGAIl. objective is JinfoGAlL == JGAIL + >1 [log g ((2|s,a)) where A controls the diversity objective weight.
For formulating our diversity objective, we build on ideas from network distillation (Teh et al.) 2017 Czarnecki et al., 2019 Chen et al., 2020). MSRD (Chen et al. 2020) learns task rewards from demonstrations, }..). over finite set of factors by employing distillation with AIRL (Fu et al. 2018), a variant of GAIL that recovers a reward function, r(s, a). MSRD then distills the reward functions for each variation, Tę.. into common reward function for the task, Āu- The distillation is done by formulating each reward as, Tę, (s. a) := fo(s, a) + ng (s, a). The factor- specific residual reward, fz. (s, a), is encouraged to be close zero with the additional objective,
3

---

Published as a conference paper at ICLR 2025
Indicative Colors
Assigned
Assigned
Assigned
dim
-1
3
-2
1
1
-2
-1
-23
-2
-1
0
2
-23
-2
-1
0
2
dim
Demonstrations
Behaviors
Behaviors
Behaviors
Stte
State
State dim
dim
22
-2 State
0
3
a) Z, Demos
b) No regularization
c) Lipschitz constraints d) GSD (Ours)
Figure 2: Fig. 2a (Top): Map shows colors assigned to the 2D latent space. Fig. (Bottom):
The agent starts at (-2, 0), moves to (2, 0), passing through (-1, 0), (0, w) and (1, 0) where
Figs. P:. Latent vectors assigned to the state space, with state-only decoder, and policy behaviors, under high importance weight X,, are visualized. Trajectories are colored according to conditioning latent vectors per Fig. 2 (Top). Baselines deviate from demonstrations arbitrarily (Fig. Fig Bottom) or uniformly (Fig. 2 Bottom), disregarding the goal. GSD (ours, Fig. Z discovers behaviors with novel latent variations (waypoints along x = 0) while reaching closer to the goal (2, 0).
1]. Full details are in Appendix
MSRD = (Fcc (s, a) encourage the reward information common across demonstra- tions pertaining to the task to be represented by the task reward function ñn.
4 NEED FOR REGULARIZATION
In this section, we show that prior diversity formulations from al. (007): )); Park et al. (2022) fail to produce novel, task-accomplishing behaviors, motivating the need for a new formulation.
No regularization: InfoGAIL's diversity objective. MI, promotes diverse behaviors by rewarding the visitation of states associated with distinct latent vectors. In a 2D PointMaze domain with continuous state-action space (see Fig. 2 we show that an increased diversity objective's 's weight, λr, does not necessarily result in more diverse behaviors that accomplish the task. This result can be attributed to the decoder, q, neural network (NN) that assigns latent vectors to state-action pairs, (8,a). a). Without regularization, the decoder, q. produces unconstrained latent assignments, with high variety smaller regions (see Fig. 2b, top, several distinct colors close to point (-2, 0))- This finding aligns with prior work (Choi et al. 2021 Park et al. 2022). Without regularization, related behaviors with close-by states can mapped to unrelated far-away regions in the latent space without any meaningful structure. This behavior cause insufficient (see Fig. [p, bottom, several trajectories clump together along y = 0) or arbitrary (no pattern that governs deviation from demonstrations) behavior diversity.
Prior regularization methods produce misaligned behaviors: Prior works in unsupervised RL (Choi et al. T Park al. 2022) imposed Lipschitz constraints the decoder, enforce that for any two state-action pairs, (s, a). (s',a), the assigned latent vectors (specifically the mean a of the approximate posterior distribution q(11s, a)). differ by most the Euclidean distance be- tween the pairs, scaled by Aç. Formally, ||gg(-ss.)) Ma(-les.a) S Ac ||s C a - s' o a'Il, where denotes vector concatenation. The Lipschitz constraints ensure smooth latent vector assignments (see P. top). which encourages behaviors to deviate from demonstrations uniformly over the state space. However, resulting behaviors do not necessarily proceed towards the goal (see Fig, 2]. bottom). Other diversity formulations focusing on controllability and temporal reachability (Park) et al., 2023) 2024) will face similar issues the auxiliary objectives are misaligned with behavior heterogeneity. We propose a general formulation that encourages behavior diversity along latent dimensions inferred from the demonstrations, without compromising task performance.
5 OUR METHOD: GUIDED STRATEGY DISCOVERY
We present our approach for achieving generalizable II from diverse demonstrations.
4

---

Published as a conference paper at ICLR 2025
5.1 ENCOURAGING f RELEEVANT DIVERSITY
First, we present general approach for encouraging diversity selectively within state-action space regions indicated by high energy with respect to a scalar energy function, f: S x A - [0. 1].
We design our approach by analysing transitions that occur during learning. Consider a scenario
visualized in Fig. where an exploring agent at a high f-energy state-action pair, (s, a), assigned a latent vector and it enters another pair, {8', ,a)) If a different vector, z' s.t. 2' & were assigned to (s', d"), the diversity rewards, log q(z]s, a). log ((2'|s,, a'), would encourage behaviors T-),,,), ((1, 2"), to visit (s,a) and (s',a') respectively. The behavior, T(-|,, z'), would be desirable if (s', a') has high f-energy, as it would visit a high energy pair different from (s, a). increasing coverage of high energy regions. On the other hand, if (s',a)) were a low f-energy pair, the behavior (-11,2") visiting a low energy pair would be undesirable. In this case, the assignment for (s', a') could be constrained close to 2, which would remove the incentive for behavior distinct from (-),,2) to specifically visit (s', as))
Selectively allowing distinct latent vector assignments only in high-energy regions encourages be- haviors that target these regions, thereby promoting diversity only in high-energy regions. Constraint shown in Eq. [ormaliiee this intuition: For transition from (s,a) (s', a'), , the latter's latent vec- tor can be far from the former's 's by at most the Euclidean distance between the two pairs, scaled by f-energy of the latter and a factor Àc:
(.. - Ma(-le',a))
S A 1lc-l1ss [|s@a-55 s' e a'll ((',aa))
The proposed constraint (Eq. D disre- gards the energy of the starting state- action pair, (() The constraint enforces the same latent assignment for pairs in a low-llww energy tran- sition and allows different latent as- signments for pairs in a logh en- ergy transition as visualized in Fig. 3p. The enforcement can lead to connected low energy regions being assigned the
O high f-energy region
one latent vector assigned
multiple possible assignments
O unassigned
Latent assignemntt for R, C
Latent assignmentt for B,
state-action pair has high ff f-enengy
state-action pair low f-energy
Figure 3: This figure shows a visualization of state transi-
same latent vector which is different from that of reachable high energy re-
tions that an agent state-action pair, A, may undergo and the effect of our constraint in Eq. [ The light blue region indicates high femergy and white state-action pairs do not yet have assigned latent vectors. The lightness of the arrows indicates the slackness of our constraint (Eq. [ from scaling with f-neeryy on the right-hand side.
gions. Distinct latent vectors for low energy regions can result in behav- iors visiting those low-energy regions, which is not desirable.
We rectify our constraints to prevent latent assignments for low energy regions, by enforcing them only with transitions with high energy starting pairs through scaling the constraint with f-energy of the starting pair, as shown in Eq. [ Thus, when the starting pair is of low energy, the constraint implemented with a Lagrange multiplier is less effective due a smaller violation.
The modified constraints encourage the decoder to effectively use the latent space to solely represent high-energy regions. We refer to the resulting objective as f-relevant diversity.
f(s, a) [IlMa(-1ssa) - Pa alll S f(s,a) [Ac- I|s @ -s e''ll f(s' a')l
5.2 INFERRING A
MEASURE FROM DEMONSTRATIONS
Diverse aos-paccomllishing behaviors can be learned if an appropriate energy function / can be used to instantiate our f-relevant diversity objective. We now present an approach to derive such an f from demonstrations, utilizing occupancies captured by the discriminator, D.
Function f should indicate regions where agent occupation is favorable for the task while also cap- turing demonstrators' heterogeneity. We classify these regions as: (I) regions occupied by all ex- perts, and (II) subspaces where different experts occupy distinct smaller regions. We propose that capturing these two types of regions in the energy function provides us with an objective that en- courages task accomplishment and generalization over behavior factors.
While the discriminator, D. could be used to model f, it would capture type II subspace insuffi- ciently. D is trained to capture the union of demonstration occupancies, covering all type I regions,
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Guided Strategy Discovery
Input: Dataset of diverse expert demonstrations, 2 = (7.9
Output: Latent-conditioned policy capturing diverse behaviors, π
1: Initialize policy æ. task relevance f. factor-specific residual g. bias b, and decoder
2: for E {0, ..... epoch do
3: Sample =" from prior, -" using policy (-1 =²);;
+-),,=")); 75 from 9, infer 2 using decoder g
4: Compute discriminator outputs according to the conditioned distillation structure (Eq.
D(s, a, 2) a(As -f,, a) + 9(s, a, 2)] + b)
5: Update f. g. with gradient ascent on the discriminator objective computed by linearly com-
bining JGAIL (Ho & Ermon 2016) and the distillation objective (Eq. 4b:
Ere [log (ss, a, aa,5]]++ [|]+EEr>> Er= [o D(8, a, ] - Ex< ([g(,, a, 25 Ez [g](, a, 2 =)))
6: Compute the decoder objective using policy behavior samples (Eysenbach al. 2018):
Er-[log.A [My( s.a), lsa))
7: Update decoder with gradient ascent on the objective while enforcing task-relevance con- straints (Eq. 2): [Ac . ||s @ a - s' @ a'll f(s', a') |1 -s f(s, a) s 0
( 8: Update policy π with RL using linearly combined behavior imitation and diversity rewards, log( D(s, a, =) and log.V(2* (Mg(ls,,):
E)(e(s..)): (og.((="(Mg()e.)))
respectively.
9: end for
but only certain regions in the type II subspace that correspond to training demonstrations. / mod- eled in this way would limit behavior discovery beyond the training dataset.
We employ the distillation of demonstration-speeie to model f and capture type II subspaces. Demonstration-sseeiine occupancies are obtained by conditioning the discriminator, D(s, a, 2), the inferred latent vector, 2. It is further split linear combination of a iatent-independent measure, i.e., our desired energy function, f(s, a). and a dependent term, g a, z). in the logit space shown in Eq. 3 where g: S A x Z [0. 1]. o is the logistic function, Äs. a scaling constant, and b, a learnable bias. Às and b are introduced to transform the sum of bounded measures and enable D to capture most of the probability range [0, 1]. The discriminator is trained with an additional distillation objective, as shown in Eq. 4 to minimize the residual, g. only capture demonstration speciiic occupancy.
occupancy measures into common measure,
a(,,a = oAs - [f(s, + g(s, a, +b)
(3)
(4)
The objective, JR, encourages f to fully capture both type I and II regions. Type I regions are captured by f. as g is driven to zero where occupancy is common across demonstrations and latent- independent. g is encouraged to be close to zero even in regions with cemoostrationnsseeiff occu- pancy, causing it to capture minimal possible information and distilling the rest into f. We posit that indicates entire subspaces where occupancy is demonstration dependent, i.e., type II subspaces, while g indicates regions in these subspaces specific to each demonstration. We call our procedure for deriving f conditioned distillation (ConDist), due to its use of latent conditioning and distillation, similar to prior reward distillation frameworks (Chen et al. 2020).
Algorithm: We combine f-relevant diversity and conditioned distillation to propose Guided Strat- egy Discovery (GSD). High-level steps are outlined in Algorithm [ Complete details are in Ap- pendix C In each epoch, we sample behaviors using the policy conditioned on latent vectors from the prior (Line 3). Latent vectors for demonstrations are inferred using the decoder (Line 3. We use the proposed discriminator structure (Line(4.. define the imitation and distillation objectives and update the energy function, residual, and bias, using gradients (Line 5). We optimize the variational lower bound (Line while enforcing with proposed constraints (Line to update the decoder using gradients. Finally, update the policy with rewards from the discriminator and decoder (Line 8)
We posit that freelevant diversity and conditioned distillation are synergistic. An accurate f function representing demonstrations can guide latent assignments and associated behaviors to generalize beyond demonstrations. A latent space representing diverse demonstrations can help distillation capture regions beyond demonstrations that generalize latent behavior factors. Fig. 21 (bottom) shows that with GSD, the learned behaviors in 2D PointMaze capture novel latent variations by passing through waypoints along x 0 while reaching closer to the goal of (2, 0) better than baselines in Figs. 2p. 2 In addition, GSD produces a higher fraction of goal-reaching trajectories despite a low weight for the imitation objective and a weaker incentive match the expert.

---

Published as a conference paper at ICLR 2025
6 EVALUATION
We present empirical evaluation to answer the following research questions:
1. How do various methods perform in generalization to behaviors with novel latent factors while maintaining task performance? (Sec. 6.1
How do various methods structure behaviors in the latent space? (Sec. 6.2
3. How do various methods perform in learning diverse task-accomplishing behaviors from real- world human demonstrations? (Sec. 6.3
Existing benchmark, D3IL (Jia et al. 2024), focuses on discrete behavior modes. We instead use domains with continuous variation and clear task objectives to evaluate generalization from limited demonstrations. For Sec. 6.1 6.2] we use HalfCheetah (Wawrzyíski, 2009), Fetch- PickPlace Pllapeert et al. 2018) and DriveLancshift (Leurent 2018)) as they provide well-defined tasks with distinct dineeninnii (ID) factors. We script expert policies based these factors. In HalfCheetah, the robot runs at various speeds; in FetchPickPlace, the arm places an objects at differ- locations; and in DriveLaneshift, the ego-car overtakes at varying headway distances. 1D factors help avoid multiple sources of heterogeneity allowing careful examination of learned policies.
Domains:
Methods: We consider InfoGAIL our base method, representative of approaches that combine online HI with MI-based diversity. Heterogeneous online II. methods capture finite sets of behaviors (Chen et al. 2020) are omitted due to less scope for novel behavior discovery. While we incorporate some improvements from ad adversarial IL (Orsini et al. 2021) across all the evaluated methods, thorough evaluation of their integration with diversity objectives left for future work. Comparison with other online IL methods (Garg et al., 2021) is omitted as they do not directly address demon- stration diversity. Comparison with offline IL approaches that learn without environment interaction is presented in Appendix D We compare the following variants of InfoGAIL.
IG: InfoGAIL (Li et al. 2017) with continuous two dimensional latent variable.
IG+Lipz: IG with Lipschitz constraints for decoder q to investigate the uniform diversity.
IG+Con: IG with conditioned discriminator D(s, a, structure to investigate the effect of con- ditioning the discriminator.
IG+ConDist: IG with our proposed conditioned distillation to investigate the effect of extraction of a task-relevance measure (Eqs. 4
IG+ConDist+Lipzz IG+ConDist with Lipschitz investigate the uniform diversity formulation
alongside conditioned distillation.
GSD (Ours): IG+ConDist with our proposed task-relevant diversity formulation (Eq. 2
6.1 QUANTITATIVE EVALUATION
We investigate whether learned behaviors represent factor values in the disjoint test region Te(S2), after training on demonstrations, 2, from the train region, Tr(S2) (see Sec. We consider factors that measurable from trajectories (only for the sake of evaluation) to assess recovery performance, i.e., how well the learned latent space can represent expert behavior, by comparing desired and measured factor values. When diverse expert behaviors form distinct modes, this frame- work checks continuous factors underlying them can be accurately identified and generalized.
Splits: We divide the bounded 1D factor range into five consecutive equal-sized intervals:
The first, third, and fifth intervals represent the train region, and the second and fourth are the test region. The split allows us to evaluate the ability to interpolate behaviors to two factor space intervals while providing three non-consecutive intervals to represent the factor. Extrapolation: The second and fourth intervals represent the train region, while the first and fifth intervals are the test region. We choose two non-consecutive intervals for the train region to have sparse dataset while providing enough diversity to represent the factor.
Interpolation:
These splits evaluate how well the latent space captures factors interpolate and extrapolate behav- iors. We use five demonstrations per interval (details in Appendix B).
We search for desired behaviors using E {10, 20, 30, 40, 50) latent vector samples from the prior (-),,, where K represents the test time search sample-complexigg varied to investigate how well we generate desired behaviors from limited samples. We roll out policies conditioned on the sampled vectors, measure the factors of the sampled behaviors, and compute the least mean absolute error (MAE) between the desired and the measured values, averaging over 1550/K rounds. We
Metries:

---

Published as a conference paper at ICLR 2025
Interpolation
Extrapolation
Average
Worst
Average
Worst
HalfCheetal
Valocityy
0.5
8.8
Headway
FetchPickPlac DriveLaneShifs
4.5
98
log
20
40
50
20
40
Sampling Budget
|G+Lipz
IG+Con
CG+Coniist
IG+ConDisteLip
GSD (Ours)
Figure 4: The figure shows average and worst-case recovery errors over the three domains and two factor space splits corresponding to interpolative and extrapolative generalization. Shaded regions standard emrors over five train seeds. GSD outperforms baselines in recovery of unseen latent factors across most domains and splits.
Interpolation
Extrapolation
1.0
refer to this metric as the recovery error. We consider the midpoints of test intervals the set of desired values. We report average and worst errors over desired values the test region, pro- viding estimates of closeness between desired values and closest available behavior's factor, on and
HalfCheetah MAE in Velocity
0.8
(mils)
0.5
0.3
average and worst case. We report standard errors over five train seeds Fig. 4 evaluate task performance by averaging environ- ment returns over 1500 latent samples. We show
0.0
We
DriveLane MAE in Min Headwwyy
Higher return better
Lower error
recovery and task performance tradeoff in Fig 5
4.5.
Exact numbers are provided in Appendix H
better
2.2
Lipschitz constraints: For HalfCheetah (Fig. top row), IG+Lipz and IG+ConDist+Lipz have worse recovery errors compared to IG and IG+ConDist, indicated by the dark blue curve above magenta, and green above black, re- spectively. For DriveLaneshift (middle row). IG+Lipz and IG+ConDist-Lipz exhibit same trend against IG and IG+ConDist: for inter- polation, errors are improved shown by dark blue falling below magenta and green below black; and for extrapolation, the errors are worsened. For FetchPickPlace (bottom row), Figure 5: The figure shows the tradeoff between IG+Lipz and IG+ConDist<Lipz improve over task and recovery performance for three domains IG and IG+ConDist indicated by dark blue and and two splits. Error bars show standard errors green consistently below magenta and black re- spectively. Lipschitz constraints seem to be ben- efiting FetchPickPlace alone, which might be due "uniform diversity" aligning with object-
9.
FatchPickklaccs MAE in Obj-y
12
(cm,
10.0 0.4 0.6
0.8 0.9
1.00,0 0.4 0.6 0.8
0.9 1.0
Normalized return (log scale)
IG
IG+Lipz IG+Con
IonDist
CGCConiisttLipz
GSD (Ours)
over five seeds. High returns (x-axis) and low er- rors (y-axis) are better. GSD (circled red) im- proves recovery while retaining or improving task performance across most domains and splits.
8

---

Published as a conference paper at ICLR 2025
position factors, that is absent in other domains. This supports our hypothesis that relevant factors for diversity must inferred from demonstrations to benefit all domains.
compared to IG, indicated by the light blue curve below magenta. However, DriveLaneshift and FetchPickPlace, IG+Con seems worsen performance, with light blue largely above magenta in the bottom two Worsened errors may be a result conditioning on a latent variable capturing arbitrary factors.
Conditioning: For HalfChectah (top row), IG+Con improves
Conditioned Distillation: For HalfCheetah (top row), IG+ConDist and IG+ConDist+Lip/ improve over IG and IG+Lipz for extrapolation, indicated by black and green curves below magenta and dark blue respectively. They remain par for interpolation. For Drivel.aneshif row), IG+ConDist and IG+tonDist+Lipz improve over IG and IG+Lipz for extrapolation (K 2 30) and remain on par for/slightly improve interpolation. For FetchPickPlace (bottom row), the trends are interesting. IG+ConDist worsens errors over IG for interpolation and extrapolation, indicated by black above magenta. However, with Lipz's addition, IG+ConDist+Lipp tends close to IG+Lipz for interpolation and outperforms it for extrapolation. The patterns firstly suggest that conditioned distillation can improve extrapolation performance. addition, for FetchPickPlace where Lipschitz constraints are particularly effective, conditioned distillation can further improve extrapolation.
well (middle
GSD improves recovery over other approaches across most domains and splits, shown by the red curve below others in all plots, except for interpolation with HalfCheetah (top row, first two columns). HalfCheetah (top row), the close performance across methods may be attributed wide differences in gait styles across velocities that challenging to interpolate extrapolate. DriveLaneshirs (middle row), GSD reduces recovery error considerably over other approaches. In FetchPickPlace, GSD is closely matched by IG+Lipz IG+ConDist+Lipz Lips- chitz constraints already capture relevant factors. Nevertheless, GSD can further improve recovery.
Task-relevant Diversíty:
Tradeoff between Task and Recovery Performance: Across all domains, GSD either matches or improves average normalized returns over the latent prior, as indicated in Fig. 5 by the red cross generally being aligned with or positioned further right than others in all domain-split combinations but one. These results demonstrate the effectiveness of GSD's task-relevant diversity formulation in learning behaviors that reduce recovery error while maintaining task performance.
6.2 QUALITATIVE EVALUATION
We visualize the nature of the learned latent spaces for extrapolation in FetchPickPlace in Fig. IG+ConDist learns large behavior set for placing the object in the red test region (indicated by red cells in Fig. (6b). but ignores the dark- blue test region. While IG+ConDist+Lipz learns behaviors for all regions, it learns several that fail to place the object quickly enough (indicated by white cells in Fig. (6c). GSD learns behaviors that achieve the task (few white cells in Fig. (d) while representing all place locations equally in proximity to each other (roughly equal number of cells across colors nearby each other). GSD exhibits potential for improving the accountability of policy learning by enabling well structured latent spaces.
1.9
1.9
1.9
dimension
7 dlimeniion
7 ddmmnnsion
0.1
22.1
00.1
1.6
1.6
1.6
1.9
-1.0
.0.1
0.7
1.6
-1.9
-1.0
-0.1
0.7
1.6
1.9
-2.0
-0.1
0.7
1.6
dimension
Z.
/.dmennsoon
Z.
2'limension
(a) Fetch
(b) IG+ConDist
(c) IG+ConDist+Lipz
(d) GSD (Ours)
Figure 6: Fig 6a shows FetchPickPlace with object placement locations color-coded. Solid and dotted boundaries indicate train and test regions respectively. Figs/6b 6d Policy behaviors
shown in the 2D latent space through colors for resulting place-locations shown in 6aa White regions indicate failed placements or placements with low task reward. Behaviors with IG+ConDist (6b), IG+ IG++onDDstt-Lipz (6c) either represent the relevant regions disproportionately or fail to accomplish the task. Behaviors with GSD (6d) accomplish the task (low presence of white cells) and repre- sent all regions well (roughly equal number of cells across colors).

---

Published as a conference paper at ICLR 2025
6.3 EVALUATION WITH HEAL-WORLDS HUMAN DEMONSTRATIONS
We further evaluate our approach to test scalabil-
1.0
Push
ity to complex tasks with human demonstrations Table Tennis (TT) do-
0.8
Success Rate Ball Strikes
Higher success rate better
main. TT represents a
Higher
dynamic domain that quires precise robot mo-
0.4
diversity
Lob
0.2
is better
tion and fast reaction times while acting on noisy ob- servations. Our physical setup consists of a Barrett WAM Arm mounted to the ceiling in front of TT ta- ble, and a racquet attached as the arm's ss effector. Balls are launched using a But- terfly Amicus launcher a fixed orientation and velocity with some noise. Balls are detected and tracked using a YOLO ob- ject detector and Kalman Filter. An expert provides kinesthetic demonstrations of push and lob strokes. We recreate the setup in simulation with PyBullet for behavior learning. Ball initialization and observation noise levels in the simulation match real data. Complete details are in Appendix E
0
-6.6 -6.1 -5,5 -4,8 -3.9 -2.5 -0.0 Entropy in Max. Ball Heights (Particle estimate, log scale) IG=Con IG-Contiet
Figure 7: Left: The images visualize ball trajectories achieved by an expert kinesthetically demonstrating two types of strokes. Right: The figure shows the tradeoff between ball striking success rate and diversity in ball heights achieved. GSD outperforms baselines for both metrics.
While multiple continuous factors may exist underlying TT stroke styles, we evaluate generalization for maximum ball height, which we assume to be one of the underlying continuous factors. We eval- uate various methods in simulation for achieving high diversity in ball heights. We compute entropy in ball height values using particle estimates (Singh et al. 2003), after disregarding unsuccessful trials that fail to strike the ball over the table. We report the success rate traded off with diversity in ball heights Fig. [ (right). Our method GSD outperforms all baselines both measures of success rate and entropy.
7 CONCLUSION, LIMITATIONS AND FUTURE WORK
We study the problem of generalization from diverse demonstrations over underlying latent factors. We investigate the shortcomings of prior MI-based methods and propose novel diversity formu- lation. Our empirical evaluation shows that our approach improves the recovery of factors over the next best baseline (for K=50) by 18.3% and 24.6% for interpolation and extrapolation respectively while retaining task performance in three domains with synthetic demonstrations. Our qualitative analysis shows potential in improving the accountability of learned policies. Lastly, experi- ments with real-world human demonstrations shows that our framework can capture a diverse range of task-accomplishing behaviors in challenging domain requiring quick response times.
Our experiments focus on demonstrations with one-dimensional latent factors. Our approach may struggle with higher dimensional non-Markovian factors, which could require spe- cialized designs for disentangling dimensions or capturing observation history dependence. Scaling visual domains, where continuous factors must be inferred from sparsely distributed demonstra- tions, may also be challenging, as simple architectures for the energy function f may not generalize well. Furthermore. our assumption that demonstration occupancies correlate with task success may be violated with non-expert demonstrators partial observability, which may require state estima- tion models. While current work prioritizes validating our core contributions, we plan to evaluate scalability in future work.
Limitations:
Our evaluation with human demonstrations is further limited quantitative metrics. We aim to conduct user studies to subjectively evaluate behaviors in human robot interaction settings. Our evaluation is further limited to simulated domains. We aim to explore the efficacy of our diversity formulation for learning novel behaviors in physical robot systems with improved data-sample effi- ciency. We further aim to explore the thcoretical implications of our formulation and its alignment with the imitation objective. Further limitations discussed in Appendix F
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Data generation and collection is detailed in Appendices A perparameters and evaluation procedures are detailed Appendices htw.oommm/CCOR-Roooooosssssssas
B E.
Implementation details, hy-
E.2 Code is available at
ACKNOWLEDGMENTS
We thank Manisha Natarajan for feedback the manuscripts. This work was supported by MIT Lincoln Laboratory grant FA8702-15-D-0001, NIH grant 1R01HL-177557-0IA, NSF grant CPS- 2219755, ONR grant -000144-2-1-2834, and a grant from Ford Motor Company.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference Machine learning, pp. 2004.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: wew6:060066666 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. Advances in Neural Information Processing Systems, 30, 2017.
David Barber and Felix Agakov, The IM algorithm: variational approach Information Maxi- mization. Advances in Neural Information Processing Systems, 16(320):201, 2004.
Sumeet Batra, Bryon Tjanaka, Stefanos Nikolaidis, and Gaurav Sukhatme. Quality diversity for robot learning: Limitations and future directions. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 587-590, 2024.
Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me the case for context in reinforcement learning. Transactions on Machine Learning Research, 2023.
Marcel Binz and Dominik M. Endres. Where do human heuristics come from? Arxiv,
abs/1902.07580, 2019. URL https: //api.
ID:
67769766
Gisela Böhm and Hans-Rüdiger Pfister. How people explain their own and others' behavior: theory of lay causal explanations. Frontiers in psychology, 6: 6:109763, 2015.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arxiv. arXiv preprini arXiv: 1606.00400 10, 2016.
Zhangjie Cao, Yilun Hao, Mengxi Li, and Dorsa Sadigh. Learning feasibility to imitate demonstra- tors with different dynamics. arXiv preprint arXiv:21 2021.
Letian Chen, Rohan Paleja, Muyleng Ghuy, and Matthew Gombolay. Joint goal and strategy infer- across heterogeneous demonstrators via reward network distillation. In Proceedings of the 2020 ACM/IEEE International Conference on human-rokee robott interaction, pp. 659-668, 2020.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pp. 1262-1277. PMLR, 2021.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel. and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.00 04317, 2023.
11

---

Published as a conference paper at ICLR 2025
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment as representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01404, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman, Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1282 1289. PMLR, 2019.
Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey, Artificial intelligence in medicine, 109: 101964, 2020.
Wojeiech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference on artifi- cial intelligence and statistics, pp. 1331-1340. PMLR, 2019.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630, 2020.
Anca Dragan and Siddhartha Srinivasa. Generating legible motion. Frontiers in psychology, 2013.
Danny Driess, Jung-Su Ha, and Marc Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. arXiv preprint arXiv:2006.053396 2020.
Yan Duan, Marcin Andrychowicz. Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. Advances in Neural Information Processing Systems, 30, 2017.
Marco Ewerton, Guilherme Maeda, Gerrit Kollegger, Josef Wiemeyer, and Jan Peters. Incremental imitation learning of context-dependent motor skills. In IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp. 351-358. IEEE, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- tions, 2018
Chelsea Finn, Paul Christiano, Pieter Abbecl, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint a6Xiv: 103852, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on machine learning, pp. 1126-1135. PMLR, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning In Conference on Robot Learning, pp. 357-368, PMLR, 2017b.
Pete Florence, Corey Lynch, Andy Zeng. Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learning Representations, 2018.
Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. Eliciting compatible demonstrations for hhmatn imitation learning. In Conference on Robot Learning, pp. 1981 1991. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Yapeng Gao, Jonas Tebbe, and Andreas Zell. Optimal stroke learning with policy gradient approach for robotic table tennis. Applied Intelligence, (1::13339-11322211 2023.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277, PMLR, 2020.
Diego Gomez, Michael Bowling, and Marlos C Machado. Proper laplacian representation learning. arXiv preprini arXiv:2310.10833 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- tion Processing Systems, 27, 2014.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1rXvv:161..07507, 2016.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances in Neural Information Processing Systems, 30, 2017.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 1:x.:::906.050300. 2019.
Mahta HassanPour Zonoozi and Vahid Seydi. A survey adversarial domain adaptation. Neural Processing Letters, 55(3):2429-2499, 2023.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. Advances in Neural Information Processing Systems, 30, 2017.
Alexander Herzog, Kanishka Rao, Karol Hausman, Yao Lu, Paul Wohlhart, Mengyuan Yan, Jessica Lin, Montserrat Gonzalez Arenas, Ted Xiao, Daniel Kappler, et al. Deep rl at scale: Sorting waste in office buildings with a fleet of mobile manipulators. arXiv preprint arXiv:2305.032700 2023.
Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. Advances in Neural Information Processing Systems, 29, 2016.
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, (155:359-366, 1989.
Maxence Hussonnois, Thommen George Karimpanal, and Santu Rana. Controlled diversity with preference: Towards learning a diverse set of desired skills. arXiv preprint arXiv:2303.04559, 2023.
Sagar Imambi, Kolla Bhanu Prakash, and GR Kanaguchidambaresan. Pytorch. Programming with TensorFlow: Solution for Edge Computing Applications, pp. 87-104, 2021.
Rohit Jena, Changliu Liu, and Katia Sycara. Augmenting gail with be for sample efficient imitation learning. In Conference Robot Learning, pp. 80-90. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Xiaogang Jia, Denis Blessing, Xinkai Jiang, Moritz Reuss, Atalay Donat, Rudolf Lioutikov, and Gerhard Neumann. Towards diverse behaviors: A benchmark for imitation learning with human demonstrations. arXiv preprint arXiv: 14606, 2024.
Liyiming Ke, Sanjiban Choudbury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning f-divergence minimization. In Algorithmic Foundations of Robotics X/V: Proceedings of the 14th Workshop on the Algorithmic Foundations of Robotics 14, pp- 313-329. Springer, 2021.
Diederik P Kingma and Max Welling. 1311v:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot gener- alisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76:201-206 264,, 2023.
W Bradley Knox and Peter Stone. Tamer: Training agent manually via evaluative reinforcement. In 7th IEEE International Conference on development and learning, pp. 292-297. IEEE, 2008.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings the 2009.
the 5th International Conference on Knowledge Capture, pp. 9-16,
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminatorractor-rriii Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems, 2021.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198 11881988210, 2020.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint 10.15121 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint 1rXiv:2202.00161, 2022.
Edouard Leurent. An environment for autonomous driving decision-making. https: com/ e eleurent//ighwayy env 2018
Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, and Georg Martius. Ver- satile skill control via self-supervised adversarial imitation unlabeled mixed motions. 2023 IEEE international conference robotics and automation (ICRA), pp.
IEEE, 2023.
Chenhao Li, Elijah Stanger-Jones, Steve Heim, and Sangbae Kim. Fid: Fourier latent dynamics for structured motion representation and learning. arXiv preprint arXiv:2402.138800 2024.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. Pro- ceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pp. 809-8177 2021.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference Robot Learning, pp. 1678-1690. PMLR, 2022.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379-2435 24391, 2021.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint / 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 5 518(7540):529-53) 2015.
Eduardo F Morales and Claude Sammut. Learning to fly by combining reinforcement learning with behavioural cloning. In Proceedings of the 21st International Conference on Machine learning. pp. 76, 2004.
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learming, pp. 1303-1315. PMLR, 2022.
Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference Robot Learning, pp. 529- 551. PMLR, 2021.
OpenAl. ol. 2024. Learning reason with Large Language Models.
retooo-rrssostiee with 1etch-1mmm 11ms/
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances Neural Information Processing Systems, 34:14666- 14668, 2021.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state snte-acttionbasee mutual information. Neural Networks, 152:90-104, 2022.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:/8100 22822, 2018.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint 10.08864, 2023.
Rohan Paleja, Andrew Silva, Letian Chen, and Matthew Gombolay. Interpretable and personalized apprenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations. Advances in Neural Information Processing Systems, 33:6417-6428,, 2020.
Scohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- fations, 2022.
15

---

Published as a conference paper at ICLR 2025
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability unsupervised skill discovery. In International Conference on Machine Learning, pp. 27225-27245. PMLR, 2023.
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable unsupervised RL with metric- aware abstraction. In The Twelfth International Conference Learning Representations, 2024.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Repre- sentations (ICLR 2023), 2023.
Jian-Wer Peng, Min-Chun Hu, and Wei-Ta Chu. An imitation learning framework for generat- ing multi-modal trajectories from unstructured demonstrations. Neurocomputing, 500:712-723, 2022a.
Xue Bin Peng. Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), (17) 2022b.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.004644 2018.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeaap7885, 2018.
Yiwen Qiu, Jialong Wu, Zhangjie Cao, and Mingsheng Long. Out-of-dynamics imitation learning from multimodal demonstrations. Conference on Robot Learning, pp. 1071-1080. PMLR, 2023.
Deepak Ramachandran and Eyal Amir Bayesian inverse reinforcement learning. In IJCAI, vol- ume 7, pp. 2586-2591, 2007.
Siddharth Reddy, Anea D Dragan, and Sergey Levine. Sqil:
Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
Allen Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint urXiv:2409.0055 2024.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth International Con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
arXiv preprint Sebastian Ruder. An overview of gradient descent optimization algorithms. 17X1v:1609.04747, 2016.
Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
Penelope M Sanderson. The human planning and scheduling role in advanced manufacturing sys- tems: An emerging human factors domain. Human Factors, 31(6))635-666 666, 1989.
Mariah L Schrum, Erin Hedlund-Bool Botti, and Matthew Gombolay. Reciprocal MIND MELD: Improv- ing learning from demonstration via personalized, reciprocal teaching. Conference on Robot Learning, pp. 956-966. PML.R, 2023a.
Mariah Schrum, Emily Sumner, Matthew C Gombolay, and Andrew Best. Maveric: A data-driven approach personalized autonomous driving. arXiv preprint arXiv:2301.0850 2023b.
16

---

Published as a conference paper at ICLR 2025
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 05f1::15060024388, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with stone. Advances in Neural Information Processing Systems, 35:22955 -29668, 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awaer awwae unsupervised discovery of skills. In International Conference Learning Representations, 2020.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan, and Matthew Gom- bolay. Lancon-leam: Learning with language enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 7(2):1635-166) 2021.
David Silver, Satinder Singh, Doina Precup, and Richard Sutton. Reward enough. Artificial Intelligence, 299: 2021.
Özgür Şimşek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the 21 International Conference on Machine learning, pp. 95, 2004.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-311, 2003.
Nate Soares and Benja Fallenstein. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report, 8, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Andrew Szot, Amy Zhang, Dhruv Batra, Zsolt Kira, and Franziska Meier. Bc-irl: Learning general- izable reward functions from demonstrations. arXiv preprint arXiv:2303.16114 2023.
Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. In The 11th International Conference on Learning Representations, 2022.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-workle successes. arXiv preprint arXiv:2408.0355. 2024.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation the 37th International Conference learning with diverse-quality demonstrations. In Proceedings the on Machine Learning, pp. 9407-9417 2020.
Yee Tch, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-9, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
17

---

Published as a conference paper at ICLR 2025
Chen Wang, Claudia Pérez-D' Arpino. Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co- gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learn- ing. pp. 1279-1290, PMLR, 2022.
Yawei Wang and Xiu Li. Reward function shape exploration in adversarial imitation learning: an empirical study. In 2021 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), pp. 52-57. IEEE, 2021.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. Advances in Neural Information Processing Systems, 30, 2017.
Pawel Wawrzyúski. A cat-like robot real-time learning to run. In Adaptive and Natural Computing Algorithms: 9th International Conference, Kuopio, Finland, Revised Selected Papers 9, pp. 380- 390. Springer, 2009.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representa- tions to influence multi- multi-agent interaction. In Conference on robot learning, pp. 575-588. PMLR, 2021.
Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint wwXiv22377003654 2023.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference on machine learning, pp. 24631-24645. PMLR, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100, PMLR, 2020.
Luyao Yuan, Xiaofeng Gao, Zilong Zheng, Mark Edmonds, Ying Nian Wu, Federico Rossano, Hongjing Lu, Yixin Zhu, and Song-Chun Zhu. In situ bidirectional human-robot value alignment. Science Robotics, 7(68):eabm4183. 2022.
Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13 13521, 2022.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint urXiv:2304.13700 2023.
Zeyu Zhu and Huijing Zhao. A survey of deep rl and il for autonomous driving policy learning. IEEE Transactions on Intelligent Transportation Systems, 23(9):14043-1400 2021.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume pp. 1433-1438. Chicago, IL. USA, 2008
18

---

Published as a conference paper at ICLR 2025
A POINT MAZE
3
2
The PointMaze domain considered in Sec. 4 is presented in Fig. 8 PointMaze is two-dimensional navigation en- vironment with continuous state and action spaces. The state vector represents the agent's current location's s x- and y- coordinates in [-00, oo]². The action space is a veloc- ity command, a two-dimensional vector in [J1, The episode length is fixed to 25 steps. Expert demonstrations are collected from a PD controller parameterized with a Figure 8: The figure visualizes expert one-dimensional (ID) factor, that determines the way- demonstrations in PointMaze with vary- point through which the agent passes (0, w)) on its way to ing waypoints along x = 0.
1
0
2
3-2
0 s(0]
2 3
the goal (2, 0).
B DOMAINS AND DEMONSTRATIONS
The bounded factor range is divided into 5 intervals for each domain, as explained in Sec. 6 For each interval, we add Gaussian noise to the mean value of the interval to generate five samples. We condition the expert policy on the five samples obtain five demonstrations for each interval.
HalfCheetah: Velocity vs Timestep
B. HALFCHEETAH
Velocity (smoothedd mist
first
Second
Third
Fourth
rifth
The HalfCheetah environ- ment considered in Sec. 6 is
from OpenAl Gym Brock- man et alaal]]2001 2016). The obser- vation vector consists of the positions and velocities of the robot joints, along with height and velocities in the vertical and horizontal direc- tions. The reward is modi- fied as shown in Eq. where interval in which they belong. is the reward at the time, t, and 24 the position of the center of mass the robot along the x-axis at time f, and J, the indicator function that outputs I if and only if (iff) its argument evaluates true. The undiscounted episode return counts the number of steps in which the cheetah moves forward by a non-zero amount. The return normalized using the range, [0, 1050). The episode length is fixed at 1000 steps. The environment is stochastic with the robot initialized at random configurations.
Timestop
Figure 9: Left: The images visualizes the HalfCheetah robot. Right: The figure shows the (smoothed) velocity plotted against the timestep of the demonstration. The trajectories are colored indicate the factor
T sstep = [(x+11 - Iz > 0)
(5)
The factor the mean velocity measured as the net change in the x-coordinate over the elapsed time. Due to environment stochasticity, we use five sampled trajectories per conditioning latent vector during evaluation and consider the mean value. Demonstrations consist of the robot running at different mean velocities [1, 2, 3, 4, 5] m/s, collected using RL policies trained using SAC (Haarnoja ct al. 2018) and auxiliary rewards for target velocities.
B.2 DRIVELANESHIFT
The PrivelLnnshinn environment is built from the highway-em library (Leurent 2018). The highway consists of two lanes. The scenario includes the ego-car in the right lane controlled by the agent, and another car in front, the same lane, that maintains a constant speed of 25 m/s. The task of the ego-car is to shift to the left lane, overtake the other car, and reach the target speed of 30 m/s. The reward at each step is as shown in Eq. 6 where r; is the reward at the time, t, bouroads evaluates to true iff the is within the road bounds time f, beafe evaluates to true iff the ego-car has not crashed until time t, biefttane evaluates to true iff the ego-car in the left lane at time t, v, is the speed at time and clip(ær, x. a, b) clips the value to lie between and b. The return is normalized
19

---

Published as a conference paper at ICLR 2025
using the range [0, 175]. The state vector includes positions (absolute for the ego-car, relative for the other), velocities, heading angles, and longitudinal, latitudinal, and angular offsets to the closest lane for both cars. The episode length is fixed at 50 steps. The environment is deterministic.
U/
30
Tstep =1 (boorood)) + (bsafe) + (bteftione) clip(
0,1)
(6)
5
The factor the min headway distance, i.e., the distance between the ego-car and the other, which the ego-car shifts to the left lane before overtaking. Demonstrations consist of the ego-car perform- ing overtaking maneuvers at varying min headway distances [10.92, 18.28, 25.62, 32.91, 40.27] m, collected using scripted PD controller.
DriveLaneshift: Ego-car trajectories (rel. to second car)
offset (m)
Second
Fourth
Fith
-50
-40
-20
10
Headway (m)
The images visualize the highway overtaking scenario with the ego-car (green) starting behind the other car (yellow) in the right lane. Bottom: The figure visualizes the position of the ego-car relative the other car recorded in the demonstrations. The trajectories colored to indicate the factor interval in which they belong.
Figure 10: Top:
FetchPickPlace: Object Trajectories (overhead view)
B.3 FETCHPICKKPACCK
1.1
The FetchPickPlace environment con- sidered from the gym library (Brockman et al. 2016). The task is to move the object from its initial lo- cation the table at [1.20, 0.75] to along the line x = 1.40, with reward at each step measured shown in Eq. where T, is the reward at the time, t. and is the position of the center of mass of the object along the x-axis at time f. The return is normalized using the range [-20, 5J. The state vec- includes the end effector position
y coordinate (m)
0.8
First
Second
Fourth
and velocity, object position and veloc-
0.4
Rifth
ity, finger gripper position and veloc- ity, and object position relative to the gripper. The episode length fixed at 100 steps. The environment is deter- ministic.
1.3
coordinate
The images visualize the Fetch robot with an object on the table. Right: The figure shows the object trajectories (from the top down) recorded in the demonstra- tions. The trajectories are colored to indicate the factor in- (7) terval in which they belong.
Figure 1I: Left:
Sstep -1 xel
The factor is the y-coordinate the final object position. Demonstrations consist of the robot picking the object up from the initial location and placing it at the target x-coordinate and varying y-coordinates, 0.75 + [-0.32, 0.16, 0, 0616, 0.32) m, collected using scripted state-based PD controller.
20

---

Published as a conference paper at ICLR 2025
C ALGORITHM AND IMPLEMENTATION
C.I ALGORITIIM DETAILED VERSION
A detailed version of Algorithm D can be found in Algorithm 2 with objectives and gradient steps for all components explicitly written down.
Algorithm 2 Guided Strategy Discovery
Input: Φ = (T,}
Output: π
1: Initialize policy æ, task relevance f. factor-specific residual 9, decoder q. with parameters 0,.
Oy. Og. 0g. Lagrange multiplier X, bias b, and learning rates M/s Vf Tga Tg T1x, 7l
2: for i {0, .... epoch do
3: Sample 2x from prior. r" using policy (-|,,2 >;); from D, infer 2f using decoder g
4: Define objective for functions f. and bias b:
D(s a, 2) = (As [f(s, a) + g(, a, 2)] +
J - E.e log ((s a, =)] + Era [log(1 a(,,a,
- Ey [[g(s, a, =1) E (g(s, a. a=")s,aa=))))))) 20))
5: Update f. b using gradients: 105,0sm == 00,0,, ,, b] + Ins Vo, . Ty V,, J' 11, Vo.14
6: Define objective for decoder y: 8(s, a, s', a') + Do ||s @ + a'll f(s', a') (1 Mo f(s,a)
82, < (= ()(-(s.))
JE + E." [log qu 8,0)
min(5(s, 0, s' ,a),
7: Update decoder q and À using gradients: 10,, A| == 10 X| + [n Vo -
8: Update policy क with RL using rewards: a(s,a, - log ((s,a, 2)) A1 log qr. (2, 8,
9: end for
C.2 IMPLEMENTATION
We implement our approach on top of the public code-base for VILD Tangkaratt et al. 2020) that implements adversarial IL algorithms using PyTorch (Imambi et al. 2021): giithu..coo com//oot t/vild.code We use implementation tricks from their codebase ensure convergence across meth- ods, such as gradient penalty (Gulrajani et al. (017) with a weight of 0. for the discriminator/task
relevance, and the positive logarithmic function (Wang & Li (021) for discriminator rewards, i.e.,
r(s, a) = - log(1 D(s, a)) instead of r88, a) = log( (D(s))))
We normal prior for the latent space across all approaches. The decoder q outputs the mean and diagonal elements of the covariance matrix of the approximate posterior distribution, which is assumed be Gaussian.
Conditioned Discriminator To infer latent code 75 for a demonstration trajectory 75, we make simplifying assumption that the posterior distributions across transitions are independent. Thus, the product of the individual distributions gives us the posterior distribution for the demonstration trajectory.
We add expert transitions with mismatched demonstration latent vectors as fake samples to the discriminator dataset to ensure that the conditioned discriminator, D(s, a, 2). does not ignore the input latent vector. We upsample "real" data points in a batch to avoid imbalanced classes for discriminator gradient updates.
Decoder Regularization We perform spectral normalization (Miyato et al. 2018) using the Py- p. pa rametri za ions swwcttal__noo rm. We scale the inputs to
Torch function nn un.util ls.. the decoder to implement Lipschitz constraint scaling with Às.
Reinforcement Learning We use PPO (Schulman et al. 2017) for policy learning from rewards. For some domains, we use PPOBC (Jena et al. 2021). PPOBC augments the policy objective with a behavior cloning (BC) loss term which improves learning stability without directly affecting the discriminator decoder. We highlight that using the BC term comes at additional human cost, as demonstrations are already available in the IL setting. Furthermore, we make no assumptions about the demonstrations' behavior factors either and use the decoder network to infer the latent factor.
21

---

Published as a conference paper at ICLR 2025
C.3 DOMAIN-SPECIHIC VARIATIONS AND TUNING
The hyperparameters used in our optimization are listed in [ Each method is independently tuned for x (and Àc for Lipz. GSD) over the specified ranges, to maximize MAE over the test split for K=10 over averaged over four rounds of evaluation and five train seeds. All hyperparameters omitted from the tables are set to default values from our base implementation.
Hyperparameter
Value
NN update minibatch size Policy learning rate Entropy bonus
256
3e-4
0.0001
Gamma
0.99
GAE coefficient (Schulman al. 2015)
0.97
NN architectures
FCN
Policy activation
Tanh
BC warmstart epochs
10000
Disc. activation
Tanh
Disc. learning rate
1e-3
Disc. gradient steps
Dec. hidden dimensions
[100, 256]
Dec. activation
ReLU
Dec. gradient steps
As
10
b initial value
-5
Constraint slack (c) Lambda learning rate Optimizers
1e-6
1e-3
Adam
Table 1: The table contains the list of hyperparameters, common across domains and generalization settings. GAE: Generalized Advantage Estimation, NN: Neural Network, FCN: Fully connected network, BC: Behavior cloning, Disc.: Discriminator, Dec.: Decoder
Hyperparam. Domain
HalfCheetah
DriveLaneshift
FetchPickPlace
Env. steps
1.5e7
0.5e7
1e7
RL algorithm
PPO
PPOBC 0.2
PPOBC
BC halflife, weight
0.1, 0.1
(PPOBC)
NN update interval (steps)
10000 [100, 100]
1000
10000
NN hidden dimensions
[100, 100]
[32, 32]
Observation norm.
False
False
True
(w/ demos.)
Policy weight decay
(0, le-3)
le-4
(Ie-4, 5e-5) le-3 5000 8 [-oo, oo] (0.0001, 0.0005)
Dec. learning rate
le-3
le-4
Lambda initial value
(100, 500)
500
Dec. gradient norm clip
oC [-o0, co] (0.02, 0.001)
25
Dec. rewards clip
[-20, (0.001, 0.001)
Distillation
objective weight
Às sweep list
[0.02, 0.05, 0.1, 0.2) ([0.9, 0.8].
[0.1, 0.5, 1.0. 5.0]
[0.1. 0.5, 1.0. 5.0]
\ sweep list
(I0.99, 0.97].
([0.9, 0.8), [0.8, 0.7))
[0.8, 0.7l)
[0.99, 0.97I)
The table contains the list of hyperparameters, specific to each domain (indicated by the column) and generalization setting (indicated by 2-tuple (left, right), where left and right corre- spond to interpolation and extrapolation respectively).
Table 2:
22

---

Published as a conference paper at ICLR 2025
D COMPARISON AGAINST OFFLINE IL APPROACHES
We compare GSD against offline IL approaches that learn solcly from data without any environment interaction. We use implementations open-sourced by D3IL (Jia et al. 2024). We focus on mul- timodal action distribution modeling by excluding architectures that incorporate state histories or predict action sequences, such as action chunking (Zhao et al. 2023). We use fully connected neural network backbones with two hidden layers each containing 100 units. Unless otherwise specified, we use default hyperparameters that are most common across tasks in D3IL. We use demonstrations corresponding to held-out factor intervals as the validation dataset for early stopping.
Behavior cloning (BC): Policy NN takes state as input and outputs actions, and trained with
mean squared error (MSH) loss.
BC with VAE (BC-VAE): BC-VAE uses state-conditioned encoder-decoder setup to model the action distribution (Sohn et al.
2015). Latent space dimension is set to size 2, same our ap- proach. A weight of 5.0 is used for scaling KL-divergence loss.
Implicit BC (IBC): Florence al. (2022) propose energy models that implicitly capture the action distribution at each state. The action is inferred by optimizing the energy function using Markov chain Monte-carlo sampling each inference step.
K-Means Discretization (Be Shafiullll et al. (2022) propose an approach to capture multimodal action distributions using learned discretization with K predicted action means and offsets. K
is set to 64.
Diffusion Policy (Diffusion): Chi al. 2023): Pearce et al. (2023) propose modeling action distributions with a diffusion model conditioned on the state. Timestep embeddings of size 16 and 24 denoising steps are used.
We show the recovery and performance trade- off of offline IL and approach in Fig. 12
Interpolation
Extrapolation
With regard to task performance (x-axis), for HalfCheetah (top row) and FetchPickPlace (bot- tom), GSD outperforms offline approaches (ex- cept for interpolation in HalfCheetah) indicated by the red cross being positioned further right than others. For DriveLaneShift (middle row). offline approaches other than BC and BC-VAE
HalfCheetah
MAE in Velocity
2.3
(mis)
1.5
0.8
73.8
MAE in Min Headway(m)
are competitive with GSD. The result sug- gests that in domains with complex dynamics like HalfCheetah and FetchPickPlace, environ- ment interaction is necessary for task completion when learning from few demonstrations.
Drive DriveLaneshif
37.5
18.8
With regard to recovery performance (y-axis). for HalfCheetah (top row) and FetchPickPlace (bottom). GSD outperforms all offline ap- proaches, indicated by the red cross being po- sitioned below others. For interpolation in Driv-
98
MAE in Objy
log scale)
14
Higher retum is better
FetchPickPlaa
Lower error is better
ShaneShift (middle row, left), approaches IBC,
2-
BeT and Diffusion are comparable to GSD. For extrapolation (middle row, right), GSD outper-
1
0.0 0.4 0.6
0.8 0.9 1 00.0 0.4 0.6 0.8
0.9 1.0
Normalized return (log scale)
forms all methods. Poor performance of offline approaches in domains with complex dynamics like HalfCheetah and FetchPickPlace may be at- tributed to the absence of environment interac- tion. In simpler domains, IBC, BeТ or Diffu- sion may be able interpolate diverse behaviors. However, for extrapolation, environment interac- tion is necessary, even for simpler domains.
BC BC-VAE
Y IBC
Diffusion
GSO (Ours)
Figure 12: The figure shows the tradeoff between task and recovery performance for three domains and two splits for offline IL approaches and GSD (indicated with red circles). Error bars show stan- dard errors over five seeds.
23

---

Published as a conference paper at ICLR 2025
E EVALUATION WITH HUMAN DEMONSTRATIONS IN TABLE TENNIS
E SETUP
The WAM arm is enabled with gravity compensation for collecting kinesthetic demonstrations. Messages published to ROS interface are collected for two seconds (starting after the ball is detected to move over the table) which is enough time to capture the return trajectory. Joint states (R") and ball positions (R²) are matched over time, and concatenated to construct state vectors (R.1). Action vectors constructed by calculating the displacements corresponding timesteps. We collect five demonstrations for each of the two-stroke types considered.
Demonstrations:
Simulation: We position control for the WAM arm 100hz with the control gains tuned to match real robot demonstrations visually when replaying action commands open loop. We tune ball flight parameters such that the ball flight paths (before being struck) visually match those from the real demonstrations when launched from similar position, velocity, and noise as the ball launcher. We add Gaussian noise the ball positions in the observation vector to mimic real recorded ball positions. We use an episode length of 200 steps that corresponds to real-life execution period of two seconds.
E.2 METRICS
We detect if the ball has been returned by checking if the velocity component along the long edge of the TT table has reversed. Once a returning ball is detected, we check if the ball remains above the table plane and within 10 beyond the sides of the table for the following 0.5 seconds. If the trajectory satisfies both criteria, we deem it a success. We calculate the factor value for each successful return which is the maximum height the ball reaches in the return trajectory.
We sample five trajectories per sampled latent vector due to the stochastic nature of the ball observa- tions. However, not every sampled trajectory for a particular latent vector is guaranteed to succeed due to the stochasticity in the domain and optimization. Thus, we consider a latent vector successful if the ball is returned in at least three out of five trials. We consider the factor value for the successful latent vector to be the mean of the values of the successful trajectories.
For each method and train seed, we sample 200 latent vectors from the prior, and sample five trajec- tories per latent vector. We report the fraction of successful latent vectors to evaluate if behaviors can accomplish the underlying task. Among the set of latent vectors, we subsample 100 successful la- tent vectors (after ensuring each method has at least 50% success rate) and report the entropy (based on particle estimates (Singh et al. 2003)) among the calculated factors using the equation shown in Eq. 8 where V {2 is the set of factor values U/ MM 100, K 50 and NNe v (V:) returns the Kthh nearest neighbor to i from the set of values V. The entropy measure is up to a proportionality constant, as use it to compare diversity achieved in return ball trajectory heights across methods.
1 M
Hx(V)= = M log l|: - NNck,v
(8)
F FURTHER LIMITATIONS AND FUTURE WORK
Our approach requires task-relevance measure, f. which we derived from demonstration occupan- cies. IL approaches that do not explicitly model expert occupancy (Reddy et al. 2020; Garg et al.) 2021 may not be readily compatible for integration with our regularization. However, Q functions learned during policy optimization may act as suitable alternatives for f.
Our regularization is implemented through approximately enforced constraints using Lagrange mul- tipliers. Approximate enforcement may permit spurious behaviors that drastically vary from demon- strations. Parametric approaches akin to spectral normalization for Lipschitz continuity (Miyato)
al. 2018 desirable.
Instead of using diversity objectives, approaches that learn generalizable reward functions (Zzo al. 2023) could also be explored in the context of generalizable heterogeneous II...
24

---

Published as a conference paper at ICLR 2025
The efficacy of our regularization could be explored with offline and non-MI-based diversity frame- works that use latent space modeling.
G GENERALITY OF f-RELEVANT DIVERSITY
Our /-relevant diversity for- mulation discussed in Sec. 5.1 designed to encourage be- havior diversity with respect to any defined energy measure f. We briefly demonstrate the
0.0
0.5
1.0
a) Example f fn.
No regularization
Lipschitz constraints
f-relevant diversity (Ours)
generality of our formulation
in the simple 2D PointMaze domain with a user-defined energy function as shown in Fig. 13 Our formulation has potential application in di- verse solution discovery (Ku- mar et al., 2020 Osa et al., 2022). where bounded form of the estimated Q function can be used as f to encourage diversity in high value regions.
Figure 13: The figure visualizes behaviors in 2D PointMaze learned with a predefined energy function f shown in 13p. f used to specify rewards in 13p-d and additionally formulate our diversity objective 13d. With no regularization (13p) or Lipschitz con-
straints (13 trajectories visit low f-energy regions of the state- space. However, with f-relevant diversity (ours, [3d), a larger por- tion of trajectories cover diverse high energy states.
H EVALUATION RESULTS
provide numerical figures for recovery errors used to plot the graphs in Fig. 4 below. We further abbreviate Con, ConDist and Lipz CO, CD and LZ respectively due to width constraints.
HalfCheetah: Interpolation, Average:
Model
K 10
K 20
K = 30
K = 40
K 50
IG
0,258 + 0.007
0.147 = 0.006
0.106 + 0.004
0.080 = 0.004
0.063 = 0.004
IG+LZ
0.265 + 0.009
0.156 - 0.010
0,089 = 0.008
0.069 0.006
IG+CO
0.229 + 0.015
0.114+0.000
0.004
0.058 + 0.003
0.049 = 0.003
IG+CD
0.266 + 0.014
0.142 + 0.007
0.101
0.077 + 0.005
0.060 0.005
IG+CD+LZ
0.307 + 0.004
0.177 + 0.005
0.124 0.004
0.091 + 0.003
0.077 = 0.003
GSD (Ours)
0.226 + 0.007
0.120 0.003
0.081 0.003
0.065 0.001
0.052 0.002
HalfCheetah:
Interpolation, Worst:
Model
K = 10
K 20
K 30
K = 40
K 50
IG
0,298 + 0.004
0.177 + 0.002
0.134 0.003
0.103 0.003
0.076 = 0.003
IG+LZ
0.348 = 0.020
0.212 +
0.159
0.131
0.099 = 0.013
IG+CO
0.029
0.014
0,007
0.072
0.061 0.005
IG+CD
0.339 +
0.181 1 0.013
0.133
0.082 0.008
IG+CD+LZ
0.373 + 0.007
0.223 + 0.010
0.158 0.008
0.117 + 0.006
0.100 0.006
GSD (Ours)
0.287 + 0.007
0.151 + 0.005
4..004
0.084 0.002
0.069 = 0.002
HalfChectah:
Extrapolation, Average:
Model
10
K 20
== 30
K = 40
K 50
IG
0.841 + 0.010
0.766 + 0.014
0.737
0.710 + 0.017
0.697 0.019
IG+LZ
0.891 = 0.004
0.829 + 0.005
0.788 + 0.005
0.779 = 0.005
IG+CO
0.741 + 0.007
0.618 1618 0.002
0.000
0.525 0.006
0.505 0.007
IG+CD
0.801 + 0.024
46622+0.00
0.581
0.538 0.019
0.513 = 0.020
IG+CD+LZ
0.686 + 0.008
0.597 0.008
0.559 0.010
0.525 0.011
GSD (Ours)
0.682 + 0.028
0.556 + 0.032
33120003
148440.000
0.472 = 0.034
25

---

Published as a conference paper at ICLR 2025
HalfCheetah: Extrapolation, Worst:
Model
10
K 20
30
K 40
K 50
IG
0.910 = 0.013
0.854 0.016
0.837 0.01
0.828 = 0.017
0.821 0.017
IG+LZ
0,999 + 0.004
0.909 + 0.007
0.009
0.869 = 0.009
0.860 = 0.010
IG+CO
0.875 0.019
0.685 0.027
0.661 0.030
IG+CD
0.963 + 0.031
0.750+00011
0.686 0.012
0.656 0.008
0.634 0.007
IG+CD+LZ
0.797 + 0.012
0.719 + 0.011
0.699 0.011
0.689
0.011
0.682
0.011
GSD (Ours)
0.808 + 0.039
0.679 + 0.038
0.646 0.040
0.632 0.041
0.621 = 0.041
DriveLaneshift:
Interpolation, Average:
Model
= 10
K 20
=0=00 30
K 40
K = 50
IG
3,435 + 0.148
2.305 + 0.132
1.808 0.122
1.489 + 0.105
1.277 = 0.098
IG+LZ
3.193 + 0.096
2.101 0.097
1.631 0.100
1.345 0.078
1.111 = 0.073
IG+CO
5.246
2.377 0.333
1.936
1.712 = 0.278
IG+CD
3.619 + 0.200
2.246 + 0.151
1.749 0.154
1.419 0.163
1.221 = 0.154
IG+CD+I.Z
2.785 + 0.196
1.652 0.172
1.302 0.156
1.114
0.985 0.128
GSD (Ours)
2.343 + 0.134
1.299 + 0.094
0.877 0.061
0.685 0.058
0.530 0.045
DriveLancshift:
Interpolation, Worst:
Model
K = 10
K = 20
K = 30
K 40
K 50
IG
4.528 + 0.262
2:944:0.22
2.693 0.228
2.280 0.193
2.008 = 0.190
IG+LZ
4.702 + 0.179
5
2.675 0.192
2.253
1.859 0.140
IG+CO
6.843 0.690
+
3.329 0.441
2.397
2.469 = 0.380
IG+CD
5.057 + 0.307
3.329
0.249
2.657 + 0.269
+
1.878 0.271
IG+CD+LZ
3.895 + 0.314
2.581 + 0.310
2.079 + 0.291
1.797 + 0.269
1.637 = 0.241
GSD (Ours)
3.009 + 0.246
1.709 = 0.171
1.138 1 0.104
0.911 0.106
0.687 = 0.079
DriveLaneshift:
Extrapolation, Average:
Model
K = 10
K 20
K = 30
K = 40
K 50
IG
5,807 + 0.231
4.889 + 0.269
4,446 0.288
1.2299
3,943 = 0.302
IG+LZ
6.435 + 0.124
5.692 0.193
5.375 0.224
5.134
4.977 = 0.257
IG+CO
6.347 + 0.769
4.335 =
3.588 0.566
0.542
2.967 = 0.521
IG+CD
3.902 0.185
2.480 0.185
1.886 0.172
0.166
1.399 = 0.168
IG+CD+LZ
5.588 + 0.511
3,809 + 0.400
2.902 0.369
2.472 + 0.362
2.206 0.369
GSD (Ours)
2.803 + 0.108
1.545 0.074
1.019 + 0.053
0.815 0.046
0.695 0.048
DriveLaneshift:
Extrapolation, Worst:
Model
10
K 20
30
= 40
K 50
IG
6.386 + 0.219
5.527 + 0.246
5.102 0.268
4.805 0.289
4.624 0.291
IG+LZ
6.908 + 0.125
6.222 + 0.224
6.071 + 0.244
5.923 0.259
5.836 = 0.269
IG+CO
8.599 + 1.302
6.673 1.140
5.827 1.107
5.324 1.069
5.033 1.026
IG+CD
5.326 + 0.338
3.606 0.341
2.907 0.339
2.438 0.322
2.206 = 0.328
IG+CD+LZ
8.723 + 1.105
6.473 0.890
5.038 0.790
4.340 0.766
3.945 = 0.766
GSD (Ours)
4.067 + 0.225
2.283 + 0.156
1.534 + 0.105
1.238 + 0.091
1.034 = 0.094
FetchPickPlace:
Interpolation, Average:
Model
= 10
K 20
=
K 40
K 50
IG
0.071 + 0.004
0.045 0.004
0.034 0.003
0.027 + 0.003
0.022 = 0.002
IG+LZ
0.044 + 0.002
0,024 0.001
0.016
0.012 1112:0000 0.001 0.001
0.010 = 0.001
IG+CO
0.072 = 0.009
0.053
1 0.010
0.046
0.040 0.011
IG+CD
0.083 + 0.011
0.067 0.012
0.000
0.055
0.052
0.012
IG+CD+LZ
0.050 + 0.001
0.027
1 0.001
0.019
1 0.155:0000
0.012 = 0.001
GSD (Ours)
0.037 + 0.001
0.020 + 0.001
0.000
0.011 0.000
0.008 0.000
26

---

Published as a conference paper at ICLR 2025
FetchPickPlace: Interpolation, Worst:
Model
10
K 20
30
K 40
K 50
IG
0.084 + 0.005
0.058 + 0.005
0.046 + 0.005
0.037 = 0.004
0.031 0.004
IG+LZ
0.056 + 0.003
0.032 = 0.002
0.001
0.016 = 0.001
0.013 = 0.001
IG+CO
0.081 + 0.009
0.061 0.011
0.011
0.048 0.011
0.047 0.011
IG+CD
0.088 + 0.010
+
0
0.059 0
0.056 0.012
IG+CD+LZ
0.053 + 0.002
0.030 0.001
0.021 0.000
0.017 0.001
0.014 0.001
GSD (Ours)
0.042 + 0.001
0,022 + 0.001
0.015 0.001
0.013 0.001
0.009 = 0.000
FetchPickPlace:
Extrapolation, Average:
Model
= 10
K 20
=
K 40
K = 50
IG
0.137 + 0.003
0.103 0.003
0.087 0.003
0.075 0.003
0.068 = 0.003
IG+LZ
0.083 + 0.003
0.057 = 0.003
0.045
0.038 = 0.003
0.034 = 0.003
IG+CO
0
0.167
0.158 0.018
0.152 = 0.018
IG+CD
0.264 + 0.012
0.233 0.014
0.225 + 0.014
0.220 0.015
IG+CD+I.Z
0.078 + 0.003
0.046 0.002
0.032 0.002
0.026 0.002
0.022 0.002
GSD (Ours)
0.068 + 0.002
0.037 + 0.002
0.027 0.002
0.021 = 0.002
0.018 0.002
FetchPickPlace:
Extrapolation, Worst:
Model
K = 10
K 20
K = 30
K = 40
K 50
IG
0.182 + 0.004
0.143 + 0.004
0.120 0.003
0.105 1 0.003
0.097 = 0.003
IG+LZ
0.097 = 0.003
0.071 0.004
0.057 + 0.004
0.049 = 0.004
0.044 0.004
IG+CO
0.249 + 0.012
0
0.208 0.017
0.017
0.191 = 0.018
IG+CD
0.303 + 0.003
0.286 +: 0.006
0.275 + 0.008
0.266 + 0.009
0.260 0.010
IG+CD+LZ
0.092 + 0.003
0.057 0.003
0.042 + 0.004
0.035 + 0.004
0.031 = 0.004
GSD (Ours)
0.094 + 0.005
0.056 + 0.005
0.041 + 0.004
0.033 0.003
0.028 0.004
27


Reviews:
[{"review_id": "GJEyXqe8Ju", "reviewer": "Reviewer_kDeW", "summary": "This paper proposes a novel diversity formulation based on a learned task-relevance measure that prioritizes behaviors exploring modeled latent factors, called Guided Strategy Discovery (GSD).\nThe proposed method is empirically validated across three continuous control benchmarks for generalizing to in-distribution (interpolation) and out-of-distribution (extrapolation) preferences, as well as demonstrated to generalize striking behaviors for table tennis in a virtual testbed while leveraging human demonstrations collected in the real world.", "questions": "What are the limiting factors of the chosen regularization in the latent space? \nHow much is the representation limited by demonstrations? Does this approach suffer from problems similar to behavior cloning?\nAre there more intuitive visualisations for your results, eg for interpolation/extrapolation?\nIs exploration limited and does this have an effect on final performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is written in a clear way and is well structured. The proposed method is formulated clearly and scoped within relevant literature. The proposed method is also evaluated experimentally, including comparisons with alternatives. Figures and visualisations are clear and help capture the contribution of the method.", "weaknesses": "Some results are not immediately clear from the plots, for example the visualisation of the interpolation/extrapolation experiments in Figure 5 is difficult to read.\nIt is unclear if the proposed method can scale up in terms, for example, of dimensions of preferences, tasks, rewards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNL1qG2H9R", "reviewer": "Reviewer_LHbJ", "summary": "The authors address the challenge of generating diverse behaviors that are useful for a wide range of tasks. They introduce a novel approach called *Guided Strategy Discovery (GSD)*, which features a diversity formulation that uses a learned task-relevance measure to prioritize behaviors that effectively explore modeled latent factors. The proposed method is  evaluated across multiple simulated tasks and in a task with real-world data.", "questions": "In Appendix C.2, the authors state that they used PPOBC for the DriveLaneshift and FetchPickPlace tasks. However, it is important to note that PPOBC is not a pure reinforcement learning (RL) algorithm, as it incorporates demonstrations. How would the proposed method differ if it were implemented using only PPO without the influence of demonstrations?\n\nAdditionally, could the authors clarify what is meant by diverse behaviors in the context of the HalfCheetah task? Specifically, how are different speeds relevant for tasks in the sense of improved task performance?\n\nFinally, will the code be made publicly available? Furthermore, will the dataset of real-world human demonstrations also be accessible to the research community?\n\n[1]Rohit Jena, Changliu Liu, and Katia Sycara. Augmenting gail with bc for sample efficient imitation learning. In Conference on Robot Learning, pp. 80–90. PMLR, 2021", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses a significant and relevant problem , contributing to our understanding of behavior diversity in various tasks. The use of real-world human data for evaluation is praiseworthy, as it enhances the applicability findings. Additionally, the paper features well-designed and illustrative figures that support the discussed concepts .", "weaknesses": "The paper builds upon InfoGAIL, an adversarial imitation learning method; however, it does not explore other adversarial imitation learning (AIL) methods or potential modifications that could enhance performance. A more comprehensive overview and comparison of relevant AIL techniques, such as those discussed in Orsini et al [1], would strengthen the contextualization of the proposed approach and highlight its novelty by discussing how different design choices influence learning.\n\nFurthermore, the tasks considered in the paper do not align with established benchmarks. For example, recent work has introduced benchmarks for diverse behaviors, as highlighted by Jia et al[2]. Additionally, the paper lacks comparisons to other methods, including diffusion-based approaches that can effectively represent diverse, multimodal data.\n\nThe writing quality of the paper could be improved for clarity and conciseness. The term \"preference\" does not align with its common usage in related fields such as Reinforcement Learning from Human Feedback (RLHF) [3], which may lead to confusion among readers.\n\nIn terms of presentation, the caption for Figure 10 appears integrated into the surrounding text, making it difficult to read. Additionally, Algorithm 1 (Lines 270-287) is quite dense and may overwhelm readers. The notation used in Algorithm 1, Line 274, could also be confusing due to the presence of commas following terms (e.g., policy, $\\pi$).\n\n[1]Orsini, M., Raichuk, A., Hussenot, L., Vincent, D., Dadashi, R., Girgin, S., Geist, M., Bachem, O., Pietquin, O., & Andrychowicz, M. (2021). What Matters for Adversarial Imitation Learning? In Advances in Neural Information Processing Systems \n\n[2]Jia, X., Blessing, D., Jiang, X., Reuss, M., Donat, A., Lioutikov, R., & Neumann, G. (2024). *Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations*. In *Proceedings of the Twelfth International Conference on Learning Representations (ICLR)* \n\n[3]Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). *Deep Reinforcement Learning from Human Preferences*. In *Advances in Neural Information Processing Systems*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Im4wI2KpMF", "reviewer": "Reviewer_F4MZ", "summary": "This paper introduces Guided Strategy Discovery (GSD), which provides a new diversity formulation that is able to learn diverse, task-accomplishing behaviors from demonstrations. Through a 2d Point Maze task, the authors demonstrate that prior works fail to generalize to new behaviors. GSD introduces a new constraint $f$-relevant diversity to assign high energy to new state-action pairs. A learned task-relevance measure is used to guide the meaningful/task-relevant behavior patterns. The empirical experiments show that GSD outperforms other baselines in novel behavior discovery.", "questions": "Overall, this paper provides the potential to learn a diverse range of novel behaviors while maintaining the task success rate. However, scaling to tasks with complex behaviors needs to be further evaluated. I have the following questions:\n\n- In the experiments, the 1-d latent factors generally refer to position, velocity, or ball height. How would the authors expect to solve more complex behaviors in robot manipulation tasks? D3IL [1] provides multiple tasks with diverse behaviors. Taking the pushing task for example, could GSD be able to address this task when interacting with objects frequently?\n- Could the diversity objectives be used in general behavior cloning?\n- Regarding the energy function $f$, how to the authors model it? Maybe I missed something, but how do we get the desired energy function $f(s,a)$\n\n[1] Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations, ICLR’24", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors design a 2D toy task PointMaze, which demonstrates the existing diversity formulations, InfoGAIL, and Lipschitz constraints, fail to generate new, task-accomplishing behaviors.\n- The authors provide an insightful motivation for the task-relevant diversity.\n- The effectiveness of GSD is demonstrated on 3 simulation tasks and 1 real-to-sim Table Tennis task (The demonstrations are collected using a real-world setup).", "weaknesses": "- The experiments only contain tasks with 1-d latent factors, which restricts the applications of GSD. More complex task evaluations would greatly improve the quality of the paper.\n- The experiments only consider the InfoGAIL setting. Extensions to other general imitation learning frameworks would be appreciated.\n- The implementation seems complex to me. It would be great to provide the code for better understanding and reproduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1J7FpDPEqf", "reviewer": "Reviewer_4BCW", "summary": "This paper presents a novel framework called Guided Strategy Discovery (GSD) for learning diverse behaviors from demonstrations, aiming to generalize these behaviors to unseen task variations. The authors introduce a task-relevance diversity objective that encourages behavior exploration of latent factors in demonstrations, outperforming baseline methods in continuous control benchmarks like robot control and virtual table tennis.", "questions": "- Some existing works aim to learn diverse task-accomplishing behaviors from demonstrations that generalize over latent preferences using mutual information-based methods [1, 2] and latent space structure design [3]. It may help to discuss these existing works to motivate the need for the proposed method. How to convince people of a need of GSD given these prior works targeting the same goal?\n- The constraint shown in Eq.1 assumes linearity of the state-action space. The proposed method may fail to scale to settings where adjacent state-action pairs have significantly different energy. How generally does this assumption apply to different tasks and state action space design? \n- As pointed out in the limitation section, the experiments are constrained to one-dimensional latent factors, limiting the evaluation of the method's scalability to more complex, higher-dimensional spaces. Please analyze if and how GSD may potentially fail when applied to higher-dimensional latent space. And if not, why not evaluate such environments?\n\n\n[1] Li, C., Blaes, S., Kolev, P., Vlastelica, M., Frey, J. and Martius, G., 2023, May. Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In 2023 IEEE international conference on robotics and automation (ICRA) (pp. 2944-2950). IEEE.\n\n[2] Peng, X.B., Guo, Y., Halper, L., Levine, S. and Fidler, S., 2022. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4), pp.1-17.\n\n[3] Li, C., Stanger-Jones, E., Heim, S. and Kim, S., 2024. FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning. arXiv preprint arXiv:2402.13820.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The work introduces a new approach for behavior learning that focuses on task-relevant diversity. The proposed GSD framework creatively balances imitation with exploration of unseen behavior spaces.\n- The empirical evaluation is robust, spanning multiple domains (robot control, driving, and manipulation) and showing consistent improvement over baselines, especially in behavior generalization. The metrics used, such as interpolation and extrapolation errors, are well-chosen to highlight the effectiveness of GSD in diverse and novel task variations.\n- The paper is well-structured, with clear explanations of the problem, related work, and contributions. Figures and diagrams effectively communicate the GSD framework and its comparative performance.", "weaknesses": "- The task-relevance diversity formulation, while promising, may still need refinement for more complex behaviors in physical systems, particularly where task success is less clear or difficult to define in terms of energy regions.\n- The writing is mostly clear, but some sections on the technical implementation of the diversity objective could benefit from further simplification for a broader audience.\n- The state-action space assumes a continuously varying task relevance, and the experiments are primarily performed in environments with this feature. The proposed method may fail to scale to settings where adjacent state-action pairs have significantly different energy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)