Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models
Authors: Andrea Tirinzoni
Ahmed Touati
Jesse Farebrother
Mateusz Guzek
Anssi Kanervisto
Yingchen Xu
Alessandro Lazaric
Matteo Pirotta
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning; foundation model; humanoid
Primary area: reinforcement learning
TL;DR: We present a novel unsupervised reinforcement learning algorithm that leverages unlabeled behavior data to create a foundation model for controlling a humanoid agent in a zero-shot fashion.

Abstract: Unsupervised reinforcement learning (RL) aims at pre-training models that can solve a wide range of downstream tasks in complex environments. Despite recent advancements, existing approaches suffer from several limitations: they may require running an RL process on each task to achieve a satisfactory performance, they may need access to datasets with good coverage or well-curated task-specific samples, or they may pre-train policies with unsupervised losses that are poorly correlated with the downstream tasks of interest. In this paper, we introduce FB-CPR, which regularizes unsupervised zero-shot RL based on the forward-backward (FB) method towards imitating trajectories from unlabeled behaviors. The resulting models learn useful policies imitating  the behaviors in the dataset, while retaining zero-shot generalization capabilities. We demonstrate the effectiveness of FB-CPR in a challenging humanoid control problem. Training FB-CPR online with observation-only motion capture datasets, we obtain the first humanoid behavioral foundation model that can be prompted to solve a variety of whole-body tasks, including motion tracking, goal reaching, and reward optimization. The resulting model is capable of expressing human-like behaviors and it achieves competitive performance with task-specific methods while outperforming state-of-the-art unsupervised RL and model-based baselines.

Published as a conference paper at ICLR 2025
ZERO-SHOT WHOLE-BODY HUMANOID CONTROL VIA BEHAVIORAL FOUNDATION MODELS
Andrea Tirinzoni Ahmed Touati' Jesse Farebrother 2,1 MMateusz Guzek', Anssi Kanervisto Yingchen Xu Alessandro L & Matteo Pirotta
Fundamental AI Research at Meta, 2 Mila, McGill University, 3 UCL {t (tirinzoni, atouati, mat matg,zek, anssik, yexu, lazaric,
Motion tracking
Reward optimizaton
Pase reschng
(r up weint_ 2) "teight
Figure 1: With FB-CPR we pre-train the first behavioral foundation model for humanoid that can solve tracking, goal-reaching, and reward optimization tasks in zero-shot fashion.
ABSTRACT
Unsupervised reinforcement learning (RL) aims pre-training models that can solve wide range of downstream tasks in complex environments. Despite recent advancements, existing approaches suffer from several limitations: they may re- quire running an RL process on each task to achieve satisfactory performance, they may need access to datasets with good coverage or well-curated task-speeific samples, they may pre-train policies with unsupervised losses that poorly correlated with the downstream tasks of interest. In this paper, we introduce FB-CPR, which regularizes unsupervised zero-shot RL based the forward- backward (FB) method towards imitating trajectories from unlabeled behaviors. The resulting models learn useful policies imitating the behaviors in the dataset, while retaining zero-shot generalization capabilities. We demonstrate the effec- tiveness of FB-CPR in challenging humanoid control problem. Training FB- CPR online with observation-onle motion capture datasets, we obtain the first humanoid behavioral foundation model that can be prompted to solve a variety of whole-body tasks, including motion tracking, goal reaching, and reward opti- mization. The resulting model is capable of expressing human-like behaviors and it achieves competitive performance with task-specific methods while outperform- ing state-of- ofttterrrt unsupervised RL and model-based baselines.'
INTRODUCTION
Foundation models pre-trained on vast amounts of unlabeled data have emerged as the state-of-the- art approach for developing AI systems that can be applied to wide range of use cases and solve complex tasks by responding to specific prompts (c.g.. Anil et al., 2023; OpenAl et al., 2024; Dubey al., 2024). A natural step forward is to extend this approach beyond language and visual domains,
"Joint first author. Work done at Meta. Joint last author. 'Code, models, and interactive demo available http://mmnaammiiiv.eeeeeemmmmmmh

---

Published as a conference paper at ICLR 2025
towards behavioral foundation models (BFMs) for agents interacting with dynamic environments through actions. In this paper. we aim to develop BFMs for humanoid agents and we focus on whole-body control from proprioceptive observations. a long-standing challenge due to the high- dimensionality and intrinsic instability of the system (Peng et al., 2021; Won et al., 2022; Luo et al., 2024a). Our goal is learn BFMs that can express diverse range of behaviors in response vari- ous prompts, including behaviors to imitate, goals to achieve, or rewards to optimize. By doing so, we could significantly simplify the creation of general-purpose humanoid agents for robotics (Cheng et al., 2024), virtual avatars, and non-player characters (Kwiatkowski et al., 2022).
While recent advancements in unsupervised reinforeement learning (RL) have demonstrated the po- tential of BFMs, several limitations still exist. Pre-trained policies or representations (e.g., Eysen- bach et al., 2019; Schwarzer et al., 2021) still require training an RL agent to solve any given down- stream task. Unsupervised zero-shot RL (e.g., Touati et al., 2023; Frans et al., 2024) addresses this limitation by pre-training policies that are promptable (e.g.. by rewards or goals) without additional learning planning. However, this approach relies on 1) access to large and diverse datasets of transitions collected through some unsupervised exploration strategy, and 2) optimize unsupervised losses that aim at learning as many and diverse policies as possible, but provide limited inductive bias on which ones to favor. As a result, zero-shot RL performs well in simple environments (e.g., low-dimensional continuous control), while struggle in complex scenarios with high-dimensional control and unstable dynamics, where unsupervised exploration is unlikely to yield useful samples and unsupervised learning may lead to policies that are not well aligned with the tasks of interest.
An alternative approach is to train sequence models (c.g.. transformer- or diffusion-based) from large demonstration datasets to clone imitate existing behaviors and rely on their generalization capa- bilities and prompt conditioning to obtain different behaviors (e.g., Schmidhuber, 2019; Chen et al., 2021: Wu et al., 2023). This approach particularly effective when high-quality task misk-oriented data are available, but it tends to generate models that are limited to reproducing the policies demon- strated in the training datasets and struggle to generalize to unseen tasks (Brandfonbrener et al., 2022). Recently, several methods (e.g., Peng et al., 2022; Gehring et al., 2023; Luo et al., 2024b) integrate demonstrations into an RL routine to learn "regularized" policies that preserve RL gen- eralization capabilities while avoiding the issues related to complete unsupervised learning. While the resulting policies can serve as behavior priors, a full hierarchical RL process often needed to solve any specific downstream task. See App. A for full review of other related works.
In this paper, we aim at leveraging an unlabled dataset of trajectories to ground zero-shot RL algo- rithms towards BFMs that not only express useful behaviors but also retain the capability of solving a wide range of tasks in a zero-shot fashion. Our main contributions in this direction are:
We introduce FB-CPR tion) a novel online unsupervised RL algorithm that grounds the unsupervised policy learning of forward-backwand (FB) representations (Touati & Ollivier, 2021) towards imitating observation- only unlabeled behaviors. The key technical novelty of FB-CPR is to leverage the FB represen- tation to embed unlabeled trajectories to the same latent space used to represent policies and use a latent-conditional discriminator to encourage policies to "cover" the states in the dataset.
representations with Conditional Policy Regulariza-
We demonstrate the effectiveness of FB- FB-CPR by training a BFM for whole-bodi control of hu- manoid that can solve wide range of tasks (i.e., motion tracking, goal reaching, reward optimiza- tion) in zero-shot. We consider humanoid agent built on the SMPL skeleton (Loper et al., 2015), which is widely used in the virtual character animation community for its human-like structure, and we use the AMASS dataset (Mahmood et al., 2019), large collection of uncurated motion capture data, for regularization. Through an extensive quantitative and qualitative evaluation, we show that our model expresses behaviors that are Bhuman-like and it is competitive with hd- methods trained for specific tasks while outperforming unsupervised RL as well as model-based baselines. Finally, run extensive ablations in the humanoid, walker (App. E), and ant maze (App. F) environments demonstrating the crucial role of the regularization scheme of FB-CPR.
PRELIMINARIES
We consider a reward-frre discounted Markov decision process M A are the state and action space respectively, P is the transition kernel, where P(ds' (s,() denotes
A, AS, where S and
2

---

Published as a conference paper at ICLR 2025
the probability measure over next states when executing action from state μ is distribution over initial states, and y E [0, 1) is a discount factor. A policy æ is the probability measure x r(da)s) that maps each state to distribution over actions. We denote Pr(- So, ag. TT) and E[- So, dg, |] the probability and expectation operators under state-action sequences (8t, az az)e20 starting at (8o, do) and following policy π with 8t P(ds,(st-1,(0--) and a, (ddar[(s).
Successor measures for zero-shot RL. For any policy т, its successor measure (Dayan, 1993; Blier al., 2021) the (discounted) distribution of future states obtained by taking action a in state and following policy thereafter. Formally, this is defined as
M M (X| (s,a) == Ei=o7 7 Pr(se+1 E X a)
S,
(1)
and it satisfies a measure- Bellman equation (Blier et al., 2021),
M= = P(X 8, + 7 - Pl)aa))) ( ( a'), XCS.
(2)
We also define (X) := (1 - Bl [M a)] as the stationary discounted distribu- tion of æ. Given MT, the action-vall value function of π for any reward function r: S is
O (s) = E Σ >r(()))) 8, -f M (ds' ())((s) J'EES
The previous expression conveniently separates the value function into two terms: 1) the successor measure that models the evolution of the policy in the environment, and 2) the reward function that captures task-relevant information. This factorization suggests that learning the successor measure for allows for the evaluation of Q any reward without further training, i.e., zero-shot policy evaluation. Remarkably, using a low-rank decomposition (FB) representation (Blier et al., 2021; Touati & Ollivier, 2021) enabling not only zero-shot policy evaluation but also the ability to perform zero-shot policy optimization.
the successor measure gives rise to the
forward-Backward (FB) representations. The FB representation aims to learn finite-rank ap- proximation to the successor measure as M (X|s, a) ~ Swex F (8, a) ((s)),,))) where is the a state distribution, while F" S A - Rd and B - Rd are the forward and backward embedding, respectively. With this decomposition, for any given reward function r" the action-value function be expressed O (s. a) 5 (s, a) 2, where = Ex-p [B (s))r(s)) is the mapping of the reward onto the backward embedding B. An extension of this approach to multiple policies proposed by Touati & Ollivier (2021), where both and π are parameterized by the same task encoding vector z. This results in the following unsupervised learning criteria for pre-training:
= (X|s,a) (s,a) a) Re SWex F(s, a, 2) B(5') p(ds'), (s) = arg max,, F(s, z)
Vs S, a A, X S, Z
(4)
V(s E A.SSA,28 2E Z,
where Z Rd (c.g., the unit hypersphere of radius Vd). Given the policies (mz). F and B are trained to minimize the temporal difference loss derived as the Bellman residual of Eq. 2
LEB (F, B) E 2nm,(s,a,'' + tp (TT('')
(F(s, [B(+) - FF's B( +
(5)
2E 2E [((s 2)) B(s')]
where v is a distribution over Z. and F. B denotes stop-gradient. In continuous action spaces, the arg max in Eq. is approximated by training an actor network to minimize
Lactor (T) = -epnesssp.anm..ss)
F (ss,a,))
(6)
In practice, FB models have been trained offline (Touati et al., 2023; Pirotta et al., 2024; Cetin et al., 2024b), where p the distribution of a dataset transitions collected by unsupervised exploration.
Zero-shot inference. Pre-trained FB models can be used to solve different tasks in zero-shot fash- ion, i.e., without performing additional task-specific learning, planning, or fine-tuning. Given a dataset of reward samples {(81, r:)] a geward-maximizing policy w... is inferred by computing 2, = + (s,,) Similarly, we can solve zero-shot goal-reaching problems for any state
"The inferred latent can also be safely normalized since optimal policies are invariant reward scaling.
3

---

Published as a conference paper at ICLR 2025
Pre- Pre-Triinn
deera 6013
(3)t)
Ova
2(;,2)
nead Ang
Holey
Figure 2: Illustration of the main components of FB-CPR: the discriminator is trained estimate the ratio between the latent-state distribution induced by policies (πæ) and the unlabeled behavior dataset M, where trajectories are embedded through EREB- The policies are trained with a regu- larized loss combining a policy improvement objective based on the FB action value function and critic trained on the discriminator. Finally, the learned policies are rolled out to collect samples that are stored into the replay buffer Dolline-
8 E S by executing the policy Ts, where 2, = Bs). Finally, Pirotta et al. (2024) showed that FB models can be used implement different imitation learning criteria. In particular, recall the empirical reward via FB approach where, given a demonstration 3 = (81....,8,) 88) from an expert policy, the zero-shot inference returns 2, ER = Σ B(st).
In the limit of and full coverage of FB can learn optimal policies for any reward function and solve any imitation learning problem (Touati Ollivier, 2021). However, when d is finite, FB training has limited inductive bias on which policies to favor, except for the low-rank dynamics assumption, and when the dataset has poor coverage, it cannot reliably optimize policies using offline learning. In this case, FB models tend to collapse to few policies with poor downstream performance on tasks of interest (see experiments on walker in App. E).
FB WITH CONDITIONAL POLICY REGULARIZATION
At pre-training, the agent has access to a dataset of unlabeled behaviors M == (T). which contains observation-only trajectories T = (s1,..., *t(r))* where states are drawn from a generic distribution p" (X), X C S. Furthermore, the agent can directly interact with the environment from initial states 8o and we denote by Douline the associated replay buffer of (unsupervised) transitions.
FB with conditional policy regularization. We now describe how we steer the unsupervised train- ing of FB towards capturing the diverse behaviors represented in M. We first outline our for- malization the problem, followed by detailed discussion of the design choices that enable the development of a scalable and effective algorithm.
In FB, we pretrain continuous set of condit-cond policies (daas, z). where 2 is drawn from distribution y defined over the latent space Z. The space of behaviors represented by FB can be compactly represented by the joint space (s,z) where z ~ y and ~ p=* We denote by (s, 2) =(=)p==(s) (s) the joint distribution induced by FB over this space. We summarize the behaviors represented in the unlabeled dataset in similar way by assuming that each trajec- tory can be produced by some FB policy и.- Since the dataset only contains states with latent variables, for each trajectory we must infer a latent z such that the policy would visit the same states as т. Pirotta al. (2024) proposed several methods for inferring such latent variables from a single trajectory using an FB model. Among these, we choose to encode trajectories using ERrB, a simple yet empirically effective method, and represent each trajectory in the dataset as
While the original method is defined for multiple trajectories, here we report the single-trajectory case for notation convenience and to match the way we will use it later. 'In humanoid, we use motion capture datasets where trajectories may contain noise and artifacts and, in general, generated by "purposeful" stationary policies.
4

---

Published as a conference paper at ICLR 2025
{(s. = ( We assume uniform distribution over M and denote by (s, the joint distribution of the dataset induced by this process.
To ensure that FB policies encode similar behaviors to the ones represented in the dataset, we regu- larize the unsupervised training of the FB actor with distributionnmaateee matching objective that minimizes the discrepancy between pr (2, 8) and (MM This results in the following actor training loss:
LIB-CPR (TT) - -E ,, Doaline a (+1s) F(s, a, 2) =] aaKL(pm: P.M),
where a is hyper-parameter that controls the strength of the regularization.
Distribution matching objective. We now explain how turn Eq. into tractable RL procedure. The key idea is that we can interpret the KL-divergence as an expected return under the polices where the reward given by the log-ratio P.M 2)
/)/px (s, 2) of the two distributions,
log Pm(8, 2) E
r log P.M(8+1122)
KL (L=, PM) = soyy, smup" (s,
~ 14, m= (8)
(=((,1,)2
To estimate the reward term, we employ a variational representation of the -ensen-Shannon diver- gence. Specifically, we introduce discriminator network D : 5 x Z - [o, 1] conditioned on the latent 2 and train it with a GAN-like objective (Goodfellow et al., 2014),
Ldiseriminator ( (D) = M,spp (Iog(D(s.))) (RRB(T)))
E.- [log(1 ((s, 2))] (9)
It is known that the optimal discriminator for the loss in Eq. 9 is D PM+PM et al., 2014; Nowozin et al., 2016), which allows us approximating the log-ratio reward function D as log RM x log T- We can then fit a critic network Q to estimate the action-value of this f+ approximate reward via off- off-policy TD learning.
PM (e.g.. Goodfellow
(2(s..0))- log 1- Dss'')))
[
D(5',2)
2
Lentic (Q) =E (ts.a,d")DDonii zoara - (-1s')
-(((s
(10)
This leads us to the final actor loss for FB-CPR,
LFR.CR((T) - -Bg.p.sseeenntn......
[F(s, a, 2)), aQ(s,
(11)
Latent space distribution. So far, we have not specified the distribution over the latent space Z. According to the FB optimality criteria (Touati & Ollivier, 2021), it is sufficient to choose a distribution that has support over the entire hypersphere. However, in practice, we can leverage v to allocate more model capacity to meaningful latent tasks and to enhance the training signal provided by and to the discriminator, while ensuring generalization over variety of tasks. particular, we choose v as a mixture of three components: 1) = = ER FRRBB(? (T) where r ~ M,, which encourages FB to accurately reproduce each trajectory in the unlabeled dataset, thus generating challenging samples for the discriminator and boosting its training signal: 2) B(s) where E Donline which focuses on goal-reaching tasks for states observed during the training process; and 3) uniform over the hypersphere, which allocates capacity for broader tasks and covers the latent space exhaustively.
Online training and off-policy implementation. FB-CPR is pre-trained online, interleaving en- vironment interactions with model updates. During interaction, we sample N policies with = and rollout each for a fixed number steps. All the collected (unsupervised) transitions are added to a finite capacity replay buffer Donline. We then use an off off-policy procedure update all components of FB-CPR: F' and B using Eq. 5, the discriminator D using Eq. 9, the critic Q using Eq. 10, and the actor using equation 11. The full pseudo-code of the algorithm reported in App. B.
Discussion. While the distribution matching term in Eq. 8 is closely related to existing imitation learning schemes, it has crucial differences that makes it more suitable for our problem. Peng et al. (2022) and Vlastelica et al. (2024) focus on the state marginal version of pa and PM. thus regularizing towards policies that globally cover the same states as the behaviors in M. In general, this may lead to situations where no policy can accurately reproduce the trajectories in M. Tessler et al. (2023) address this problem by employing a conditional objective similar to Eq. 8, where a trajectory encoder is learned end-to-end together with the policy space (Tz). In our case, distribution matching is used to regularize the FB unsupervised learning process and we directly use ERFB to embed trajectories into the latent policy space. Not only this simplifies the learning process by removing an ad-hoc trajectory encoding, but it also binds FB and policy training together, thus ensuring a more stable and consistent learning algorithm.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS ON HUMANOID
We propose a novel suite of whole-body humanoid control tasks based on the SMPL hu- manoid (Loper et al., 2015), which is widely adopted in virtual character animation (e.g., Luo et al., 2021; 2024a). The SMPL skeleton contains 24 rigid bodies, of which 23 are actuated. The body proportion can vary based on body shape parameter, but in this work use neutral body shape. The state consists of proprioceptive observations containing body pose (70D), body rotation ( (144D), and linear and angular velocities (144D), resulting in a state space S C R-35s All the components of the state are normalized w.r.t. the current facing direction and root position (e.g.. Won et al,, 2022: Luo et al., 2023). We use proportional derivative (PD) controller and the action space A (-1, 1]60 thus specifies the "normalized" PD target. Unlike previous work, which considered an under-constrrined skeleton and over-actuated controllers, we define joint ranges and torque limits to create "physically plausible" movements, The simulation is performed using MuJoCo (Todorov et al., 2012) at 450 Hz, while the control frequency is 30 Hz. More details in App. C.I.
Motion datasets. For the behavior dataset we use a subset of the popular AMASS dataset (Mahmood et al., 2019), which contains combination of short, task-specific motions (e.g., few seconds of running or walking), long mixed behaviors (e.g., more than 3 minutes of dancing or daily house activities) and almost static motions (e.g.. greeting, throwing). Following previous approaches (e-g.. Luo et al., 2021: 2023; 2024b), we removed motions involving interactions with objects (e.g., stepping on boxes). After a 10% train-test split, we obtained a train dataset MM of 8902 motions and a test dataset MTEST of 990 motions, with total duration of approximately 29 hours and 3 hours, respectively (see Tab. 2 in App. C.2). Motions are action-free, comprising only body position and orientation information, which we supplement with estimated velocities using finite difference method. Some motions may exhibit variations in frequency, discontinuities such as joint flickering, or artifacts like body penetration, making exact reproduction impossible in simulation, thereby increasing the realism and complexity of our experimental setting.
Downstream tasks and metries. The evaluation suite comprises three categories (see App. C.3 for details): 1) reward optimization, which involves 45 rewards designed to elicit range of behav- iors, including static/slow and dynamic/fast movements that require control of different body parts and movement at various heights. The performance is evaluated based on the average return over episodes of 300 steps, with some reward functions yielding policies similar to motions in the dataset and others resulting in distinct behaviors. 2) goal reaching. where the model's ability to reach a goal from an arbitrary initial condition is assessed using 50 manually selected "stable" poses. Two metrics are employed: success rate, indicating whether the goal position has been attained at any point, and proximity, calculated as the normalized distance to the goal position averaged over time. 3) tracking, which assesses the model's 'ss capacity to reproduce target motion when starting from its initial pose. A motion is considered successfully tracked if the agent remains within a specified distance (in joint position and rotation) to the motion along its entire length (Luo et al., 2021). Ad- ditionally, the earth mover's distance (Rubner et al., 2000, EMD) is used as a less-restrictive metric that does not require perfect time-alignment between the agent's trajectory and the target motion.
Protocol and baselines. We first define single-task baselines for each category. We use TD3 (Fu- jimoto et al., 2018) trained from scratch for each reward-maximization and goal-reaching task. We also train Goal-GAIL (Ding et al., 2019) and PHC (Luo et al., 2023) on each individual motion to have strong baselines for motion tracking. All the algorithms are trained online." 5 We then con- sider "multi-task" unsupervised RL. algorithms. Goal-GAIL. and Goal-TD3 are state-of-th therthe-rtt goal- conditioned RL algorithms. PHC is coal-conditioned algorithm specialized for motion tracking and CALM (Tessler et al., 2023) is an algorithm for these baselines are trained online and leverage M in the process. ASE (Peng et al., 2022) is the closest BFM approach to ours it allows for zero-shot learning and leverages motions for regular- ization. We train ASE online with M using an off -olicy routine. An extensive comparison to other unsupervised skill discovery methods is reported in App. H. We also test planning- based approaches such as MPPI (Williams et al., 2017), DIFFUSER (Janner et al., 2022) and H-GAP (Jiang et al., 2024). All these methods offline and require action-labeled datasets. For this purpose, first create an action-labeled version of the AMASS dataset by replaying policies from single-motion
imitation learning. All
'We pick the best performance over 5 seeds for reward and goal-based tasks, and run only one seed for single-motion tracking due the high volume of motions. Standard deviations thus omitted in Tab. 1.

---

Published as a conference paper at ICLR 2025
Algorithm
Reward (1)
Goal
Tracking EMD
Tracking
Success (f)
Proximity (1)
Success (T)
Train
Test
Train
Test
TD3
249.74
0.98
0.98
GOAL--GAID
1.08
1.09
0.22
0.23
PHCI
1.14
1.14
0.91
0.91
ORACLE MPPT
178.50
0.47
0.73
GOAL- TD3
0.67 (0.34)
0.44 (0.47)
1,39 (0.08)
1.41 (0.09)
0.90 (0.01)
0.91 (0.01)
GOAL-GAIL
0.61 (0.35)
0.35 (0.14)
1.08 (0.02)
1.70 (0.02)
0.25 (0.01)
0.25 (0.02)
PHC
0.07 (0.11)
0.05 (0.11)
1.66 (0.06)
1.65 (0:07)
0.82 (0.01)
0.83 (0.02)
CALM ASE
0.18 (0.27)
0.01 (0.17)
1.67 (0.02)
1.70 (0.03)
0.71 (0.02)
0.73 (0.02)
105.73 (3.82)
0.46 (0.37)
0.22 (0.37)
(0.02)
1.99 (0.02)
0.37 (0.02)
0.40 (0.03)
DIFFUSER
85,27 (0.99)
0.20 (0:03)
0.14 (0.01)
FB-CPR
151.68 (7.53)
0.68
0.48 (0.16)
1.37
1.39 (0.01)
0.83
0.83 (0.01)
SCORF
0.61
0.69
01.48
0.80
0.80
0.88
0.88
Table I: Summary results comparing FB-C CB-CPR to different single-task baselines (i.e., retrained for each task) and "multi-task" unsupervised baselines across three different evaluation categories. We report mean and standard deviation across seeds. For FB-CHB CPRC we report the normalized per- formance against the best algorithm, i.e., SCOREnorm that the best algorithm may vary depending on the metric being evaluated (TD3 for reward and goal, Goal-GAIL for tracking EMD and PHC for tracking success). For each metric, we highlight the best "multi-task" baseline and the second best "multi-task" baseline. i are top-liner runs on individual tasks, goals or motions (we use the best performance over seeds).
E task FBB FFB--PR((ten (tR(task), (BEST(task) Note
Goal-GAIL and then combine it with the replay buffer generated by FB-CPR to obtain diverse dataset with good coverage that can be used for offline training (more details in App. C.I).
We use a comparable architecture and hyperparameter search for all models. Online algorithms are trained for 3M gradient steps corresponding to 30M interaction steps. Evaluation is done by averaging results over 100 episodes for reward and goal, and with a single episode for tracking, as the initial state fixed. Due the high computational cost, we were able compute metrics over only 20 episodes for MPPI and DIFFUSER. We provide further implementation details in App. C.5.
4.1
MAIN RESULTS
Table presents the aggregate performance of each algorithm for each evaluation category. MPPI with a learned model and H-GAP exhibit poor performance in all tasks, thus their results are not included in the table (see App. D.1); instead, an oracle version of MPPI serves as a planning- top-line. On average, FB-CPR achieves 73.4% of the top-line algorithms' performance across all categories, a remarkable result given its lack of explicit training for downstream tasks and ability to perform zero-shot inference without additional learning or planning. Furthermore, FB-CPR out- performs ASE by more than 1.4 times in each task category and matches or surpasses specialized unsupervised RL algorithms. We now provide in-depth analysis of each category, while finer breakdown of the results is available in App. D.I.
mwvardmaxxiizzttion. In reward-based tasks FB-CPR achieves 61 % of the performance of TD3, which is re-trained from scratch for each reward. Compared to unsupervised baselines, FB-CPR outperforms all the baselines that requires planning on learned model. For example, FB-CPR achieves 177% of the performance of DIFFUSER that relies on larger and more complex model to perform reward optimization. ORACLEMPPI performs better than FB-CPR, while still lagging behind model-free TD3. This improvement (+17.8% w.r.t. FB-CPR) comes at the cost of signifi- cant increase in computational cost. ORACLEMPPI requires at least 30 minutes complete a 300 step episode compared to the 12 seconds needed by FB-CPR to perform inference and execute the policy (about 7, 3 and seconds for reward relabeling, inference, and policy rollout). DIFFUSER takes even more, about 5 hours for a single episode. While this comparison is subject specific im- plementation details, it provides an interesting comparison between pre-training zero-shot policies and using test-time compute for planning. Finally, ASE, which has the same zero-shot properties as FB-CPR, only achieves 70% of its performance across all tasks.
Goal-reaching, Table shows that FB-CPR performs similarly to specialized goal- based baselines (i.e.. Goal-GAll GAIL and Goal-TD3) and outperforms the zero-shot bascline (48% and 118% perfor- mance increase w.r.t. ASE proximity and success). When compared with planning-base ap-
7

---

Published as a conference paper at ICLR 2025
Success [%)]
Naturalness (l.e., human-likeness) [%]
100.0
100.0
Reward
Goal
83.5
60.0
69-1
63.2
60.0
49.8
54.0
60.0
40.0
20.3
20.0
20.0
6.0
5.3
6.3
4.6
0.0
TD3
FD-CPR
TD3 FB-CPR
FB-CPR TD3
Draw
None
Figure 3: Human-evaluation. Left figure reports the percentage of times a behavior solved a reward- based (blue) goal-reaching (pink) task (tasks are independently evaluated). Right figure reports the score for numan-likeness by direct comparison of the two algorithms.
proaches, FB-CPR achieves a higher proximity but lower success rate. This means that FB-CPR is able to spend more time close to the goal, whereas ORACLEMPPI is able to reach the goal but not keeping a stable pose thereafter. We believe this is due to the fact that ORACLEMPPI aims to mini- mize only the distance w.r.t. position at planning without considering velocities." Finally, similarly to the reward case, all other algorithms under-perform w.r.t. TD3 trained to reach each individual goal independently." Since Goal-TD3 is trained using the same reward signal, the conjecture is that the unsupervised algorithm learns behaviors that are biased by the demonstrations. Indeed, by visually inspecting the motions, noticed that TD3 tends to reach the goal in faster way, while sacrificing the "quality" of the behaviors (further details below).
Tracking. We first notice that the same algorithm may have quite different success and EMD met- rics. This is the case for Goal-GAlL, which achieves low EMD but quite poor success rate. This is due the fact that Goal-GAIL is trained to reach the goal in few steps, rather than in sin- gle step. On the other hand, Goal-TD3 is trained to reach the goal in the shortest time possible and obtain good scores in both EMD and success metries. We thus used two different algorithms trained on single motions for the top-line performance in EMD (Goal-GAIL) and success (PHC). The performance of FB-CPP BB-CPR about 80% and 88% of the top-line scorer for EMD and success, and it achieves an overall 83% success rate on the test dataset. Similarly to previous categories, baselines. Among "multi-task" baselines. only Goal-TD3 is able to do better than FB-CPR on average (about 9% improvement in success and a 1% drop in EMD). Interestingly. PHC achieves the same performance of FB-CPR despite being an algorithm designed specifically for tracking". Due to the high computation cost, were not able to test MPPI and DIFFUSER on tracking.
FB-CPR outperforms both zero-shot and planning-
Qualitative Evaluation. A qualitative evaluation was conducted to assess the quality of learned be- haviors, as quantitative metrics alone do not capture this aspect. In line with previous work (Hansen al., 2024a), we employed 50 human evaluators compare clips generated by TD3 and FB-CPR for episodes of the same task. The evaluation involved rating whether the model solved the task or achieved the goal, and which model exhibited more natural behavior (see App. D.4 for details). This study encompassed all 45 rewards and 50 goals, with results indicating that despite TD3 achieving higher rewards, both algorithms demonstrated similar success rates reward- based tasks, producing intended behaviors such as jumping and moving forward (cf. Fig. 3). Notably, FB-CPR was per- ceived as more human-like in 83% of cases, whereas TD3 was considered more natural in only 4% of cases. This disparity highlights the issue of underspecified reward functions and how motion reg- ularization in FB-CPR compensates for it by capturing human-like biases. In App. D.4.2, we provide further examples of this "human bias" in underspecified and composed rewards. In goal-reaching tasks, human evaluators' assessments of success aligned with our qualitative analysis, showing that FB-CPR exhibited 6% improvement while TD3 experienced an 11% drop. Furthermore, FB-CPR was deemed more human-like in 69% of cases, even though TD3 had a higher success rate. In the remaining cases, evaluators considered TD3 and FB-CPR equally good for 20% of the goals, while
"We tried to train with full distance (i.c., position and velocities) but we did not get any significant result. TTD3 is trained using the full distance to the goal as reward function.
'The original PPO-hased implementation of PHC (Luo al., 2024b) achieves 0.95 tracking accuracy on both the train and test set, but leverages information not available to FB-CPR (e.g., global positions).
8

---

Published as a conference paper at ICLR 2025
Discriminator Policy Conditioning
Agent Controllability
Reward Evaluation (1)
Tracking Evaluation (4)
Reward Evaluation (1)
Tracking Evaluation (4)
Earth Earth Mover's Distance
Earth Mover's Distance
Normalzed Return
0.60
1,5
Return
0.60
1.2
0.9
0.9
0.30
0.30
0.3
0.3
0.0
(4.2)
No
F
No
F1z
Scaling Capacity & Data Tracking Evaluation (4)
Offline FB vs. Online FB-CPR
Reward Evaluation (f)
Tracking Evaluation (4)
Earth Mover's Distance
1.65
0.
Normalized Return
0.6
Earth Mover's Distance
1.60
1.55
0.5
3.0
1.50
0.3
1.45
0.2
1.0
0.0
0.0
FB FB-AW
FB:CPR
FB-AW FB-CPR
Data Proportion
Figure 4: FB-CPR Ablations. (ToP LEFT) Ablating the FB-CPR discriminator's policy condi- tioning. (ToP RIGHT) Ablating the contribution of F(=)T> 2 in the FB-CPR actor loss (Eq. 11). (BOTTOM LEFT) The effect of increasing model capacity along with the number of motions in the dataset M. (BOTTOM RIGHT) Contrasting Advantage-Weige
(FB-AW) trained from a large diverse offline dataset versus FB-CPP trained fully online with policy regularization. All ablations are averaged over seeds with ranges representing bootstrapped 95% confidence intervals.
TD3 was better in only 6% of the goals. Finally, we report additional qualitative investigation on the embedding and the space of policies in App. G.
4.2 ABLATIONS
Various design decisions have gone into FB-CPR that deserves further analysis. In the following, we seek to answer key questions surrounding the necessity of online interaction and how components of our algorithm affect different axes performance. Additionally, Appendix D..2 provides further ablations on design decisions regarding the FB-CPR diseriminator, sampling distribution v, and other forms policy regularization when provided action labels.
Is online policy regularization necessary given large diverse dataset? Prior works unsu- pervised RL have relied on large and diverse datasets that contain sufficient coverage any down- stream task. If such datuset exists is there anything to be gained from the guided approach of online FB-CPR outlined herein? In order to test this hypothesis. we evaluate training offfine FB with an advantage weighted actor update (Nair al., 2020) (FB-AW) which compensates for overestima- tion when performing policy optimization with an offline dataset (Cetin et al., 2024b). As no dataset with our criterion exists, curate dataset by collating all 30M transition from an online FB-CPR agent. The offline agent is trained for the same total number of gradients steps as the online agent and all hypereparameters shared between the two methods remain fixed. In the bottom right quad- rant of Figure 4. we can see that FB-AW perform substantially worse than FB-CPR highlighting the difficulty of offline policy optimization and the efficacy of guiding online interactions through the conditional policy regularization of FB-CPR.
How important is maximizing the unsupervised RL term F(z) 2? The primary mechanism by which FB-CPR regularizes its policy is through the discriminator's 's critic (Eq. 10). This begs the question to what extent is maximizing the unsupervised value-function F'ss, a, x) contributes to the overall performance of FB-CPR. To answer this question, train FB-CPR while omitting this unsupervised term when updating the actor. This has the effect reducing FB-CPR to more akin to CALM (Tessler et al., 2023), except that motions encoded with FB through ER These results are presented top right quadrant of Figure 4 for both reward and tracking based performance measures. We can see that including the unsupervised falue-function from FB results

---

Published as a conference paper at ICLR 2025
in improved performance in both reward and tracking evaluation emphasizing that FB is providing much more than just a motion encoder through ER
How important is policy conditioning for the discriminator? FB-CPR relies on latent- conditional discriminator to evaluate the distance between a specific motion and a policy selected through the trajectory embedding of ER We hypothesize that this policy-conditioned discrimina- tor should provide a stronger signal to the agent and lead to better overall performance. We test this hypothesis by comparing FB-CPR with discriminator that solely depends on state, thus converting the regularization term into a marginal state distribution matching. The top left quadrant of Figure shows that the cecet-conditioned discriminator outperforms the state-only configuration in tracking tasks while performing similarly in reward tasks. These findings demonstrate the importance of the FR embedding in enabling FB-CPR to more accurately reproduce motions.
How does network capacity and expert dataset size impact FB-CPR performance? Many recent works in RL have shown vast performance improvements when scaling the capacity of neural net- works (Schwarzer et al., 2023; Obando-Ceron et al., 2024; Nauman et al., 2024) along with dataset size (Brohan et al., 2023; Zitkovich et al., 2023) or task diversity (Kumar et al., 2023; Ali Taïga et al., 2023). Given these findings, we seek to understand the capabilities of FB-CPR when scaling both the network capacity and the number of expert demonstrations. To this end, we perform a grid sweep over three configurations of model sizes that alters the amount of compute by roughly (0.5x, 1 x, 2x) } of the base models; as well as datasets that are {6.25 25,, 12.5%, 25%, 50%, 100%) the size of our largest motion dataset via subsampling. For each of these combinations tracking performance on all motions and present these results in the bottom left quadrant of Figure 4 with additional evaluation metrics in Appendix D.2. Consistent with prior results we can see that larger capacity models are better able to leverage larger motion datasets resulting in significantly improved performance for our 2 larger model over the results of the 1 model reported in Table
report the
Scaling FB-CPR to very deep architectures. To scale further and avoid vanishing/oxploding gradi- ents, we replace MLP layers with blocks akin to those of transformer architectures (Vaswani, 2017), involving residual connections, layer normalization, and Mish activation functions (Misra, 2019). With this simple modification, we could train our largest and most capable model, outperforming our base model both in size (from 25M to 288M parameters) and performance (see table below).
Algorithm
Reward (f)
Goal
Tracking
EMD (L)
Tracking
Success (f)
Proximity (f)
Success (f)
Train
Test
Train
Test
FB-CPR
179.94
0.82 0.84
0.66
1.11
1.13
0.84
0.84
SpOREporm
0.72
0.67
0.97
0.96
0.89
0.89
CONCLUSIONS
We introduced FB-CPR, novel algorithm combining the zero-shot properties of FB models with a regularization grounding online training and policy learning on a dataset of unlabeled behaviors. We demonstrated the effectiveness of FB-CPR by training the first BFM for zero-shot control of complex humanoid agent with state-of-the-an ahrt performance across a variety of tasks.
While FB-CPR effectively grounds unsupervised RL with behavior trajectories, a theoretical under- standing of its components is still lacking and alternative formulations may be possible. In prac- tice, FB-CPR struggles with problems far from motion capture datasets, such as tracking motions or solving reward- -eseed-based tasks involving ground movements. Although FB-CPR produces more human- like behaviors than pure swward-ottinization algorithms and achieves good tracking performance, sometimes generates imperfect and unnatural movements, particularly for behaviors like falling or standing. The BFM trained with FB-CPR limited to proprioceptive observations and cannot solve tasks requiring environmental navigation or object interaction. Integrating additional state variables, including complex perception, could allow models to tackle harder tasks, but this might necessitate test-time planning or fast online adaptation. Currently, FB-CPR relies on expensive motion capture datasets; extending it to leverage videos of various human activities could refine and expand its ca- pabilities. Finally, while language prompting could be added by leveraging text-to-motion models to set tracking targets, an interesting research direction is to align language and policies more directly.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors thank Zhengyi Luo for providing the tracking evaluation of the original PHC implementation on our humanoid environment. The authors thank Claire Roberts, Dominic Burt, Jiemin Zhang, Leonel Sentana, Maria Ruiz, Matt Hanson, Morteza Behrooz, Ryan Win- stead, Spaso Ilievski, Vincent Moens, Vlad Bodurov, William Ngan for significant contribu- tions to the project management and the development of the interactive demo and website at http://metamotiiv.mtaaadmmaaaaaa
REFERENCES
Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G. Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. International Conference on Learning Representations (ICLR), 2023.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS). 2017.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Hean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler. Tim- othy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Dani- helka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Gemini: A Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and family of highly capable multimodal models. CoRR, abs/2312. 12. 11005, 2023.
Bowen Baker, lgge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): learning to act by watching unlabeled online videos. In Neural Information Processing Systems (NeurIPS), 2022.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. CoRR, abs/2101.07123, 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return- supervised learning work for offline reinforcement learning? In Neural Information Processing Systems (NeurIPS), 2022.
David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good representations for multitask imitation. In Neural Information Processing Systems (NeurIPS), 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang- Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems, 2023.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations (ICLR), 2019.
Edoardo Cetin, Andrea Tirinzoni, Matteo Pirotta, Alessandro Lazaric, Yann Ollivier, and Ahmed Touati. Simple ingredients for offline reinforcement learning. In International Conference on Machine Learning (ICMI.,, 2024a.
11

---

Published as a conference paper at ICLR 2025
Edoardo Cetin, Ahmed Touati, and Yann Ollivier. Finer behavioral foundation models via auto- regressive features and advantage weighting, 2024b. URL attpps://axii... org/ abs/ 2412.04368
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. CoRR, abs/2402.16796, 2024.
Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. International Conference on Learning Representations (ICLR), 2023.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5:613-624, 1993.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Neural Information Processing Systems (NeurIPS), 2019.
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. CoRR, abs/2402.03570, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde. Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and al. The llama 3 herd of models. CoRR, 0bs/2407.211783 2024.
Boston Dynamies. Atlas, 2024. URL www www.:oottooyoo.. www.bbsstonddnaaics.com atlas.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- tions (ICLR), 2019.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Marc G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In International Conference on Learning Representations (ICLR), 2023.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforce- ment learning via functional reward encodings. In International Conference on Machine Learning (ICML), 2024.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. In Neural Information Processing Systems (NeurIPS), 2021.
12

---

Published as a conference paper at ICLR 2025
Jonas Gehring, Deepak Gopinath, Jungdam Won, Andreas Krause, Gabriel Synnaeve, and Nicolas Usunier. Leveraging demonstrations with latent space priors. Transactions on Machine Learning Research (TMLR), 2023.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference Machine Learning (ICML.), 2023.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Pro- cessing Systems (NeurIPS), 2014.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. CoRR, abs/1611.0722 1.05007, 2016.
Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In Neural Information Processing Systems (NeurIPS), 2017.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. CoRR, abs/2301.04104 2024.
Nicklas Hansen, Jyothir au2, Vlad Sobal, Yann LeCun, Xiaolong Wang. and Hao Su. Hierar- chical world models as visual whole-body humanoid controllers. CoRR, abs/2405. 18418, 2024a.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: scalable, robust world models for con- tinuous control. In International Conference Learning Representations (ICLR), 2024b.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model an effective planner and data synthesizer for multi-task reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems (NeurIPS), pp. 4565 4573, 2016.
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, and Yuval Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco. CoRR, abs/2212.00544 2022.
Tyler Ingebrand. Amy Zhang, and Ufuk Topcu. Zero-shot reinforcement learning via function en- coders. In International Conference Machine Learning (ICML), 2024.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning (ICML), 2022.
ScottJeen, Tom Bewley, and Jonathan M. Cullen. Zero-shott learning from low quality data. CoRR, abs/2309.15178, 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: Robot manipulation with multimodal prompts. In International Conference on Machine Learning (ICML), 2023.
Zhengyao Jiang, Yingchen Xu, Nolan Wagener, Yicheng Luo, Michael Janner, Edward Grefenstette, Tim Rocktäschel, and Yuandong Tian. H-GAP: humanoid control with generalist planner. In International Conference on Learning Representations (ICLR), 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Martin Klissarov and Marlos C. Machado. Deep laplacian-based options for temporally-extended exploration. In International Conference on Machine Learning (ICML), 2023.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. In International Conference on Learning Representations (ICLR), 2023.
13

---

Published as a conference paper at ICLR 2025
Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C. Karen Liu, Julien Pettré, Michiel van de Panne, and Marie-Paule Cani. A survey on reinforcement learning methods in charac- ter animation. Computer Graphics Forum, 41(2):613-639) 2022.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement leamning benchmark. In Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: contrastive intrinsic control for unsupervised skill discovery. CoRR, abs/2202.00161, 2022.
Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. In Neural Information Processing Systems (NeurIPS), 2022.
Hao Liu and Pieter Abbeel. Behavior from the void: unsupervised active pre-training. In Proceed- ings of the 35th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model. ACM Transactions Graphics, 34(6):248:1- 248: 16, 2015.
Zhengyi Luo. SMPLSim: Simulating smpl/smplx humanoids in mujoco and isaac gym. https: //github. com/ 2eengyinnngyoo SSSLuuo/SMPLSSS ssmm, 2023.
Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. regulttc-regullted kinematic policy for egocentric pose estimation. In Neural Information Processing Systems (NeurIPS), 2021.
Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In International Conference on Computer Vision (ICCV), 2023.
Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Kris Kitani, and Weipeng Xu. Real-time simulated avatar from Pattern Recognition (CVPR), 2024a.
sensors. In Conference on Computer Vision and
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris M. Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. In International Con- ference Learning Representations (ICLR), 2024b.
Zhengyi Luo, Jiashun Wang, Kangni Liu, Haotian Zhang, Chen Tessler, Jingbo Wang, Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, Jessica Hodgins, and Kris Kitani. SMPLOlympics: Sports environments for physically simulated humanoids. CoRR, abs/2407.00187,, 2024c.
Yecheng Jason Ma. Jason Yan. Dinesh Jayaraman, and Osbert Bastani. Offline conl-conditioned reinforcement learning via f f-advantage regression. In Neural Information Processing Systems (NeurIPS), 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations (ICLR), 2023.
Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In AAAI Conference on Artificial Intelligence, 2020.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.
AMASS: archive of motion capture as surface shapes. In International Conference on Computer
Vision (ICCV), 2019.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.
14

---

Published as a conference paper at ICLR 2025
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. In Neural Information Processing Systems (NeurIPS), 2021.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations (ICLR), 2019.
Lina Mezghani. Sainbayar Sukhbaatar, Piotr Bojanowski. Alessandro Lazaric, and Karteek Alahari. Learning goal-conditioned policies offline with self-supervised reward shaping. In Conference on Robot Learning (CoRL), 2022.
D Misra. Mish: A self regularized non-monotonic neural activation function. arxiv. arXiv preprint arXiv:1908.08699 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICIR), 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online
reinforcement learning with offline datasets. CoRR, abs/2006.09359, 2020.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Milos, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Neural Information Processing Systems (NeurIPS), 2024.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Neural Information Processing Systems (NeurIPS), 2016.
Johan Samir Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nico- laus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. In International Conference Machine Learning (ICML), 2024.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Semmadet-SShaiine Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey. Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv
15

---

Published as a conference paper at ICLR 2025
Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine MeLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Oeeeffe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto. Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wo- jeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. CoRR, abs/2303.08774, 2024.
Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- fations, 2022. URL https: / wwwp:eiwwwwweeeeeeeeeeeeeeeeeeee
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: offline goal- conditioned RL with latent states as actions. In Neural Information Processing Systems (NeurIPS), 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned rl. CoRR, abs/2410.20092, 2024a.
Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024b.
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: scalable unsupervised RL with metric- aware abstraction. ICLR. OpenReview.net, 2024c.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imi- tating human behaviour with diffusion models. In International Conference Learning Repre- sentations (ICLR), 2023.
Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics, 40 (4):144:11114:20, 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics, 41 -14 2022.
Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imi- tation via behavior foundation models. In International Conference on Learning Representations (ICLR), 2024.
16

---

Published as a conference paper at ICLR 2025
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervvsd reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Cheng Qian, Julen Urain. Kevin Zakka, and Jan Peters. Pianomime: Learning generalist, dexterous piano player from internet demonstrations. CoRR, abs/2407.18178 2024.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron C. Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 28598-286117 PMLR, 2023.
Daniele Reda, Jungdam Won, Yuting Ye, Michiel van de Panne, and Alexander Winkler. Physics- based motion retargeting from sparse inputs, Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3), 2023.
Juntao Ren, Gokul Swamy, Steven Wu, Drew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning. International Conference on Muchine Learning, (ICML), 2024.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, 2000.
Jürgen Schmidhuber. Reinforcement learning upside down: Don't predict rewards just map them to actions. CoRR, abs/1912.02875. 2019.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data- efficient reinforcement learning. In Neural Information Processing (NeurIPS), 2021.
Max Schwarzer, Johan Samir Obando-Ceron, Aaron C. Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference Machine Learning (ICML), 2023.
Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. CoRR, 0hs/23009009551 2023.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. CoRR, hbss/2403.10506 2024.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behav- ior transformers: Cloning modes with one stone. In Neural Information Processing Systems (NeurIPS), 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representations (ICLR), 2020.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning. In Conference on Robot Learning (CoRL.), 2022.
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Steven Wu. Of moments and match- ing: A thee-the framework for closing the imitation gap. In International Conference on Machine Learning (ICML), 2021.
Gokul Swamy, Nived Rajaraman, Matthew Peng, Sanjiban Choudhury. J. Andrew Bagnell, Steven Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via P Processing Systems (NeurIPS), 2022.
replay estimation. In Neural Information
17

---

Published as a conference paper at ICLR 2025
SIMA Team, Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, Stephanie C. Y. Chan, Jeff Clune, Adrian Collister. Vikki Copeman, Alex Cullum, Ishita Dasgupta, Dario de Cesare, Julia Di Trapani, Yani Donchev, Emma Dunleavy, Martin Engelcke, Ryan Faulkner, Frankie Gar- cia, Charles Ghadamosi, Zhitao Gong, Luey Gonzales, Kshitij Gupta, Karol Gregor, Arne Olav Hallingstad, Tim Harley, Sam Haves, Felix Hill, Ed Hirst, Drew A. Hudson, Jony Hudson, Steph Hughes-Fitt, Danilo J. Rezende, Mimi Jasarevic, Laura Kampis, Rosemary Ke, Thomas Keck, Junkyung Kim, Oscar Knagg, Kavya Kopparapu, Andrew Lampinen, Shane Legg, Alexander Lerchner, Marjorie Limont, Yulan Liu, Maria Loks Thommson Joseph Marino, Kathryn Mar- tin Cussons, Loic Matthey, Siobhan Mcloughlin, Piermaria Mendolicchio, Hamza Merzic, Anna Mitenkova, Alexandre Moufarek, Valeria Oliveira, Yanko Oliveira, Hannah Openshaw, Renke Pan. Aneesh Pappu, Alex Platonov, Ollie Purkiss, David Reichert, John Reid, Pierre Harvey Richemond, Tyson Roberts, Giles Ruscoe, Jaume Sanchez Elias, Tasha Sandars, Daniel P. Sawyer, Tim Scholtes, Guy Simmons, Daniel Slater, Hubert Soyer, Heiko Strathmann, Peter Stys, Alli- son C. Tam, Denis Teplyashin, Tayfun Terzi, Davide Vercelli, Bojan Vujatovic, Marcus Wain- wright, Jane X. Wang, Zhengdong Wang, Daan Wierstra, Dunean Williams, Nathaniel Wong, Sarah York, and Nick Young. Scaling instructable agents across many simulated worlds. CoRR, 101/2404.10179, 2024.
Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. In Neural Information Processing Systems (NeurlPS), 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations (ICLR), 2023.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638.
UniTree. HI, 2024. URL www. www.wueiieee.e.m.m com/h..
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Marin Vlastelica, Jin Cheng, Georg Martius, and Pavel Kolev. Offline diversity maximization under imitation constraints. In Reinforcement Learning Conference (RLC), 2024.
Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, and Matthew J. Hausknecht. Mocapact: A multi-task dataset for simulated humanoid control. In Neural Information Processing Systems (NeurlPS), 2022.
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research (TMLR), 2024.
Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics- based imitation of dynamic human-object interaction. CoRR, abs/2312.004993, 2023.
David Warde-Farley, Tom Van de Wiele, Tejas D. Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. In In- ternational Conference on Learning Representations (ICLR), 2019.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40 (2):344-357, 2017. doi: 10.2514/1. G000192....
Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Physics-based character controllers using conditional vaes. ACM Transactions on Graphies, 41(4):96:1-96:1 2022.
18

---

Published as a conference paper at ICLR 2025
Philipp Wu, Arjun Majumdar, Kevin Stone, Yixin Lin, Igor Mordatch, Pieter Abbeel, and Aravind Rajeswaran. Masked trajectory models for prediction, representation, and control. Interna- tional Conference on Machine Learning (ICML), 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning (ICML) 2021.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonzalez Arenas, Hao-Ties Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. In Conference on Robot Learning (CoRL), 2023.
Chuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du, and Abhishek Gupta. Transferable reinforce- ment learning via generalized occupancy models. In Neural Information Processing Systems (NeurIPS), 2024.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang- Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalash- nikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Ku- mar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yev- gen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez. Arenas, and Kehang Han. RT-2: Vision-language-actioo models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.
19

---

Published as a conference paper at ICLR 2025
Appendices
Appendix A
Related Work
20
Appendix Algorithmic details
23
Appendix C
Experimental Details for the Humanoid Environment
23
C.1 The SMPL MuloCo Model
23
C.2 Data
23
C33 Tasks and Metrics
23
C.4 Training Protocols
27
C.5 Algorithms Implementation and Parameters
27
Appendix D Additional Experimental Results
35
D.I Detailed Results
35
D.2 Ablations
D.3 Diversity, Dataset Coverage and Transitions
42
D.4 Qualitative Evaluation
44
Appendix E
Ablations on Bipedal Walker
50
Appendix Ablations on AntMaze
51
Appendix G
Understanding the Behavioral Latent Space
52
G.1
Dimensionality Reduction of the Behavioral Latent Space
52
G.2 Behavior Interpolation
52
Appendix H
Comparison to Unsupervised Skill Discovery Methods
54
A RELATED WORK
RL for Humanoid Control. Controlling humanoid agent is considered a major objective for both in robotic (UniTree, 2024; Dynamics, 2024) and simulated (Peng et al., 2021; Won et al., 2022; Luo et al., 2024a) domains and it has emerged as major challenge for reinforcement learning due to its high dimensionality and intrinsic instability. In robotics, predominant approach is to perform direct behavior cloning of task-specific demonstrations (e.g., Seo et al., 2023) or combing imitation and reinforcement learning (RL) to regularize task-driven policies by using human-lite priors (e.g., Cheng et al., 2024). virtual domains, RL is often used for physics-based character animation by leveraging motion-capture datasets to perform motion tracking (Luo et al., 2023; Merel et al., 2019; Wagener et al., 2022: Reda et al., 2023) or to learn policies solving specific tasks, such as locomotion
manipulation (Luo et al., 2024c; Wang et al., 2023; Hansen al., 2024a). Despite its popularity across different research communities, no well-established platform, data, or benchmark for multi-task whole-body humanoid control is available. Standard simulation platforms such as di_ cont rol (Tunyasuvunakool et al., 2020) or IsaacGym (Makoviychuk et al., 2021) employ different humanoid skeletons and propose only handful of reward-bard based tasks. Luo et al.
20

---

Published as a conference paper at ICLR 2025
(2024c) and Sferrazza et al. (2024) recently introduced broader suite of humanoid tasks, but they all require task-specifie observations to include object interaction and world navigation. Regarding datasets, MoCapAct Wagener al. (2022) relies on CMU motion capture data mapped onto CMU humanoid skeleton, Peng et al. (2022) uses a well curated animation dataset related to a few specific movements mapped onto the IsaacGym humanoid, and Luo et al. (2023) use the AMASS dataset mapped to an SMPL skeleton.
Unsupervised RL. Pre-trained unsupervised representations from interaction data (Yarats et al., 2021; Schwarzer et al., 2021; Farebrother et al., 2023) or passive data (Baker et al., 2022; Ma et al., 2023: Brandfonbrener et al., 2023; Ghosh et al., 2023). such as unlabeled videos, significantly reduce the sample complexity and improve performance in solving downstream tasks such as goal-based, reward-based, or imitation learning by providing effective state embeddings that simplify observa- tions (e.g., image-based RI. and capture the dynamical features of the dynamics. Another option is to pre-train a set of policies through skill diversity metrics (e.g. Gregor et al., 2016; Eysenbach et al., 2019; Sharma et al., 2020; Laskin et al., 2022; Klissarov & Machado, 2023; Park et al., 2024c) or exploration-driven metrics (e.g. Pathak et al., 2017; Machado et al., 2020; Mendonca et al., 2021; Rajeswar et al., 2023) that can serve behavior priors. While both pre-trained representations and policies can greatly reduce sample complexity and improve performance, a full RL model still needs to be trained from scratch solve any downstream task.
Zero-shot RL. Goal-conditioned methods (Andrychowicz et al., 2017: Pong et al., 2020; Warde- Farley et al., 2019; Mezghani al., 2022; Ma et al., 2022; Park et al., 2023) train goal-conditioned policies to reach any goal state from any other state. While they are the most classical form of zero- shot RL, they are limited learn goal-reaching behaviors. Successor features based methods are the most related to our approach. They achieve zero-shot capabilities by modeling a discounted sum of state features learned via low-rank decomposition (Touati & Ollivier, 2021; Touati et al., 2023; Pirotta et al., 2024; Jeen et al., 2024) or Hilbert representation (Park et al., 2024b). One of the key advantages of these methods is their low inference complexity, as they can infer near-optimal pol- icy for given task through a simple regression problem. Generalized occupancy models (Zhu et al., 2024) Iearn a distribution of successor features but requires planning for solving novel downstream tasks. Building general world models is another popular technique (Yu et al., 2023; Ding et al., 2024; Jiang et al., 2024) for zero-shot RL when combined with search/planning algorithms (e.g. Williams et al., 2017: Howell et al., 2022). While this category hold the promise of being zero-shot, sev- eral successful world-modeling algorithms uses a lask- training to obtain the best downstream task performance (Hansen et al., 2024b;a; Hafner et al., 2024; Sikchi et al., 2022). Finally, recent works (Frans et al., 2024; Ingebrand et al., 2024) have achieved zero-shot capabilities by learning an encoding of reward function at pre-train time by generating random unsupervised rewards.
Integrating demonstrations. Our method is related to the vast literature of learning from demon- strations. Transformer-based approaches have became a popular solution for integrating expert demonstrations in the learning process. The simplest solution is to pre-train a model through condi- tioned or masked behavioral cloning (Cui et al., 2023; Shafiullah et al., 2022; Schmidhuber, 2019; Chen et al., 2021; Liu et al., 2022; Wu et al., 2023; Jiang et al., 2023). If provided with sufficiently curated expert datasets at pre-training. these models can be prompted with different information (e.g., state, reward, etc) to solve various downstream tasks. While these models are used in purely generative way, H-GAP (Jiang et al., 2024) combines them with model predictive control to optimize policies that solve downstream tasks. Similar works leverage diffusion models an alternative to transformer architectures for conditioned trajectory generation (e.g.. Pearce et al., 2023; He et al., 2023) or solve downstream tasks via planning (Janner al., 2022). Another popular approach is to rely on discriminator- based techniques to integrate demonstrations into an RL model either for imitation (e.g., Ho & Ermon, 2016; Ding et al., 2019; Tessler al., 2023), reward-driven (hierarchi- cal) tasks (Peng et al., 2021: Gehring et al., 2021: 2023; Vlastelica et al., 2024) or zero-shot (Peng al., 2022)². When the demonstrations are of "good" quality, the demonstrated behaviors can be distilled into the learned policies by constructing one-step tracking problem (c.g., Luo et al., 2023; 2024b; Qian et al., 2024). These skills can be then used as behavior priors to train task-oriented controllers using hierarchical RL. Finally, recent papers leverage internet-scale data to learn general controllers for video games or robotic control. These methods leverage curated data with action
While the original ASE algorithm is designed to create behavior priors that are then used in hierarchi- cal RL routine, we show in our experiments that it is possible to leverage the learned discriminator to solve downstream tasks in zero-shot manner.
21

---

Published as a conference paper at ICLR 2025
labeling (Wang et al., 2024; Team al., 2024; Zitkovich et al., 2023) the existence of high-level API for low-level control (Zitkovich et al., 2023).
Algorithm FB-CPR
Inputs: unlabeled dataset M. Polyak coefficient Ç, number of parallel networks wi, randomly initialized networks (Fes he(m)- B.. Ta. (Q ]k[[m] Dy. learning rate &, batch size n, B regularization coeffi- cient X, Fz-regularization coefficient 3. actor regularization coefficient number of rollouts per update Niollouts, rollout length Tollout, sampling distribution y (Vonline: Munlabeled)) sequenee length Tseq
relabeling probability Prelabel
2: Initialize empty train buffer: Dontine + 0
3: for 1,... do
4: 10 Rollout
5: for i 1,..., Nrolloute do
((s)
where Donlines where (si,..., 8Tan M,
with prob Vonline
6: Sample
Eem B(s:)
with prob Tunlabeled with prob Tonline
~ Ia)
Vd [=|la 8: Rollout Ta(:; 2) for Trollot steps, and store data into Daaan
7:
9:
end for
10: Sampling
11:
Sample a mini-batche of n transitions {(s,, o,, s/s 2))== from Douline
Sample a mini-batch of =h sequences {(5,11 8j...... ,i,TT,
Tann from M
Tayy
1e Encode Expert sequences
14:
2j Tuee Eter ((sja) Vd
15:
10 Compute discriminator loss
16:
L discriminssor (v)) = - 1 - و Tang log 2,) Σ. log(1 - (st, 21))
17:
/ Sampling and Relabeling latent variables (no relabel)
with prob 1 Prelabel with prob Protabel Tonline with prob Prelabel Tunlabeled with prob Prelabel * (1 Tonline Tunlabeled
(ss))
where & [((1)))
18:
Set E [1],
They
B(sa,)
where j [((2
Tamq tel
~ N0, Ia)
19:
Compute loss
20:
Sample di ~ To (8, 2i) for all [n]
21:
Loa (02, w) 1m End Foz a.) Bs())) - Eiema (o, (s4, 24) BB())))
22:
Σ, Fo (ss, ai 2) (s [m]
23:
1e Compute orthonormalice regularization loss
24:
Lortho (w) 1 E,,, (o)) Ba(()))² E. (8) BC (x))
25:
1a Compute Fz-regularization loss
(0x) = Exe|u] Fr, (s1, as, T Bu(s - mineepa Fe, (5t, 21) Ta)) 2)²² .kk
26:
27:
1a Compute critic loss Compute discriminator reward: r, log(Ds( =1) log(1 Dw Vi [n]
28:
29:
Leritie (mx) = Eien (Qm, (st, 2.) rs min Qm (5), aiy 21)) Wk (m]
30:
1e Compute actor loss
31:
Sample af To(s,, 21) for all [m]
32:
Let F < stopgrad R En minue[m] Fo, (si, af =1) =2)
33:
Lactor (0) - ΣR -
2.) (mimegm) (m,, 2.) z af minuelm Jay (st, (f),a{
34:
1 Update all networks
35:
s + 1 (V..Lassariinan (v)
36:
O + 02 - -Va, (.Ea(OO + (..... for all lo E [m]
37:
{Va(Ziemm) (r) (w))
38:
< N {V mz (Tr (Ta E [m]]
39:
end
22

---

Published as a conference paper at ICLR 2025
B ALGORITHMIC DETAILS
In Alg. I we provide a detailed pseudo-code of FB-CPR including how all losses are computed. Following Touati et al. (2023), we add two regularization losses to improve FB training: an or- thonormality loss pushing the covariance Σg ( of B towards the identity, and temporal difference loss pushing F(s,, a, z) toward the action-value function of the correspond- ing reward B(s) Σn² z. The former is helpful to make sure that B is well-conditioned and does not collapse, while the latter makes F spend more capacity on the directions in z space that matter for policy optimization.
C EXPERIMENTAL DETAILS FOR THE HUMANOID ENVIRONMENT
C.I. THE SMPL MUJOCo MODEL
Our implementation of the humanoid agent is build on the MuJoCo model for SMPL humanoid by Luo (2023). Previous work in this domain considers unconstrained joint and over-actuated con- trollers with the objective of perfectly matching any behavior in motion datasets and then the learned policies as frozen behavioral priors to perform hierarchical RL (e.g., Luo et al., 2024b). Unfortunately, this approach strongly relies on motion tracking as the only modality to extract behaviors and it often leads to simulation instabilities during training. Instead, we refined the agent specification and designed more natural joint ranges and PD controllers by building on the cmm ca__oottro (Tunyasuvunakool et al., 2020) CMU humanoid definition and successive iterations based on qualitative evaluation. While this does not prevent the agent to express non-natural behav- iors (see e.g., policies optimized purely by reward maximization), it does provide more stability and defines a more reasonable control space. We will release the full agent specification and environment code later time for full reproducibility.
C.2 DATA
The AMASS dataset (Mahmood et al., 2019) unifies 15 different motion capture datasets into single SMPL-based dataset (Loper et al., 2015). For our purposes, we only consider the kinematic aspects of the dataset and ignore the full meshed body reconstruction. In order to enable the compar- ison to algorithms that require action-labele to (Wagener et al., 2022) and train a single instance of Goal-GAIL to accurately match each motion in the dataset and then roll out the learned policies to generate a dataset of trajectories with ac- tions. The resulting dataset, named AMASS-Act, contains as many motions the original AMASS dataset.
demonstration datasets, we follow a similar procedure
As mentioned in the main paper, we select only a subset of the AMASS (AMASS-Acu dataset. Following previous approaches (e.g., Luo et al., 2021; 2023; 2024b), we removed motions involving interactions with objects (e.g., stepping on boxes). We also sub-sampled the BMLhandball dataset to just 50 motions since it contains many redundant behaviors. Finally, we removed two dataset S SSM_synced and TCD. We report several statistics about the datasets Tab. 2.
C.3 TASKS AND METRICS
In this section we provide a complete description of the tasks and metrics.
C.3.1 REWARD-BASED EVALUATION
Similarly to (Tunyasuvunakool et al., 2020), rewards are defined as a function of next state and optionally action and are normalized, i.e., the reward range is [o, 1). Here provide a high level description of the 8 categories of rewards, we refer the reader to the code (that we aim to release after the submission) for details.
23

---

Published as a conference paper at ICLR 2025
Datnset
M Total Steps
Must Total Steps
Motion count
Average longth
Total Time (s)
Motion count
Average longth
Total Time (5)
ACCAD
223
189,00
42146
1404.87 87
25
174.48
4362
145.40
BMLmovi
1456
167.36
243683
8122.7 77
162
165. 95
26855
896.27
BioMotionLat
1445
348 88
504134
16804 47
161
266, 89
42969
1432,30
DFaust
179.39
14351
478.37
9
134,67
1212
40.40
Donce138
1768.91
401685
1356 17
2
835.00
1710
57.00
Eycs
862.41
4846777
16155
872.95
54123
HumanEva
25
540.68
13517
450.57
3
582.35
1747
58.23
KET
2858
235.56
673239
22441.25
318
232,09
73806
2460.20
V74.24
29
908
26349
878,
SFU
30
569.37
17081
569.35
3
849.67
2549
84.97
TocalCapture
2034,06
67124
2237
1715.50
6862
228.73
Transitions
2517
296a6m559
357m
Table AMASS statistics split into .M (train) and Mtest (test) datasets.
Locomotion. This category includes all the reward functions that require the agent to move a certain speed, in cer- tain direction and at certain height. The speed is the xy- linear velocity of the center mass of the kinematic subtree rooted at the chest. We require the velocity to lie in small band around the target velocity. The direction defined as an- gular displacement w.r.t. the robot facing direction, that is com- puted w.r.t. the chest body. We defined high and low tasks. In high locomotion tasks, constrain the head z-coordinate be above threshold, while in low tasks the agent is encour- aged to keep the pelvis z-coordinate inside a predetined range. Finally, we also includes a term penalizing high control ac- tions. To
We use the following name structure for tasks in this cat- [low-] -
egory: smpl_move
Standing. This category includes tasks that require tical stable position. Similarly to locomotion we defined standing "high" and "low". These two tasks ob- tained from locomotion tasks by setting the speed to 0 (i.e.. smplmove-ego- [low-]-0-0),
Handstand. This reverse standing position on the hands (i.e., smpl_handddeeee
To achieve this, the robot must place its feet and head above specific thresholds, with the feet being the highest point and the head being the lowest. Additionally, the robot's 's velocities and rotations should be zero, and control inputs should be minimal.
This is a common penalization used to avoid RL agents to learn rapid unnatural movements. Nonetheless, notice that FB-CPR leverages only state-based information for reward inference through B(s). This means that entirely rely the regularized pre-training learn to avoid high-speed movements.
24

---

Published as a conference paper at ICLR 2025
Arm raising. Similar to the previous category, this task quires the robot to maintain standing position while reach- ing specific vertical positions with its hands, measured at the wrist joints. We define three hand positions: Low (z- range: 0-0.8), Medium (z-range: 1.4-1.6), and High (z- range: 1.8 and above). The left and right hands are con- trolled independently, resulting in nine distinct tasks. Ad- ditionally, we incorporate penalty component for unneces- sary movements and high actions. These tasks are denoted as smpl r sea Seearms-(2 searss- (1eft. pos)-(risg) p
Rotation. The tasks in this category require the robot to achieve a specific angular velocity around one of the cardi- nal axes (x, y, or 2) while maintaining proper body align- ment. This aligmment component is crucial to prevent unwanted movement other directions. Similar to locomotion tasks, the robot must keep its angular velocity within narrow range of the target velocity, use minimal control inputs, and main- tain a minimum height above the ground, as measured by the pelvis z-coordinate. The tasks this category are denoted as smpl.rotate (aarratte(xxiss (speeed]))))))) {height).
Jump. The jump task is defined as reaching a target height with the head while maintaining sufficiently high vertical velocity. These tasks are named smpl-jump {height].
Ground poses. This category includes tasks that require the robot to achieve a stable position on the ground, such sitting, crouching. lying down, and splitting. The sit- ting task (smpl.sitonground) requires the robot's knees touch the ground, whereas crouching does not have this constraint. The lie-down task has two variants: facing upward (smpl weeollleeeggruudd-eee (smpl 1 lieongroound-down) split task, which is similar to sitting on the ground but re- quires the robot spread its feet apart by certain distance poplllppitt-disttanee
and facing downward Additionally,
define the
Crawl. The crawl task requires the agent to move across the floor in crawling position, maintaining specific target height at the spine link. Similar to locomotion tasks, the agent must move in its facing direction denoted as smpl.crawl-(hhighe (sppeei)))-ss-p-----(--- (ffcingg)f We provide two options for the agent's orientation: crawling while facing downwards (towards the floor) or upwards (towards the sky), with the latter being significantly more challenging.
desired speed. The crawl tasks
While suite allows to generate virtually infinite tasks, extracted 55 representative tasks for evaluation. See Tab. 18 and Tab. 19 for the complete list. We evaluate the performance of policy in solving the task via the cumulative return Esg Σ/ r(ar, $t41]] The initial distribution used in test a mixture between a random falling position and subset of the whole AMASS dataset, this is different from the distribution used training (see App. C.4).
episodes of H 300 steps:
25

---

Published as a conference paper at ICLR 2025
Figure 5: Examples of the poses used for goal-based evaluation.
C.3.2 MOTION TRACKING EVALUATION
This evaluation aims to assess the ability of the model to accurately replicate a motion, ideally by exactly matching the sequence of motion states. At the beginning of each episode, we initialize the agent in the first state of the motion and simulate many steps as the motion length. Similarly to (Luo et al., 2021; 2023), we use success to evaluate the ability of the agent replicate set of motions. Let M {r; i==1 the set of motions to track and denote by 7, the trajectory generated by agent 2 when asked to track T,. Then, given a threshold & = 0.5, we define
1 Σ success(M) M E < len(T:) dampl (87 To
S {e}
where sf is the state of trajectory T at step t, dampl (s, = [X, - [X', 8'1ll and [X, 0] is the subset of the state containing joint positions and rotations. This metric is very restrictive since it re- quires accurate alignment at each step. Unfortunately. exactly matching the motion at each time step may not be possible due discontinuities (the motion may flicker, i.e., joint position changes abruptly in a way that is not physical), physical constraints (the motion is not physically realizable by our robot), object interaction" 11 etc. We thus consider the Earth Mover's Distance (Rubner al., 2000, EMD) with dampl as an additional metric. EMD measures the cost of transforming one distribution into another. In our case, two trajectories that are slightly misaligned time may still be similar in EMD because the alignment cost is small, while the success metric may still be zero. While these metries capture different dimensions, if motions are accurately tracked on average, we expect low EMD) and high success rate.
C.3.3 GOAL-BASED BASED EVALUATION
The main challenge in defining goal-based problems for humanoid is to generate target poses that are attainable and (mostly) stable. For this reason, we have manually extracted 50 poses from the motion dataset, 38 from motions in the training dataset and 12 from motions in the test dataset, trying to cover poses involving different heights and different positions for the body parts. In Fig. report sample of 10 poses.
In order to assess how close the agent is to the target pose, we use dampt (s, s') as in tracking, where the distance is only measured between position and rotation variables. while velocity variables ignored. Let be the goal state obtained by setting positions and rotations to the desired pose and 2 be threshold parameter, and be margin parameter, we then define
velocities to
We curated our datasets but cannot exclude missed some non-realizable motion given that this process hand made.
26

---

Published as a conference paper at ICLR 2025
two evaluation metrics
[T(34 < 300 dampi ( S {
success Meoo
300
1
d(dampl (st, ()t,g) (fddmmp((t)))) |
proximity = Pesess
300
+I{ dampl (8t, > 8 A dampl (t, (8t, S B + o - dammpp((s g) }}}}
The success metric matches the standard shortest-path metric, where the problem is solved as as the agent reaches state that is close enough to the goal. The proximity metric is computing a "soft" average distance across the full episode of 300 steps. The "score" for each step distance is within the threshold B, while it decreases linearly down to 0 when the current state is further than 3 + a from the goal. Finally, the metrics are averaged over multiple episodes when starting from initial states randomly sampled from Ptest.
the
When evaluating FB-CPR. CALM, ASE, and GOAL-GAIL, we need to pass full goal state g. which includes the zero-velocity variables. On the other hand, PHC and GOAL-TD3 are directly trained match only the position and rotation part of the goal state. Finally, for both MPPI and TD3 directly optimizing for the distance to the pose (i.e., no velocity) led to the better results.
C.4 TRAINING PROTOCOLS
In this section we provide description of the training protocol, you can refer the next section for algorithm dependent details. We have two train protocols depending on whether the algorithm is trained online or offline.
Online training. The agent interacts with the environment via episodes of fix length H 300 steps. We simulate 50 parallel (and independent) environments at each step. The algorithm has also access to the dataset M containing observation-only motions. The initial state distribution of an episode is a mixture between randomly generated falling positions (named "Fall" initialization) and states in M (named "MoCap" initialization 2) We select the "Fall" modality with probability 0.2. For "MoCap", uniformly sampled. We change the prioritization during training based on the ability of the agent to track motions. Every 1M interaction steps, evaluate the tracking performance of the agent all the motions in M and update the priorities based on the following scheme. We clip the EMD in [0.5, 5] and construct bins of length 0.5. This leads to 10 bins. Let b(m) the bin to which motion m is mapped to and (b(m)] the cardinality of the bin. Then,
prioritization sample motions from MM and, inside motion, the state is
Vm E Derain:
priority(m)
[b(m)]
We train all the agents for 3M gradient steps corresponding to 30M environment steps. The only exception is PHC where we had to change the update/step ratio and run 300M steps to achieve 3M gradient steps (we also updated the priorities every 10M steps instead of 1M).
Offline training. Offline algorithms (i.e., Diffuser and H-GAP) require a dataset label with actions and sufficiently diverse. We thus decided to use a combination of the in-house generated AMASS- Act and the replay buffer of trained FB-CPR agent. We selected the same motions in M from the AMASS-Acu dataset. The FB-CPR replay buffer corresponds to the buffer the agent after being trained for 30M environment steps. The resulting dataset contains about 8. 8.1M 1M transitions.
C.5 AL.GORITHMS IMPLEMENTATION AND PARAMETERS
In this section, we describe how each considered algorithm was implemented and the hyperparame- ters used to obtain the results of Tab. 1.
We use both velocity and position information for the initialization.
27

---

Published as a conference paper at ICLR 2025
C.5.1 SHARED CONFIGURATIONS
We first report some configurations shared across multiple algorithms, unless otherwise stated in each section below.
General training parameters. We use replay buffer of capacity 5M transitions and update agents by sampling mini-batches of 1024 transitions. Algorithms that need trajectories from the unlabeled dataset sample segments of these of length 8 steps. During online training, we interleave rollout phase, where we collect 500 transitions across 50 parallel environments, with a model update phase, where update each network 50 times. During rollouts of latent- or goal-conditioned agents, we store into the online buffer transitions (s, a, s', 2), where is the latent parameter of the policy that generated the corresponding trajectory. To make off off-policy training of all networks (except for discriminators) more efficient, we sample mini-batches containing (s, a, 2 from the online buffer but relabel each with randomly -egnerrtee one from the corresponding distribution v with some "relabeling probability" (reported in the tables below).
All algorithms keep the running mean and standard deviation of states in batches sampled from the online buffer and the unlabeled dataset at each update. These are used to normalize states before feeding them into each network. Unless otherwise stated we use the Adam optimizer (Kingma & Ba, 2015) with (31, 82) (0.9, 0.999) and 10
Table 3: Summary of general traning parameters.
Hyperparameter
Value
Number of environment steps
Number of parallel envirouments
50
Number of rollout steps between each agent update
500
Number of gradient steps per agent update
50
Number of initial steps with random actions
50000
Replay buffer
Batch size
1024
Discount factor
0.98
We report also the parameters used for motion prioritization.
Table 4: Summary of prioritization parameters.
Hyperparameter
Value
Update priorities every N enviromment steps
IM
EMD clip
[0.5, 5]
Bin width
0.5
Network architectures. All networks are MLPs with ReLU activations, except for the first hidden layer which uses a layernorm followed by tanh. Each z-conditioned network has two initial "em- bedding layers", one processing (s, 2). and the other processing s alone (or s and a). The second embedding layer has half the hidden units of the first layer, and their outputs are concatenated and fed into the main MLP. On the other hand, networks that do not depend on directly concatenate all inputs and feed them into a simple MLP. The shared parameters used for these two architectures are reported in the table below. Each actor network outputs the mean of a Gaussian distribution with fixed standard deviation 0.2.
Table 5: Hyperparameters used for the "simple MIP" architectures.
Hyperparameter
critics
actors
embeddings
Input variables
(8,a) a)
Hidden layers
1
Hidden units
1024
1024
256
Activations
ReLU
ReLU
ReLU
layer activation
layemorm lincar
layeroom
layemorm + tanh 12-nomaaizaatin
Output activation
tanh
Number of parallel networks
28

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters used for the architectures with embedding layers.
Hyperparameter
critics (e.g. F.
actors
Inpur variables
(8, R, 8)
(6, #)
Embeddings
over (s, and one over (s,
over (+) and over (e,
Embedding hidden layers
Embedding hidden units
1024
1024
Embedding output dim
$12
512
Hidden layers
Hidden units
1024
1024
Activations
ReLU
ReLU
First-layer activation
layernorm tanh
layernorm tanh
Ouiput activation
linear
lnnh
Number of parallel networks
The discriminator an MLP with hidden layers of 1024 hidden units, each with ReLU activations except for the first hidden layer which uses a layernorm followed by tanh. It takes as input a state observation $ and a latent variable 2, and has a sigmoidal unit at the output. It is trained by minimizing the standard cross-entropy loss with learning rate of 10-5 regularized by the gradient penalty used in Wasserstein GANs (Gulrajani et al., 2017) with coefficient 10. Note that this is different gradient penalty than the used by Peng et al. (2022); Tessler et al. (2023). We provide an in depth ablation into the choice of gradient penalty in App. D.2.
Discriminator.
Table 7: Hyperparameters used for the discriminator.
Hyperparameter
FB CPR
CALM
ASE
Goil
Input variables
[s,:)
(+.g)
Hidden layers
3
Hidden units
1024
1024
1024
1024
Activations
ReLU
Ourput activation
sigmoid
sigmoid
sigmoid
sigmoid
WGAN gradient penalty coefficient
10
10
Learning
10-5
10-5
10-6
10-5
C.5.2 TD3
We follow the original implementation of algorithm by Fujimoto et al. (2018), except that we re- place the minimum operator over target networks to compute the TD targets and the actor loss by penalization wrt the absolute difference between the Q functions in the ensemble, proposed by Cetin et al. (2024a). This penalty is used in the actor and the critic of all TD3-based algorithms, with the coefficients reported in the tables below. Note that will report only the values O, for which the target is the average of the Q networks in the ensemble, and 0.5, for which the target is the minimum of these networks.
Table 8: Hyperparameters used for TD3 training.
Hyperparameter
Value Tab. See Tah 4
General training parameters
General prioritization parameters
actor network
third column Tab. 5, output action
critic network
second column of Tab. 5, output dim 10-4
1 Learning rate for actor
Learning rate for critie
10
Polyak cocfticient for target network update
0.005
Actor penally cuefficient
0
Critic penalty coefficient
C.5.3 FB-CPR
The algorithm is implemented following the pseudocode App. B. The values of its hyperparameters are reported in the table below.
Inference methods. For reward-based inference, use weighted regression method 2, oc Ege (())) where we estimate the expectation with 100k samples from the online buffer. We found this work better than standard regression, likely due to the high
29

---

We implemented an off-policy version of ASE to be consistent with the training protocol FB- CPR. In particular, we use a TD3-based scheme optimize all networks instead of PPO as in the original implementation of Peng et al. (2022). As for FB-CPR, we fit a critic to predict the expected discounted sum of rewards from the discriminator by temporal difference (see Eq. 10), and another critic to predict E )*0(( ((tt1)) 2/s, a, æz), where ó is the representation learned by the DIAYN- based (Eysenbach et al., 2019) skill discovery part of the algorithm. We train such representation by an off-policy version of Eq. 13 in (Peng al., 2022), where we sample couples (s', =) from the 2]. Note that this is consistent with the original Published as a conference paper at ICLR 2025
diversity of behaviors present in the data. For goal-based inference, the original method Zg B(g), while for motion tracking a motion 7 we infer one for each time step in the +++++ B(s;), where sj is the j-th state in the motion and L is the same encoding sequence length used during pre-training.
motion o 2 jj=t+1
Table 9: Hyperparameters used for FB-CPR pretraining,
Hyperparameter
Value
General training parameters
See Tah. 3
General prioritization parametens
See Tab 4
Sequeace length for trajectory sampling from D
8
update frequency during rollouts
once every 150 steps
dimension
256
Regulatization coeflicient
0.01
netwoek
6, output
actor network
rhird column of Tah. 6, output dim action dim
critic network
sccond column 6, output
B network
fourth column Tab. 5, output dim 256
Discriminator
Tab: 7
Learning rate
Learning rate
10-4
Learning rale for critic
10-4
Learning rate for B
10-5
Cocflicient for orihonormality loss distribution
100
-encoding of unlabeled trjeectories
60%
-goals from the online buffer
20%
-uniform on unit sphere
20%
Probubility relabeling
Polyak coefficient for target network update
0.005
FB penally coetlicient
Actor penulty coefficient
Critic penalty coefficient
0.5
Coefficient F2-rouularization loss
C.5.4 ASE
online buffer and maximize "e) off- off-policy implementation of DIAYN (Eysenbach et al., 2019). The output of á is normalized on the hypersphere of radius Vd. We also add othornormality loss (same as the used by FB) found this to be essential for preventing collapse of the encoder.
Inference methods. For reward-bacc and goal- boal-based inference CPR, with replaced with ó. For tracking we use 2, x B(s++)) for each timestep in the target motion.
the same methods FB-
30

---

Published as a conference paper at ICLR 2025
Table 10: Hyperparameters used for ASE pretraining.
Hyperparameter
Value
General training parameters
General prioritization parameters
See Tab.
update frequency during rollouts
once every 150 steps
dimension
64
Regularization coefficient
0.01
actor network
third column of output action
critic networks
second column of Tab. 6, output dim I
& encoder network
fourth column Tab. 5, output 64
Discriminator
Tab
Learning rale for actor
10
Learning rate for critic Learning rate for
10
10
Coeflicient for uribonormality loss distriburion
100
-goals from unlabeled dataset
60%
-geals from the online buffer
20%
-uniform
sphere
20%
Probahility of relabeling
0.8
Polyak coeflicient for target network update
0.005
Coefficient for diversity loss (Eq. 15 in (Peng al., 2022))
Actor penalty coefficient
0.5
Critic penalty coefficient
C.5.5 CALM
As for ASE, we implemented an off-policy TD3-based version of CALM to be consistent with the training protocol of FB-CPR. We critic Q(s,, a, to predict the expected discounted sum of rewards from the discriminator by temporal difference (see Eq. 10). We also train sequence encoder ((T) which embeds a sub-trajectory T from unlabeled dataset into space through a transformer. The encoder and the actor trained end-to-end by maximizing Q(s, TTs, 2 o(T)), ②(T), plus the constrastive regularization loss designed to prevent the encoder from collapsing (Eq, 5,6 in (Tessler al., 2023)). The transformer interleaves attention and ferd-forward blocks. The former uses a layernorm followed by multi-head self-attention plus a residual connection, while the latter uses a layernorm followed by linear layers interleaved by GELU activation. Its output is normalized the hypersphere of radius Vd.
Inference methods. We use the same methods as FB-CPR for goal-based and tracking inference.
Table 11: Hyperparameters used for CALM pretraining.
Hyperparameter
Value
General training parameters
See Tub. 3
General prioritization paramelers
See Tab. 4
Sequence length for trajectory sampling from D
update frequency during rollouts
once every 150 steps 256
dimension
actor network
Uhid column of Tab. 6, output dim action dim
critic
6, output
encoder network
transformer (see text above)
attention blocks
2
emmeedding dim
256
-MLP first linear layer
256x 1024
-MLP
layer
1024x2266
Discriminator
Tub. 7
Learning rate for actor
Learning rate for critic
Learning rate
10-7
Coefficient for construstive loss
0.1
distribution
-cacoding of unlabeled trajectories -guals from online bufler
100%
-uniform
sphere
Probability of relabeling 2s
Polyak coefficient for target network update
Actor penalty coefficient
0.5
Critic penalty coefficient
31

---

Published as a conference paper at ICLR 2025
C.5.6 PHC
PHC is similar to goal-conditioned algorithm except that the goal is "forced" to be the next state in the motion. This makes PHC an algorithm specifically designed for one-step tracking. We use TD3-based variant of the original implementation (Luo et al., 2023). Concretely the implementation is exactly the same of TD3 but we changed the underlying environment. In this tracking environment the state is defined as the concafenation of the current state and the state y to track. The resulting sample motion from the motion set
state space is R716, At the beginning of episode, (either M or Drest) and we initialize the agent to a randomly selected state of the motion. Let f being the randomly selected initial step of the motion, then at any episode step & E [1, len(m) f f-1] 1] the target state It correspond to the motion state m We use the negative distance in position/orientatioon reward function, i.e., r((s, a. (s',g')
dampi s').
Inference methods. By being a goal-conditioned algorithm we just need to pass the desired goal as target reference and can be evaluated for goal and tracking tasks.
Table 12: Hyperparameters used for PHC pretraining.
Hyperparameter
Value
General training parameters
See Tab 3
General prioritization parameters
Update priorities every N environment steps
10M
Number enviromment stops
300MM
Number of gradient steps per agent update
TD3 configuration
See Tab 8
C.5.7 GOAL-GAIL
We use a TD3-based variant of the original implementation (Ding et al., 2019). Concretely, the implementation is very similar to the one of CALM, except that there is no trajectory encoder and the discriminator directly receives couples (s, g). where g is a goal state sampled from the online buffer or the unlabeled dataset. In particular, the negative pairs (s,g) g) for updating the discriminator are sampled uniformly from the online buffer (where g is the goal that was targeted when rolling out the policy that generated s), while the positive pairs are obtained by sampling sub-trajectory of length 8 from the unlabeled dataset and taking g as the last state and s as another random state. Similarly to CALM, we train goal- conditooid critic Q(s, a, g) to predict the expected discounted sum of discriminator rewards, and an coal-conditioned actor x (8, g) to maximize the predictions of such critic.
Inference methods. We use the same methods as ASE for goal-based and tracking inference.
Table 13: Hyperparameters used for GOAL-GAIL, DorL--Gil
Hyperparameter
Value See Tab. 3 See Tah. 4
General training parameters
General prioritization parameters
Sequenee length trajectory sampling from goal update frequency during rollouts
once every 150 steps
actor network
third column 6, cutpur action
critic network
secend column of Tab. 6, output dim
Diseriminntor
Tab. 7
Learning rale actor
I Learning rate for critic
10
goal sampling distribation
-goals
unlabeled dataset
50%
-goals from the online buffer
50%
Probubility of relabeling zs
Polyak coeflicient For target network update
0.005
Acior penalty coe eeficientt
Critic penalty coefficient
0.5
C.5.8 GOAL-TD3
We closely follow the implementation of Pirotta et al. (2024). For reaching each goal g. we use the reward function r(s',g) -llpos(s') pos(g)ll2, where pos(:) extracts only the position
32

---

Published as a conference paper at ICLR 2025
of each joint, ignoring their velocities. We then train gond-conni TD3 agent to optimize such a reward for all 9. We sample a percentage of training goals from the unlabeled dataset, and a percentage using hindsight experience replay (HER, Andrychowicz al., 2017) on trajectories from the online buffer.
Inference methods. We use the same methods as ASE for goal-based and tracking inference.
Table 14: Hyperparameters used for GOAL-TD3 pretraining.
Hyperparameter
Value
General training parameters
See Tab. 3 See Tab 4
General prioritization parameters
Sequence length for HER sampling
goal update frequency during rollouts actor network
every 150 steps third column of Tab. 6, output dim action dim
critic network
secoad column 6, outpur dim
I Learning rate for actor Learning rate for critic goal sampling distribution
10
10
"gouls from the unlabeled dataset
100%
-gaal from the online huffer (HER)
0%
Probability of relabeling
0.5
Polyak coefficicnt for target network update
0.005
Actor penalty coefficient
Critic penalty cocfficient
0,5
C.5.9 MPPI
We use MPPI with the real dynamic and real reward function for each task. For each evaluation state, action plans are sampled according to a factorized Gaussian distribution. Initially, mean and standard variation of the Gaussian are set with O and 1, respectively. actions plans are evaluated by deploying them in the real dynamics and computed the cumulative return over some planning horizon. Subsequently, the Gaussian parameters are updated using the top-k most rewarding plans. For goal-reaching tasks, we use the reward r(s', g) (spp pos(g))l2
Table 15: Hyperparameters used for MPPI planning.
Hyperparameter
Value
Number plans
256
Planning botizon
32 for reward-based tasks, 8 for goals
for the top-k
Maximum of standard deviation
2
Minimum of standard deviation
0.2
Temperature
I
Number of optimization steps
10
C.5.10 DIFFUSER
We train Diffuser offline on FB-CPR replay buffer and AMASS-Act dataset described in C.4. We follow the original implementation in Janner et al. (2022). We use diffusion probabilistic model to learn generative model over sequence of state-action pairs. Diffusion employs forward diffusion process q(r'(r²-)) (typically pre-specified) to slowly corrupt the data by adding noise and learn [0. n] which induces the following data
a parametric reverse denoising process ((T'")(7)), Vi distribution:
II po po /
pa(79) = p(T")
d dr
(12)
where 70 denotes the real data and 7" is sampled from a standard Gaussian prior. The parametric models is trained using variationnal bound on the ligelikel objective Eyanp (log Pl ()()")). We use Temporal U-net architecture as Janner et al. (2022) for po-
At test time, we learn value function to predict the cumulative sum of reward given sequence T: (T) E(r 7d (rrr)))) To do that, relabel the offline dataset according the task's reward
33

---

Published as a conference paper at ICLR 2025
and we train Ry by regression on the same noise distribution used in the diffusion training:
( - -
27
1(7
Ro (a')
B E Bro
r(s))
(13)
We use then guiding sampling to solve the task by following the gradient of the value function Ri (r') each denoising step. For goal-reaching tasks, we condition the diffuser sampling by replacing the last state of the sampled sequence r' by the goal state after each diffusion steps. We sample several sequences and we select the one that maximizes the cumulative sum of the reward r(s'.) -pos(s') pos(g)]l2.
Table 16: Hyperparameters used for Diffuser pretraining and planning.
Hyperparameter
Value
Learning
Number of gradient steps
3 10
Sequence length
U-Net hidden dimension
1024
Number diffasion steps
50
Weight the action loss
Plamning horizon Gradient scale
32
Number of plans
128
Number guided steps
2
Number of guided-free denaising steps
4
C.5.11 H-GAP
We train the H-GAP model on the FB-CPR replay buffer and the AMASS dataset outlined in C.4. Following the methodology described in Jiang et al. (2024), we first train VQ-VAE on the dataset to discretize the state-action trajectories. Subsequently, we train a decoder-only Prior Transformer to model the latent codes autoregressively. In line with the procedures detailed in Jiang et al. (2024), we integrate H-GAP within a Model Predictive Control (MPC) framework. This integration involves employing top-p sampling to generate a set of probable latent trajectories, which were then decoded back into the original state-action space. At test time, we selected the most optimal trajectory based on the task task-specifie reward functions, assuming access these functions.
Table 17: Hyperparameters used for H-GAP.
Hyperparameter
Valuc
batch size
128
training steps
10
Modeling horizon
32
VQ- VAE chunk size
VQ. VAE code per chunk
32
VQ- VAE number of code
512
VQ- VAE learning rale
x
VQ. VAE number of heals
VQ- VAE number layers
Prior Transformer number of heads
10
Prioe Transformer number of layers
10
Prior Transforner learning rate
3 10
34

---

Published as a conference paper at ICLR 2025
Tack
MPPI
ASE
Norm.
Noemalized
Nomalize
Nonsalized
275.08
200.33
0.74
227.27 (3,19)
0.83 (0.01)
260.03 (1.41)
0.97 (0.013
274.08 (1-48)
0.00)
novo-eoo--10 10o--ov-0000
273.67
2:0.12
0.91
116.50 (15. 56)
0.43 (0.06)
222. 14 (19.85)
0.51 (0.07)
215.61 (27.63)
0.70 0.110
handatand
251.3 30
3,58
0.01
5.21 (8.76)
0.02 (0.01)
15.04 (0.05)
0.58 (0.00)
41:27 (10.20)
0.11 (0.04)
20000-000--22
255.57
263.67
1,03
238.09 (5.79)
0.94
224.29 (00.55)
250. (5.21)
1.022 (0.02)
ego
1.03
179.82
0.74
211.65
0.57
novereeee (ow.-reoo------
253.45
200.71
2010 88 (7.00))
0.81 (0.00)
200.40 (9: 72)
(0.04
210030 (6.50)
0.83 (0.03)
nove-ego--- 90-4
245
260.29
137,80 (9.
0.36 (0.04)
143.12 (26-14)
0. 1.58
(9:35)
0.83 (0:01)
nove ego 00--2
253.77
262 62
1.003
207.27 (1.74)
0.82 (0022)
194.18 (6). 550
0.77 (0.25)
224 (3:15)
0.89 (0.04)
nove-ogo-500
247.45 49
251.61
1.00
132.03 (10.50)
0.54 (0.44)
131.14 (12.22)
0.54 (0,000
185. 60 (14.42)
0.75 (0.05)
1180-2
258.25 28
261.46
0.97
195. 45 (7
0.76 (0000)
237 73 Ga
0.92
227 54 (27
0.88 109
xove-o00-1880 1804
249.81
252.28
1.01
132,89
0.53
134.54
0.54
205.51 (14.40)
(0.06)
2009-6ç0-1ov--0
273.65
100.84 (01
0.57 (0,30)
(10011)
0.21 (0.04)
207 27 (3x.
(0.21)
nove wgo- Loy 20-2
270.89
266.74
0.59
80.23 (4.53
0,30 (0,02)
65.01 (4411))
0.24
221 37 (35 (35
0.82 (0.15)
zove-eco-lov 100-------22
259.97
267.52
1.003
98 (6.79)
0.37 (0.00)
38.71 (47.10)
0.25 (0.18)
222.5 81 (21.94)
0.86 (0.05)
nove-oço- 100-lov-100--
250.15
277.37
0.96
65.01 (7.75
0.25 (0.50)
13.77 (36 251
0.35 (0.06)
65. 20 (32.64)
0.23 00.12)
jump-2 2
90..83
67.45
0.74
16.8G (0.64)
0.17 (0.01)
8.73 80
0.30 (0.08)
34.88 (S.52)
0.38 (0.04)
0.77
8,31
(0.05)
0.30
7.42
210.20
0.90
(3.13)
(0.00)
(0.01)
2.20
(0.00)
rotate y.. 5- 0.0
272.11
270.84
1077-14
0.29 (0.60
124.52 (52.52)
217. (c3
15
-otate-y-5-3.
273.77 74
272.60 66
10
70 (10.00
0.36 (0.04)
14 48 (36 (2
0. 55 100
199.00 08 (5)) 75)
0.75 19h
-otare-s--- 5- 0..8
257.35 90
208.33 33
0.51
6.67 (1.50
000 (000)
0.39 (7.77)
0.00 (0.00)
35.23 (15.75)
0:37 (0.09)
2otato-a-S-3.-
266 16
206.55
0.78
5.63 (2.45)
0.02 (0.01)
0.01 (0000)
0.30 (0.00)
121. 95 (17.61)
0.47 (0,07)
s01seams-1-1
264 61
104.60
0.74
221.11
0.84
265. 15 (1)5
1.00
270 (1:37)
1.02
295.03
187.47 43
0.20
180.50
(0:00)
63.67
57.
0.37 (0.31)
sei3camm--1-1
268
11,05
0,15
67.44
(33.21)
0.33 (0.00)
258.00 (1.00)
243. (19.
(0.07)
cwl.0ammr m-
178.86
0.0G
765 120.84
116
0.43 (0.00)
70:06
(3 321
0.25 (0.13)
134. 83 (70.28)
(0.25)
267.55
/82
0.51
(32.04)
0.22 (0.04)
11.02 (0.14)
0.04 (0.00)
87. 27.25 (95. 421
0.33 (0.87)
264.
34.64
0.13
91.54
(8-02)
0.35 (0.00)
62.70 (1 O0
0.20 (0.01)
75.05 (99
0.28 (0
mi3 more-me--- 5-1
273.91
40.11
0.15
62.36
(9.57)
0.23 (0.55)
240 (22 960
0.58 (0.08)
167 98 (82.05)
0.61 (0.59)
xalseams--b-t
264 67
36.41
0.14
20 016
0.30
51.58
0.21
104 26
0.50 (0.31)
265.
(0.07)
(:2-05)
16
crouch-0
222.00
0.55
0.51 (0.60)
181.96
0 (0.22)
225 28 (25 17)
(0.19)
aitonground
271.76
243.64
0.90
61.18
(2.02)
0.23 (0053)
114.03 (57 201
0. 2. (0.21)
199. 64 (22 15)
0.73 (0.15)
1 Lieongeound-up up
278.00 665
240.31
0.89
29.05
(7.71)
0.10 (0.00)
204.26 (16.05)
0.733 (0.07)
155.. 06 (33. 18)
0.69 (0.12)
Lioonground-dooss
277.51
242.08
0.87
70.70
(30,52)
0.27 (0.04)
158. 10 (08.06)
0.57 (0.25)
193. 50 (18
0.70 (0,07)
276 13
260.66
0.91
104.29
(12.85)
0.35 (0.06)
112 (71 .22
0.41 (0.26)
252 18 (20.26)
0.84 (0.07)
279.22
263.28
27.28 (6.74)
12
0.45
0.95
(0.81)
0.07 (010)
(15-97)
cmoo 4--4-2--
257.01
60.50
0.21
1,81 (1:25)
0.1 (0.00)
0.03 (1.8))
0.01 (0.01)
12.02 (6.03)
(0.07)
cxoxi-C. 5-0mu
166.02
124.75
0.85
4.84 (3.67)
0.03 (0050)
77.72 (37 (I))
0.53 (3.00..0 (0.25)
101 92 (16.39)
0.70 (1.1
crowi--C 5-52--
234.51
60.16
0.35
1.77 (1:27)
0.01 (0:01)
3.97 (101)
0.02 (0.00)
15.61 (6.10)
0.07 (0.00
czavi-C. C-0d
145.79
112.27
0.77
27.44 (0.15)
0.19 (0.06)
20:32 (14.12))
0.14 (0.10)
191.75 (43.60)
1.32 (0,30)
exaxI=C, 4-2-d
289.55 55
105.70
0.57
4,00
0.01
15.50
0:35
19.00
0.07 (0.00)
146.46
0.7G
24.08 (3.74)
0.17
(200)2200
0.35
13 (64.97)
(0.64)
64.94
4.64
0,02 (0.01)
19: (955))
0.07
22.58 (5.31)
(0.02)
Average
289.74
178.50
0.72
85.27
0.33
108.73
0.41
151.68
0.61
Madian
26
206.59
0.83
80.33
0.30
77.46
0.41
19L.75
0.78
Table 18: Humanoid Environment. Average return per task for reward-optimizzatoon evaluation.
Group
Num.
TD3
MPFI
Diffuuer
ASB:
F16-UW9
Noemalized
Nommaliand
Noenalized
Zermalized
Stand
274.18
226.22 (21.55)
0.52 (8334)
172,80 (54.00
264.00 (21.94)
(0085)
(25.55)
Handstand
(0.30
3.21 (0.00)
0.04 (2.00)
00.000
41
251
Loce
271
Jump
140 05.009
67.45.000
(0.000
15 03.55-.00
0:17
8.73 (0.00)
0.30
34
03:28
Rocation
(28
216.3: 08
0.55 10)
75 43)
05.72
0:37 00:345
71 70
0.00
RaiscAms
367 08 (2.95)
05,46 (72.040
0,35 (0.27)
111,08 (40.67)
0.42 (1.18)
141.98 (MX2.75)
0.53 (0.25)
139.39 (57.20)
0.57 (0.25)
On-Gmmand
575 30 (3
913.51 (10.14)
03.88 (0.00)
02:06
(97 77)
0.23 (0 10
1500 70. 051 966
01:88 (0.730
195 (066770677.3000
0.71 (0.11)
Croil
210.77 (67.08)
06.03 (36 977
0.54 (0.20) 200
9.96 (B. 90
0.06 (1)
26. 16 (29) 15)
0.18 (0.23)
74.91 (12. c22
0.48 (6 (5)
Table 19: Humanoid Environment. Average return per category for reward-optimizzaion evaluation.
D ADDITIONAL EXPERIMENTAL RESULTS
In this section we report more detailed analysis of the experiments.
D.1 DETAILED RESULTS
In this section we report detailed results split across tasks.
Table 18 shows the average return for each reward-based task and Table 19 groups the results per task category.
Table 20 shows the proximity metric for each goal pose, while Table 21 shows the success rate.
Table 22 shows the train and test tracking performance for both EMD and success rate grouped over the AMASS datasets.
35

---

Published as a conference paper at ICLR 2025
Gool
TD3
MPPI
Difluser
GoIL
Goal-TD3
PHC
CALM
ASE
FB-CPR
Proximity
t.pose
0.59
0.21
0.80 (0.07)
0.98 (0.00)
0:99 (0.00)
0.24
0.33 (0.34)
0.98 (0:01)
0.599 (0.00)
t.pose. poose..ooer.arms
0.99
0.25
0.52 (0.04)
0.96 (0.05)
0.99 (0.00)
0.44 (0.04)
0.81 (0.17)
0.95 (0.06)
0.99 (0.00)
t.pope box head
0.99
0.23
0.60 (0.13)
0.98 (0.00)
0.99 (0.00)
0.21 (0.06)
0.63 (1.27)
0.52 (0.121
0.99 (0.00)
u.strotch...riiac
0.99
0.19
(0.12)
0.79 (0.17)
0.87 (0:07)
(0.01)
0.16 (0.14)
(0
0.70 (0.21)
u.stzetchlyy.o yleff
0.98
0.20
0.01 (0.01)
0.55 (0.11)
0.77 (0.0G)
0.02 (0.011
0. (0.20)
0.37 (0:225)
0.73 (0.18)
sustretchu7,7
0.59
0.28
0.02 (0.01)
0.96 (0.28)
0.81 (0. 14)
0.04 (0.00)
0.09 (0.14)
0.31 (0:23)
0.83 (0.10)
u.stretah.2.left
0.99
0.16
(0.09)
0.95 (0.04)
0.95
0.97 (0.03)
L. u.stzetch back
0.98
0.07
0.10 (0.111
0.81 (0.114
0.72 (0.17)
0.02 (0.013
0.01 (1.001)
0.76 (0.22)
0.53 (0.14)
L.ttretch...troot.poa
0.99
0.63
0.55 (0.13)
0.9L (0.07)
0:99 (0.00)
0.14 (0.02)
0.31 (0.20)
0.71 (0.16)
0.99 (0.00)
u.stzetchh Feo.eettel....ootttooo
0.98
0.98
(0.003)
0.84 (0.19)
0.90 (0.07)
(0.01)
(0.229)
(0.222)
0.95 (0.02)
crossed_arm
0.98
0.20
0.26 (0. 10)
0.801 (0.06)
0.86 (0.06)
0.02 (0.01)
0.14 (0.17)
0.56 (0.07)
0.89 (0.05)
scrazchinggheed
0.99
0.21
0 (0.14)
0.98 (0.00)
0:99 (0.01)
0,06 (0.02)
0:15 (0.25)
0.97 (0.01)
0.99 (0.00)
zich: hand wava
0.99
0.23
O. (0. (0.17)
0.92 (0.01)
0.95 (0.00)
0.12 (0.013
(0.200)
(0.022
0.95 (0.90)
K.stzoch
0.98
0.11
0.90 (0.08)
0.93 (0.05)
0.06 (0.02)
0. (0.14)
0.52 (0.13)
0.91 (0.05)
i.stzeoth
0.86
0.07
0.20
0.71 (0.07)
0.74
0.01 (0,00)
0.02
0.69 (0.001
0.88 (0.08)
armsust retic
0.98
0.08
(0.
0.58 (0.08)
0.72 (0. 14)
0.07 (0.01)
(0.10)
0.39 (0.
0.68 (0.06)
0.98
0.23
0.17 (0.07)
0.69 (0.09)
0.88 (0.08)
0.04 (0.02)
0.07 CO.
0.50 (0.08)
0.97 (0.04)
chest
0.98
0.15
0.17 (UU7))
0.92 (0.05)
0.99
0.04
0.16
0.95
0.98 (0.00)
pro,thoow
0.56
0.03
0.00 (0.00)
0.08 (0.07)
0.23 (0. 13)
0.04 (0.01)
0.00 (0.00)
0.02 (0.00)
0.08 (0.10)
agyptian
0.99
0.18
0.18 (0.08)
0.80 (0.11)
0.94 (0.0G)
0.12 (0.03)
0.22 (1.22)
0.60 (0:271
0.98 (0.00)
0.98
0.14
0.47 (0.00)
0.96 (0.03)
0:99
0.15
(0.05)
0.58 (0.00)
stand.aatttal.ass
0.99
0.41
0.11 (0.17)
0.94 (0.05)
0.99 (0.01)
0.05 (0.05)
0.34 (0.23)
0.94 (0.02)
0.58 (0.00)
packaboo
0.90
0.25
0.27 (0.12)
0.91 (0.10)
0.75 (0:20)
0.06 (0.03)
08.18 (1.233)
0.87 (0.15)
0.95 (0.34)
dancco
0.98
0.17
(0.06)
0.97 (0.02)
(0.04)
(0.24)
0.86 (0.16)
0.99 (0.00)
kneet
0.99
0.97
0.10 (0.07)
0.79 (0.12)
0.94 (0.05)
0.04 (0.00)
0:23 (1.300
0.34 (0.19)
0.95 (0.12)
crouch Jhigh
0.99
0.89
0.39 (0,05)
0.98 (0.00)
0.99 (0.00)
0.46 (0.05)
0.76 (0.18)
0.85 (0.12)
0.59 (0.00)
crouat.meddiu
0.99
0.95
(0.06)
0.98 (0.00)
(0.12)
0.99 (0.00)
crouch: Low
0.99
0.63
0.08 (0.003)
0.73 (0.201)
0.85 (0.059)
0.07 (0.003)
0.16 (1.15)
0.17 (0.111
0.85 (0.06)
equat.ppre_jun
0.98
0.9T
0.03 (0.01)
0.17 (0.13)
0:22 (0.20)
0,02
0.03 (0.05)
0.31 (0
0.56 (0.04)
0.98
0.77
(0,07)
0.72 (0.08)
0.93 (0.04)
0.02 (0.013
(0.25)
(9.39)
0.74 (0.10)
*ideJhiggheei
0.98
0.34
0.00 (0.00)
0.02 (0.02)
0.02 (0.01)
0.01 (0.01)
0.00 (0.
0.01 (0.01)
0 (0.05)
pro.trant.
0.99
0.33
0,01 (0.00)
0.54 (0.22)
0.75 (0:09)
0,06
0.08 (0.
0.20 (0.
0.69 (0.21)
arabosqus held Cooz
0.85
0.17
(0.03)
0:11 (0.06)
0.30 13:
(0.00)
(0.004
(0.02)
0.11 (0.05))
hold_rightt,
0.99
0.17
0.04 (0.05)
0.28 (0.11)
0.56 (0.20)
0.03 (0.01)
0.01 (0.103)
0.10 (0.07)
0.64 (0.12)
holdiarccrood
0.99
0.41
0.04 (0101)
0.51 (1.19)
0.76
0,20 (0,02)
0.29 (1.10)
0.17
0.72 (0.07)
benduon_]ee Jend _lg))
0.98
0.69
0.01 (0.00)
0.09 (0.10)
0.40 (0.06)
0.02 (0.01)
(0.08)
0.09 (0.05)
0.57 (0.12)
Lic.tront
0.97
0.87
0.16 (0. 16)
0:67 (0.11)
0.52 (0 08)
0.01 (0.00)
0.05 (0.04)
0.46 (0.
0.61 (0.10)
backward
0.98
0.92
0.13
0.38 (0.19)
0.37
0,00 (0.00)
0.01
0.03 (0.04)
0.13 (0.13)
0.97
0.79
0.07 (0.07)
0.15 (0.13)
0:03 (0.05)
0.02 (0.01)
(0.00)
0.09 (0.34)
0.04 (0.08)
1ic.sida
0.97
0.59
0,20 (0.08)
0.36 (0.18)
0.19 (0.11)
0.02 (0.01)
0.00 (1..0)
0.08 (0.05)
0.36 (0,34)
crunch
0.98
0.44
0.24 (0. (0.00)
0.00 (0.00)
0.04
0.01 (0.00)
0.00
0.00 (0.00)
Liebback
0.97
0.86
0.30 (0.28)
D.28 (O) 18)
0.05 (0.01)
0.19 (0.
0.54 23)
0.43 (0.22)
sit side
0.98
0.93
0.03 (0.01)
0.18 (0.10)
0.35 (0:17)
0.00 (0.00)
0.01 (1.03)
0.05 (0.
0.28 (0.17)
1it_2hatd_onnneee
0.98
0.97
0.29
0.42 (0.10)
0.53
0.00
0.04
0.04 (0.03)
0.59 (0.13)
sit.handbbhhnd
0.99
0.93
0.23 (0.16)
0.66 (0L018)
0.60 (0.11)
0.02 (0.02)
0.00 (1..0)
0.15 (0.16)
0.60 (0.11)
knwwes Jiw
0.98
0.92
0.38 (0.15)
0.71 (0.08)
0.83 (0.06)
0.03 (0.01)
0.18 (0.15)
0.46 (0.13)
0.73 (0.11)
bei .ge.tront
0.98
0.82
(0.
0.30 (0.41)
0.74
(0.02)
push up
0.97
0.59
0.01 (0.05)
0.35 (0.22)
D.16 (0.11)
0.01 (0.01)
(..01 (0.001
0,02 (0.02)
0.11 (0.95)
handal and
0.84
0.00
0.00 (0.00)
0.01 (0.01)
0.00 (0.00)
0.02 (0.01)
0.00 (0.00)
0.00 (0.00)
0.05 (0.34)
dan and Loht bent
0.96
0.05
(0.00)
0.00 (0.00)
0.00 (0.00)
(0.01)
(1.00)
(0.001
0.02 (0.02)
Average
0.385
0.61
0.47
0.08
Medias
0.98
0.31
0.17
0.70
0.77
0.04
0.11
0.46
0.74
Table 20: Humanoid Environment. Proximity over goal poses for goal-reaching evaluation.
We further mention results for two baselines that performed poorly in our tests. First, similarly to DIFFUSER, we tested H-GAP (Jiang et al., 2024) trained on the union of the AMASS-ACT dataset and FB-CPR replay buffer. Despite conducting extensive hyper-paramete search based on the default settings reported in Jiang et al. (2024) and scaling the model size, we encountered challenges in training an accurate Prior Transformer and we were unable to achieve satisfactory performance on the downstream tasks. We obtained an average normalized performance of 0.05 in reward optimization on a subset of stand and locomotion tasks. We did not test the other modalities. Second, we also tested planning with a learned model. Specifically, we trained an MLP network on the same offline dataset to predict the next state given a state-action pair. We then used this learned model in MPPI and evaluated its performance on the same subset of tasks as H-GAP. The results showed that MPPI with the learned model achieved low normalized return of 0.03. We believe that this is due to MPPI's s action sampling leading to outoo--distribution action plans, which can cause the model to struggle with distribution shift and compounding errors when chaining predictions. Some form of pessimistic planning is necessary when using a learned model to avoid deviating too much from the observed samples. Unlike MPPI, Diffuser achieves this by sampling action plans that are likely under the offline data distribution. For more details on the results of H-GAP and MPPI with the learned model, see Table 23.
36

---

Published as a conference paper at ICLR 2025
Gool
TD3
MPPI
Difluser
GoIL
Goal-TD3
PHC
CALM
ASE
FB-CPR
Success
t.pose
1,00
0.75
(0.07)
(0.00)
1.00:00.00)
0.00
0.98 (0.04)
1,00 (0.00)
apose..oxer.armm
1,00
0.75
0.78 (0.13)
1,00 (0.00)
1.00 (0.00)
0.35 (0.13)
0:49 (0.33)
0.90 (0.19)
1,00 (0.00)
t.poae box head
1.00
0.90
0.77 (0.15)
1.00 (0.00)
1.00 (0.00)
0.06 (4.06)
0.29 (1.299
0.37 (0.321
1.00 (0.00)
u.strotch...riiac
1,00
0.65
(0.02)
0.36 (0.28)
0.80
(0.02)
(0.00)
(0.05)
(0.32)
1,00
0.65
0.00 (0.00)
0.10 (0.17)
0.16 (0.33)
0.00 (0.001
0.00 (1.000)
0.00 (0.00)
0.30 (0.200)
1.00
0.80
0.00 (0.00)
0.23 (0.30)
0.38 (0.44)
0.04 (0.01)
0.00 (0.00)
0.01 (0.02)
0.55 (0.24)
u.stretah.2.left
1,00
0.70
(0.02)
(0.36)
0:99
(0.02)
(0.09)
0.96 (0.07)
8. u.stzetch Back
1.00
0.25
0.00 (0.00)
0.26 (11216)
0.40 (2.42)
0.04 (0.003)
0.00 (1.000)
0.39 (0.45)
0.87 (0.18)
L.ttretch...troot.pos
1,00
1.00
0.59 (0.18)
0.93 (0.11)
0.00 (0.00)
0.05
0.05 (0.09)
0.36 (0.24)
1,00 (0.00)
1,00
1.00
(0.022)
0.34 (0.222
0.64
(0.001
(0..00)
(0.38)
(0.20)
crossed_arm
1.00
0.60
0.04 (0.06)
0.44 (0.29)
0.56 (0.32)
0.01 (0.02)
0.01 (0.02)
0.06 (0.07)
0.63 (0.22)
scrasching.heed
1,00
0.80
0.30 (0
1.00 (0.00)
0.99 (2)
0,D4 (0.02)
0.01 (0.02)
0.96 (0.04)
1,00 (0.00)
sich: hand wava
1.00
0.70
(0.16)
0.99 (1.022
1.00 (0.00)
0.02 (0.02)
(0.12)
(0,021
(0.00)
K.stzoch
1.00
0.60
0.12 (0.09)
0.51 (0.40)
0.87 (0: 15)
0.03 (0.03)
0.00 (0.00)
0. 0:15 (0:37)
0.50 (0.23)
1.stzeoth
0.67
0.00
0.00
0.00 (ULIN))
0.30
0.00 (0,00)
0.00
0.00 (0.001
0.25 (0.28)
armsust retich
1.00
0.60
0.04 (0.05)
0.00 (0.00)
0.21 (0.25)
0.04 (0.03)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
1.00
0.70
0.01 (0.02)
0.00 (0.00)
0:10 00. 19
0.02 (0.02)
0.00 (0.00)
0.00 (0.00)
0. 86 (0.28)
chest
1,00
0.80
0.02 (0.04)
0.88 (0.116)
1.00
0.00
0.001
0,51
0.99 (0.02)
pro,thoow
0.00
0.00
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.06 (0.04)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
agyptian
1.00
0.65
0.03 (0.021
0.43 (0.36)
0.50 (0.300
0.02 (0,02)
0.00 (1LD0)
0.30 (0.351
1.00 (OL00)
1,00
0.75
(0.
0.97 (0.06)
0.04
0.74
1,00 (0.00)
standmartiaal.rss
1,00
0.90
0.11 (0. 18)
1,00 (0.00)
(0.00)
0.04 (0.04)
0.00 (0. (0)
0.82 (0.
1.00 (0.00)
packaboo
0.66
0.60
0.00 (0.000
0.76 (0.35)
0.51 (0:29)
0.04 (0.05)
0.00 (1L00)
0,58 (0.333)
0.89 (0.22)
dancco
1,00
0.70
(0.08)
0.91 (0.12)
(0.03)
(0.39)
(0.00)
kneet
1.00
1.00
0.10 (0.12)
0.31 (11.00)
1.00 (0.00)
0.00 (0.00)
0.00 (1.000)
0.00 (0.00)
0.90 (0.10)
crouch Jhigh
1.00
1.00
0.75 (0.10)
1.00 (0.00)
1.00 (0.00)
0.55 (0.11)
0.37 (0.41)
0.67 (0:25)
1.000 (0.00)
crosat.mediin
1,00
1.00
(0.04)
(0.00)
(0.33)
(0.00)
crouch: Low
1.00
0.95
0.00 (0.00)
0.57 (0.28)
0.45 (0.43)
0.02 (0.01)
0.000 (1.000
0 (0.003
0.72 (0.27)
equat.ppre_jum
1,00
1.00
0.02 (0.02)
0.01 (0.02)
0 (0.03)
0.01 (0.02)
0.00 (0.00)
0.09 (0.16)
0 0.25 (0.25)
squar.handd.an.ggoondd
1.00
0.40
(0.000
0.00 (0.00)
0.64 45)
(0.00)
(11.00
(0.00)
(0.20)
*ide_hiigh.
1.00
0.65
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
pre.trant.kie
1,00
0.70
0.01 (0.02)
0:23 (0.39)
0:30 (0 19)
0,D4 (0.
0.00 (0.00)
0 (0.03)
0.57 (0.36)
arabosque held Cooz
0.68
0.60
(0.02)
0.00 (0.00)
0.00
(0.01)
(0.00)
(0.00)
0,00 (0.00)
hold_righht,
1.00
0.70
0.00 (0.00)
0.00 (0.00)
0.01 (0.03)
0.01 (0.01)
0.00 (0.00)
0.11 (0:21)
0.44 (0.22)
hold.aacc.roc
1,00
0.70
0.00 (0.00)
0.20 (1.266
0:25
0,00 (0,00)
0.00
0.00 (0,000
0.25 (0.28)
benduon_]eff _eg
1.00
1.00
(0.00)
0.00 (0.00)
0.00 (0.00)
0.05 (0.04)
(0.00)
(0.00)
0.00 (0.00)
Lic.cront
1.00
0.90
0.10 (0:20)
0.01 (0.02)
0.00 (0:00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.02)
0.00 (0.00)
backward
1.00
0.95
0.00
0.00 (0.00)
0.00
0,00
0.00
0.00
0.00 (0.00)
Liw.basik bowntt
1.00
0.85
(0.00)
0.00 (0.00)
0.00 (0.00)
0.02 (0.03)
(0.00)
(0.00)
0.00 (0.00)
lic.sida
1.00
0.90
0.00 (0.00)
0.00 (0.I0)
0.00 (0.00)
0.02 (0.02)
0.00 (1L00)
0.00 (0.00)
0.00 (0,00)
1,00
0.55
0.00 (0.00)
0.00
0.00
0.00
0.00 (0.00)
Liebback
1,00
0.90
0.02 (0.04)
0.31 (0.39)
0.00 (0.00)
0.08 (0.05)
0.00 (0. 00)
0.13 (0.27)
0.00 (0.00)
sit side
1.00
0.95
0.00 (0.00)
0.00 (0.08)
0.00 (0.00)
0.00 (0.00)
0.00 (1.00)
0.00 (0.00)
0.00 (0,00)
sit_2hatd_oon_egg
1,00
1.00
0.00 (0.00)
0.00
0.00
0.00
0.00 (0.00)
sit.handbbhhidd
1.00
0.95
0.01 (0002)
0.00 (I.20I)
0.00 (0.00)
0.02 (0.05)
0.00 (11.00)
0,00 (0.00)
0.00 (0.00)
kneess wew.eenddhhemtte Jhendd
1.00
0.95
0.06 (0.07)
0.00 (0.00)
0.18 (0:27)
0.04 (0.02)
0.00 (0.00)
0.00 (0.00)
0.01 (0.02)
bet ige.trontt
1,00
(0.00)
(0.08)
0.00
(0.00)
0.37 (0.31)
push up
1.00
0.95
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
1.000 (0.00)
0,00 (0.00)
0.00 (0.00)
handal and
0.67
0.00
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.02)
0.00 (0.00)
0.00 (0.00)
0.00 (0.30)
handat and right bent
1,00
0.10
(0.00)
0.00 (0.00)
0.00 (0.00)
(0.02)
(0.00)
(0.001
(0.00)
Average
0.31
0.73
0.33
0:44
0.48
Median
1.00
0.75
0.01
0.22
0.39
0.02
0.00
0.01
0.48
Table 21: Humanoid Environment. Success rate over different goal poses in the goal-reaching
evaluation.
Task
R-GAP
MPPI with learned world model Normalized
Normalized
move-ego-0-0
0.123
33.78
0.069
19.05
move ego 0
0.036
9.16
0.040
10.24
move-ego-0-4
0.028
6.82
0.038
9.21
900-
0.041
10.56
0.032
8.26
m0ve-eg0-90044
0.032
7.97
0.026
6.41
move ego 90
0.049
12.46
0.036
9.19
move-ego-90-4
0.039
9.54
0.024
6.00
move-ego-180-2
0.053
13.68
0.024
6.26
move-ego-
0.042
10.41
0.019
4.76
Average
0.05
12.71
0.03
8.82
Median
0.04
10.41
0.03
8.26
Table 23: Humanoid Environment. Average Return of H-GAP and MPPI with learned world model on subset of stand and locomotion tasks.
37

---

Published as a conference paper at ICLR 2025
1.84 (0.0S)
1.76 1.76:00.5)
1.38 (0,02)
1.47 (0.01)
1.57 (0.02)
1:23
1.55(0000)) (0,01)
1.13 (0,0))
1.76 (0.00)
2.27 (004))
1.30
0.40 (0.05)
0.24 (0.15) 15
0.79 (0U0))
0.71 (0.02)
0.8110.00
0.50 (0,00)
0.81 (1,9))
0.79 (0.05)
0.50 (0L13)
0.56 (0005)
d. (0.16) (6)
0.82 (9.04) 0.05 (0.1
1.54
2.08
1,55
1.59
0.80
0.50
0.35
FR- CP-CPR
1.87 (0.01)
1.37 (0.01)
1:48 (0.01)
1.43 (0.01)
1.94 (9.2))
1.000.000
1.57 (0,01)
1.12 (0,)
1.08 (0.01)
(..03
U.67 (0.05
0.30 (0.13)
0.75 (0.0)
0.75 (0:01)
0.27 (0.06)
0.94 (3.4)
0.79 (0.92)
0.82 (0.34)
0.95 (0.1))
0.51 (0,02)
0,50 (0.06)
1.75
1.52
1.41
1.58
1.41
1.54
0.58
0,56
Table 22: Humanoid Environment Average performance over each sub-set set of the AMASS dataset used in the tracking evaluation.
2,0 (0.04)
1.74 (0.00)
1.67 (0.00)
1.74 (3.07)
242 (0.00)
1.45 (0.05)
1.81 (1,77
1.40 (147)
2,00 (.007.00
241 2.11
1.71 (0.10
0.40 (8.18)
0.78 (BAL)
0.78 (8.5
0-10 (0.20
0.54 (8.0))
0.53 (8.05)
0.83 (0.13)
0.94 (0,01)
0.50 0.50 (0.08)
0.47 (0.27)
0.58 (0o0 0.40 (0.04)
0.50 (050(000))
2.07
1.79
1.74
1.53
US7
0,10
PHC
1.95 ((00)
1.78 1.76 (007)
1.70 (0000)
1.78 (0.7))
1.71 (0.00
2.19 2.19 (000)
1.38 (0.0
1.85 (007)
1.42 (007)
1996 (.00)
1.73 (3100
1.7 (005))
0.08 (0:05)
0.50 (0.44)
0:87 (0:01)
0,72 (002))
0.77 (001)
0.30 0.30 (008)
8.94 (0U2)
0.76 (0O)
0.94 (000))
0,95 (0.00)
0.53 (000))
0.55 (Un8)
2.06
1.61
2.04
0.86
0.46
1:77 (0000)
1.44 (0: 101
17 (0.05)
1.51 (0.08)
1.21 (0.00)
1.63 (0.1)
1.17 (OOOS)
1.72 (0005)
1.51 (0 12)
L.59 (0009)
0:06 (000)
0.96 (0.01)
0.53 (002))
D. 06 (001)
0.70 (020
1.00 (X00)
0.91 (002)
1.00 (0000)
0.98 (000)
0.74 (0.00)
0.78 (0.00)
0.S8 (0.
0.00 (0
1.54
2. 2.16
2
1.00
0,70
Goal-113
1.65 (0.00)
(0.05)
1.12 (0.08)
1:18 (0.07)
1.41 (0.0%)
1.06 1.06 (0.11
1.17 1117(0.00
1.62 1.62 (0L10)
1.42 1.42 (0.05)
1.17 7.17 00.08
1.68 1.63 (0.05)
1.77 (0.89)
1-15 (0.09)
150 (0.08)
0.80 (0.00) (0.0%)
0.50 (0. (0)12)
0.97 (0.00)
0.50 00-89)
0.86 (0,01)
0.62 (0.08)
0.99 (1L01)
0.92 (6.01)
0.99 (0.02)
0.99 (0.00)
0.71 0.71 (0.02)
0.78 (0L09)
0.79 (0,00)
0.81 (0.05)
made
1.51
0.95
2. 2.22 (0 05)
2. 19 (0.06)
1.69 (0)
S6 (0.00)
196 (0 03)
2.79 (0.00)
1.45 (5)
1.55 (0.04)
1.86 (0.06)
1-56 (0
2. (.100.000)
2.60 (0.04)
1.56 (0 03)
2.05 (0.02)
0.00 (0.00)
0 0.25 (0.02)
0.06 (0-013
0. 0.22 (0
0.58 (0.04)
0 0.00 00)
0.34 (0108)
0 (5)
0.00 (0.000 (0.00)
0. 0.40 (0.04)
0.33 (3.0))
0.00 (0.00)
0.00 (0 00)
0.00 (1
180
0
Goal-GAll
0.02 (0.04)
1.77 (0.01)
86 (0.02)
1.57 10-22
L.22 (0.00)
2.8 2.09 (0.03)
1.35 (0.00
1.83 (0.05
1.90 (0.00)
2.04 (0.03)
1.76 (0.00)
2.01 (0.00)
0.24 (0.00)
0.00 (0.00)
0.04 (0.00)
0.21 (0,00
0.05 (0.02)
0.32 (0.12)
0. (0-02
0.17 (0.02)
0.40 (0-02)
0.00 (0.01
0.00 (0,30)
0.00 (0,00)
0.00 (0.00)
2.20
31
0. 0.28
0.68
2.25 (.25(00.0) (0.00)
1.74 (0.04)
1.76 (0 02)
1 1.85 (0. (0)
2.76 (0.88)
1.44 (0.02)
1.92 (0,00)
1.37 (BAR)
2.34 (0.00)
1.85 (0.02)
2.04 (84M)
SUCCESS
0-4G (0.05)
0.04 (OLO8)
0.00 (UA))
(..3(0..0)
0.63 (0.04)
0.00 (0.00 (0)
0.8L (BLM)
0.06 (0-02)
0.57 (0.16)
0.39 (0.00)
0.07 (0.
0.36 (0. 16)
EMD
2.21
0.82
0.02
0.20
CALM
2.85 (7
1.71 (0.0))
7.7 02)
1.71 (0.05)
2.09 (0)2)
1.3 (0.0))
1.00 (120)
1.35 (001)
2.08 (0.02)
2.25 (028))
1.71 (0.02)
1,09 (0)44)
0.58 (5)
0.10 (107)
0.53 (0/60)
0.60 (2)
(.74
0.10 (000)
0. 9,00 (0122)
0-85 (0)02)
0. 0.83 (0.08)
0.91 (0.0)
0.35 (0.02)
0.38 (0:05)
0.16 (0.04)
0:33 (0,03)
16
1.56
7.74
0.7%
(0.03)
1.96 (0-023
2.23 (0-02)
2.28 (0.10)
3.05 (0-06)
1.79 (0.05)
2.15 (0.04)
216
1.08 0-333
2 (0.05)
(038)
2.16 (0.05)
0.00 (0,00
0.14 (0.83)
0.31 (0.02)
0.00 (0:00
0.51 (1.11)
0.29 (0)
0.27 (0.80
0.50 (0-05
0.14 (0:04)
0.00 (0.00)
0.04 (0.04)
2.06 00
U.25 (0
2.53
2.66
2
0,00
ASE
2.00 (L.0))
(0 02)
2.05 (0.08)
2.14 00.011
1.71 6L00)
2.09 01.0
2.06 (0 00)
0.02 60.011
0.12 (L02)
0.27 (0.02)
0.05 00)
0.49 0L.G)
0.24 (5)
0.45 (0.08)
0.32 (0.00)
0.00 (0.00
0.04 (0.02)
2.34 (0.
1.70
(2.00.0
0.31 (0
3.30
2.23
2.70
2-12
2.51
0.51
0.50
0.00
PHC (mnotoo))
(0 27)
1.15
1.26 11 63)
1.08 (1
0.8 20
1.54 (0.0)
1.49 (2)
1.44 (1.00)
1.00
0.98 (1-03
1.57 (1.90
1.29
1.10 (0 281
1 (O (0)
1.00 (1. (0.00)
0.92 (0 22
0.95 00. 23)
1.00 (0.00)
1.00 (0 00)
0.86 (1.
0.85 (0 (5)
1.00 (0.00 00.001
0.83 0.83 (0.38)
0:07 (0 (1-47)
0.75 (0.42)
0.82 (0 300
0.87
0.96
2.65
0:96
0.97
1.13 1.13 (1.46)
1.11 (1.80)
1.13 (7.54)
1,23 (1. 58)
1.15 (1.6)
1.71 (2.87)
211 (2,35)
0,8 (1.60)
1.35 (1.12)
1.00 (1.24)
1.23 (1.19)
1.23 (1.2D)
0.54 (0-25)
0,91 (0.28)
0.91 (0.28)
061 (0-10)
0,96 (0
0.91 (0.280
0,96 (0
0,97 (0.17)
0,56 (0.25)
0.97 (0.18)
0,73 (0.48)
0.84 (0 (0336)
2.12 (2
0,83 (0.
0,58
0.96
0
0,59
Goal- GAIL (1motion)
1.22 (0.35)
1,08 (0:20)
1.25 (0.36)
1.15 (0.33)
1.63 (0.0)
(0.2)
1,32 (0.22)
1.11 (02))
(0.20)
0,991020
1.26 (022
1.49.0614)
1.17 (0.16)
0.24
0.00 0.00 (0.00)
(0.9
0.08 0.08 (0.20)
0.1 (0,39)
0.00 (0.00
0,36 0,36 (0.05)
0.00 (0,00)
0.000 (0.00)
0.44 (0.50
0.07 (0.25)
0.00 0.00 (8,00)
0.00 0.00 (0.00)
1.55
1.19
0,89
1.17
0,33
0.00
1.18 (0:35)
1.06 1.06 (0:26)
1.21 1.21 25)
1.17
0.96 0.06 (0.20) 20)
(0 17)
1.32 (0.23)
0.30 (0.9
25)
1,28 (0 28)
1 (0 10
0.20 (G (0)
0.00 0.000(0L00) (OL00)
0,22 (0
0.04 (8)18)
0.16 (0:35)
0.01(0020 20)
0.36 0.30 20)
0.00 (0,00
0.20 0.20:(0.000 (0. 20)
0.41 (0
0.07 0.0T(03) (0.20) 20)
0.00 (0:00)
0.00 (0)00)
1.55
1.45
1.02
1.5
0.47
0,90
BMLhundball
BioMosioLaa
ToplCappuree
BM RiMhhandhall
BioMotionnab
Capuur
Datrsee
ACCAD
aMImmovi
CMU
DFaust
DunceDD
EKUT
Eyes
KIT
MPI
Transitooss
MCCAD
HM MMLamovi
CMU
DFaust
DanceDB
EKUT
Eyes
Evaantiva
KTT
MPР
SPU
Transitions
38

---

Published as a conference paper at ICLR 2025
Sampling Distribution (v)
Reward Evaluation (T)
Tracking Evaluation (1)
Earth Mover's Distance
Normalized Retumm
0.45
0.30
1.0
0.15
0.5
0.00
Unifom
ER
Mixed
Uniform
ER
Mixed
Discriminator Penalty Method
Pollcy Regularization
Reward Evaluation (1)
Tracking Evaluation (4)
Reward Evaluation (1)
Tracking Evaluation (4)
Earth Mover's Distance
Earth Mover's Distance
Normalized Return
0.00
0.60
Normalized
0.45
0.30
0.30
0.15
0.00
0.00
WCAAN
MM
KMM
Figure 6: Additional FB-CPR Ablations. (ToP) Ablating the sampling distribution (BOTTOM LEFT) Ablating the discriminator gradient penalty method. (BOTTOM RIGHT) Ablating the policy regularization method between behavior cloning and moment matching when given action labels. All ablations are averaged over seeds with ranges denoting bootstrapped 95% confidence intervals.
D.2 ABLATIONS
In this section we detail additional ablations into the components of FB-CPR.
Which gradient penalty better stabilizes the discriminator in FB-CPR? Algorithms requiring bi-level optimization through min-max game are known to be unstable and typically require strong forms of regularization (e.g.. Gulrajani et al., 2017; Miyato et al., 2018). Prior works like CALM (Tessler et al., 2023), ASE (Peng et al., 2022), and AMP (Peng et al., 2021) employ what will refer to as the simplified gradient penalty on the discriminator to stabilize training:
|| V v..: (x.2.)
App B.t.M
Alternatively, other works in Inverse Reinforcement Learning (e.g., Swamy et al., 2021; 2022; Ren al., 2024) have had success employing the Wasserstein gradient penalty of Gulrajani et al. (2017):
(1 x²) 22) (o0A
-1)- 27
Dap E Ezweamp TTMM Uninf((0.1)
We want to verify which of these methods better stabilizes training of the discriminator in FB- CPR. To this end, we perform sweep over Aap 1, 1, 10, 15} for both the aforementioned gradient penalties and further averaged over 5 independent seeds. We found that without a gradi- ent penalty, i.e., Age = 0 training was unstable and lead to subpar performance. For both gradient penalty methods found that λgr = 10 performed best and seen in Figure (Left) the Wasser- stein gradient penalty ultimately performed best.
mix- What is gained lost when ablating the mixture components of i/? By modelling ture distribution we hypothesize that a tradeoff is introduced depending on the proportion of each component. One of the most natural questions only sampling M and encoding with z = ERpB(7) (T)).)). If indeed this component is enabling FB- CPR to accurately reproduce trajectories in M we may see an improvement in tracking performance perhaps the cost of diversity impacting ceward-optimization performance. On the other hand, in- creased diversity by only sampling uniformly from the hypersphere may improve reward evaluation performance for reward functions that are not well aligned with any motion in M. We test these hy- potheses by training FB-CPR on 1) only ERPB encoded subtrajectories from M, 2) only uniformly sampled embeddings from the hypersphere, and 3) the default mixture weights reported in Table 9.
ask is whether there is anything to be gained by
39

---

Published as a conference paper at ICLR 2025
Reward Evaluation (f)
Tracking Evaluation (+)
Goal Evaluation (1)
2.8
0.
0.6
Earth Nover's Distance
0.6
Proxtmiity
Normall zod
0.5
1.0-
0.2
32
64
128
256
512
32
64
128
258
512
32
64
120
256
512
monsion
dimension
Figure 7: Performance of FB-CPR in the same setting as Table but with different dimensions of the latent space. Results are averaged over seeds with ranges denoting bootstrapped 95% confidence intervals.
Figure confirms that mixed sampling strikes a nice balance between these trade-offs. Indeed, only using ERFB encoded subtrajectories from M harms reward evaluation performance but surprisingly does not improve on tracking performance. Perhaps unsurprisingly sampling only uniformly from the hypersphere is weak prior and does not fully leverage the motion dataset resulting in substan- tially degraded performance across the board.
Is CPR regularization better than BC if' given action labels? In our work we adopt the moment matching framework to perform policy regularization (Swamy et al., 2021). This framework can be naturally extended to the action-free setting whereas most imitation learning methods require action labels. If we are provided dataset with action-labels should we continue to adopt the moment matching framework with the conditional discriminator presented herein? To answer this question we curate our own action labelled dataset by relabelling the AMASS dataset with a pre-trained FB- CPR policy. Given this dataset we directly compare the conditional discriminator (Eq. 11) with modified form of the FB-CPR actor loss that instead performs regularization via behavior cloning,
BC(T) Egwni Deniiiin an (-1l) [F(s, 2)) O'gc Egwu((s.)) [log (a s)]
(14)
We perform sweep over the strength of the behavior cloning regularization term age E {0.1, 0.2, 0.4, 0.5) and further average these results over 5 seeds. Furthermore, we re-train FB- CPR on the relabelled dataset and also perform sweep over the CPR regularization coefficient Ocrs E (0.01, 0.03, 0.05). Ultimately, One 0.2 and Ocrr = 0.01 performed best with results on reward and tracking evaluation presented in the bottom right panel of Figure We can see that even when given action-labels our action-free discriminator outperforms the BC regularization in both reward and tracking evaluation. This highlights the positive interaction of the conditional dis- criminator with FB to provide a robust method capable of leveraging action-free demonstrations and notably outperforming a strong action-dependent baseline.
How does the latent space dimension affect the performance of FB-CPR? Choosing the dimen- sion d of the latent space built by FB-CPR involves an important trade-off: on the one hand, we would like d be large so to have an accurate estimation of the successor measure of the learned policies, which in turns would yield accurate evaluation of the Q function for many rewards and accurate trajectory encoding through ER (cf. Section 2). Moreover, we recall that task infer- ence involves mapping functions of the state space to latent vectors (e.g., by for a reward function R and B(g) for a goal g). large dimension is desirable to make sure as many tasks/behaviors as possible are learned reliably. On the other hand, it is desirable to use small d to learn set of behaviors which as suceinct as possible, which would be more efficient to train and to query at inference time, argued in several works unsupervised skill discovery (e.g.. Eysenbach et al., 2019; Peng et al., 2022; Tessler et al., 2023; Park al., 2024c).
(B, R(s)
We demonstrate this trade-off empirically in Figure 7, where we repeat the same experiment as in Table for different values of d. We observe a nearly monotonic performance improvement up to dimensions 128 and 256, were performance saturate (with the latter being slightly better reward tasks and the former being slightly better on tracking and goal reaching). As expected, we qualitatively observe that d 32 and d 64 limit too much the capacity of the latent space, as
40

---

Published as a conference paper at ICLR 2025
Algorithm
Reward (f)
Goal
Tracking
EMD (J)
Tracking
Successs (1)
Proximity (1)
Success (f)
Train
Test
Train
Test
FB
24.47 (1.88)
(0)
(0)
8.09 (0.21)
8.19 (0.14)
(
0 (0)
gcORE
0.10
0
0.13
0.13
0
Table 24: Performance of the FB algorithm (Touati & Ollivier, 2021) in the same setting as Table 1,
where ScOREporm are normalized w.rt. the performance the best baseline in such table.
several of the hardest tasks (e.g., cartwheels or backflips) or the hardest goals (e.g.. yoga poses) are not learned at all. On the other hand, we observe a collapse in the learned representation B when moving very large d, which results in the performance drop the fact that several parameters used for the "default" configuration reported in Table I, and kept constant for all runs in this ablation, are not suitable for training with such large d. For instance, the network architecture of F is too small to predict successor features over 512 dimensions, and should be scaled proportionally to d. Similarly, a batch size of 1024 is likely not sufficient to accurately estimate the covariance matrix of B, which is required by the orthonormality and temporal difference losses (cf. Appendix B). Overall we found d = 256 to be a good trade-off between capacity, succinctness, and training stability, FB+CPR with such dimension does not suffer the collapsing issue of d = 512 and learns more difficult behaviors than = 128.
512. This mostly due to
What is the importance of regularizing with unlabeled data? One may wonder whether regular- izing the learned policies towards behaviors in the unlabeled dataset is really needed, or whether the plain FB algorithm of Touai & Ollivier (2021) (i.e., without the CPR part) trained online can already learn useful behaviors and solve many tasks. We report the results of such algorithm, trained with the same parameters used for FB-CPR, in Table 24. The algorithm achieves near-zero performance in all tasks, with only a small improvement over based problems and tracking. Such small improvements is due to the fact that the algorithm learned how to roughly stand up, although without being able to maintain standing position. The main reason behind this failure that the FB algorithm has no explicit component to encourage discovery of diverse behaviors, except for the purely myopic exploration of TD3 (i.e., perturbing each action component with random noise) which obviously would fail in problems with large state and action spaces. On the other hand, the regularization in FB-CPR overcomes this problem by directing the agent towards learning behaviors in the unlabeled dataset.
inittaliiee untrained policy in reward-
41

---

Published as a conference paper at ICLR 2025
D.3 DIVERSITY, DATASET COVERAGE AND TRANSITIONS
In this section we intend to further investigate the behaviors learned by FB-CPR beyond its perfor- mance in solving downstream tasks.
Distance of two random motions
3000
Algoritmm
FB-CPR
CALM
2500
Algorithm
Diversity
Count
FB-CPR
4.70 (0.66)
1500
CALM
3.36 (1.15)
ASE
3.91 (0.73)
1000
500
Figure 9: Average diversity.
EMD
Figure& 8: Distribution of EMD distance between trajectories gen-
erated by two randomly sampled policies πTş and TT²
How diverse are the behaviors learned by FB-CPR? We want to evaluate the diversity of behav- iors encoded in (Tz). Given two randomly drawn = and z', we run the two associated policies from the same initial state and we compute the EMD distance between the two resulting trajectories. We
repeat this procedure for n = 100, 000 times and compute
Diversity = 1 EMD(T.,
(15)
The values of diversity are presented in Table 9. FB-CPR has the highest diversity. This result confirmed by looking at the distribution of EMD values between т; and T/ Fig. 8. FB-CPR has consistently the most diverse results. ASE distribution is shifted toward lower EMD values, which means that its behaviors are less diverse. CALM has mode around 2, which means that its representation has clusters of similar motions, but it is also the algorithm with the wider distribution with EMD distance above 7.0.
Are FB-CPR behaviors grounded in the behavior dataset Mr While this question is partially answered in the tracking evaluation, we would like to evaluate how much the motion dataset is actually covered. In fact, a common failure mode of imitation regularization algorithms is the collapse of the learned policies towards accurately matching only small portion of the demonstrated behaviors. In order to evaluate the level of coverage of the training motion dataset metric to the one proposed (Peng et al., 2022), while accounting for the differences in the dataset: have much larger (8902 vs 187 motions) and less curated dataset, where the length of the motions has much larger variance.
a similar
We first sample random and generate trajectory т. by executing the corresponding policy Ra for 200 steps starting from a T-pose configuration. Then, we calculate the EMD between 7. and each motion in M and we select the motion m with the lowest EMD the one best matching T:
m arg min EMD(T,, m²). M.
(16)
We use EMD instead of almeallggned distance metries account for the fact that т, is executed from an initial state that could be fairly far from a motion in M. We repeat this procedure 10,000
Notice that here are not trying to evaluate the generalization capabilities of the model, which is the focus of Sect. 4.
42

---

Published as a conference paper at ICLR 2025
Algorithm
FB CPR
CALM
0.8
ASE
coverage
0.6
0.4
0.2
0.0
0.1
0.2
0.3
0.4
0.5
Matching Threshold [EMD]
Figure 10: Relation between the threshold used determine motion matching and the coverage of the train dataset by the randomly sampled policies.
1.0
ASE
0.8
0.8
frequency
Frequency
0.6
0.4
02
02
0.2
0.0
Motion
Motion
Motion
The frequency of the 50 most matched motions with multi-matching and MATCHTHRESHOLD = 0.1. Note that each algorithm matches to a different set most frequent motions.
Figure 11:
times and calculate the frequency of selecting each motion from the dataset. The dataset coverage defined as the ratio of the number of the motions selected at least once to the number of motions in the training dataset.
As the train motion dataset is two orders of magnitude larger than the used in (Peng al., 2022), it is naturally harder to cover M. To mitigate this issue, we propose multiple-matching approach: motion m is considered matching, if its EMD to the closest motion from M is larger than
EMD (Tyy mz) + MATCII
(17)
By definition, greater values of the MATCH are matched. Additionally. we observed by qualitative assessment, that when EMD larger than 4.5, then the two trajectories are distinct enough to be considered different behaviors. We therefore discard matching the EMD distance of is above 4.5. The relation between MATCHTERSSHOLD and the coverage is presented highest coverage and it smoothly increases with the EMD threshold. CALM has lower coverage, but presents similar coverage pattern. In comparison, the coverage of ASE remains consistently low.
results in greater coverage, as further motions
Fig. 10. It be observed that FB-CPR has consistently the
In order to calculate the matching of the top 50 most matched motions used in the further compar- ison, used this multi-matching variant with MATCHTURESOLD 0.1. In Fig. 11 report the frequency of the top 50 most matched motions through this procedure for FB-CPR, CALM, and ASE. ASE has very skewed distribution, meaning that many policies πं tend produce trajecto-
43

---

Published as a conference paper at ICLR 2025
ries similar to very small subset of motions, which suggests form of coverage collapse. On the other extreme, FB-CPR has a very flat distribution, suggesting that it has a more even coverage of the motions dataset.
Is FB-CPR capable of motion stitching? Another possible failure mode is to learn policies that accurately tracking individual motions but unable stitch together different motions, i.e., to smoothly transition from one behavior to another. In this case, we sample two embeddings 2ş and zD (respectively source and destination) and use them generate trajectory which is composed of two disjoint sub-trajectories: the first 200 steps are generated with T2ş and form sub- trajectory Ts; after that, the second sub-trajectory TD generated the continuation of TS. while running policy Tap. After their generation. Ts and TD are separately matched to the motions using Eq. 16, and pair of source and destination motion recorded. make the process computation- ally feasible, we restrict our attention to the 50 most frequently matched motions selected in the previous evaluation with Eq. 16, and presented in Fig. 11. The procedure of generating transitioning trajectory is repeated 10,000 times. The pairwise transition probability is defined the probability of matching destination motion, conditioned on the source motion.
We also define pairwise transition coverage a dataset the ratio of the number of pairwise transitions with frequency larger than 0, to the number of all possible pairwise transitions. The pairwise transition probability and respective coverage is reported in Fig. 12. All algorithms have similar overall coverage.
FB-CPR Coverage: 0.127
CALM Coverage: 0.139
ASE
Coverage: 0.101
0.5
Destination Motion
0.1
Source Motion
Source Motion
Source Motion
Figure 12: The probability of transitioning to destination motion conditioned on the source mo- tion. For ASE, there corresponding columns of the heatmap are left empty.
random trajectory matched to source motion in three cases, and the
Is FB-CPR learning more than imitating the motions in M? While the good coverage high- lighted above and the good tracking performance shown in Sect. 4 illustrate that FB-CPR success- fully ground its behaviors on the training motions, remaining question is whether it has learned more than what is strictly in M. In order to investigate this aspect we analyze the distribution of the closest EMD distance FM DMT,, m3) w.r.t. random policies Fig. 13 highlights the most of the behaviors in (T²)) do not necessarily have a very tight connection with motions in the dataset. This contrast with CALM and ASE, which have much smaller EMD distances, thus showing that they tend to use a larger part of the policy capacity to accurately reproduce motions rather than learning other behaviors.
D.4 QUALITATIVE EVALUATION
D.4.1 HUMAN EVALUATION
In most of reward-based tasks, the reward function under-specified and different policies may achieve good performance while having different levels of humar-likeness, In the worst case, the agent can learn to hack the reward function and maximize performance while performing very unnat- ural behaviors. On the other hand, in some cases, more human-like policies may not be "optimal". Similarly, goal-based tasks, different policies may achieve similar success rate and proximity, while expressing very different behaviors.
44

---

Published as a conference paper at ICLR 2025
EMD of random trajectory to the best matching motion
Algorithm
7000
FB CPR
CALM
ASE
5000
4000
2000
1000
Figure 13: Histogram of the values distance of trajectories generated from random z the best matching motion from the training dataset.
In this section, we complement the quantitative analysis in Sect. with qualitative evaluation assessing whether FB-CPR is able to express more "human-like" behaviors, similar to what is done (Hansen et al., 2024a). For this purpose, we enroll human raters to compare TD3 and FB-CPR policies over 45 reward and 50 goal tasks. Similar to the protocol in Sect. for each single reward or goal task, we train three single-task TD3 agents with different random seeds. We then compare the performance of the TD3 agent with the best metric against the zero-shot policy of FB-CPR.
We generate videos of the two agents for each task. Each pair of matching videos is presented to 50 human raters, who fill the forms presented Fig. 14. The position of the videos is randomized and the type of the agent on a video is not disclosed to the raters.
Figure 14: The online forms presented to the human raters to evaluate human- likeness for goal and reward tasks.
45

---

Published as a conference paper at ICLR 2025
Tosk
TD5
ORACLE MPP]
DIPLUEER Nommalzzed
ASE Normalized
Normulized
Normallzed
nove-ego-0--
egp-0-2-ralseerre
191.13
105-12
0.58
168 1040.47)
0.77
145.78
(.2606.00
145.50 (6.28)
0.20
move-eggo
1-m
176-97
195.66
1:11
125 (2.100
0.7246.001
100.56 (20.3))
0.65-00-17
(4.000
0.20.0.00
0.2 ralseatts
194.72
114.30
0.39
103.11 (1.32
0.55
129.21 (23.111
0.00
123. (15.00)
0.65 (0.09)
0-2 r4iimats
179.12
100.26
1.11
124.51 (1.25)
0.69 (0.62)
125 c5
0.70 (8.10)
136.74
(0.01
noverwgo
178.12
105-28
0.57
12: 03 (3.97)
0,68 (6.52)
(6.19 (24.80
0.34
150.15 (15.08
0.78 (0.
179
120.99
11.73
116.70 (3,65)
0.65 (8.62)
123.58 (8.10)
0.69 (8.00)
128.35 (0.99)
0.72 (0.00)
201.00
115.20
1..00
10 (2.72)
0,53 (6.01)
80. (7.07%
0.45 (8.04)
111.92 (1.23)
0.20 (0.01)
175.72
130..66
11.74
113,81 (3.34)
0.63 (0,62)
121 19 (4.20)
0U69 (8.02)
128. 30 (0.78)
(0.00-0.00
nove-cg3-0-2-1000 1oa--opp-----011c000------
165.19
112.05
0.68
102.09
0.62 (0.0c0
133,00 (14.35)
0.81 (8.00)
143,83 (14.21)
0.57
Avenge
0.00%
Median
173,22
0.74
116.80
0.09
136.74
Table 25: Average return for each task in the composite reward evaluation. These tasks combine between locomotion and arm-raising behaviors
We gather two subjective metries: success, and human-likeness. For success, we ask the rater to evaluate whether the presented behavior is actually achieving the desired objective. For goal-based task, the objective is directly illustrated as the target pose, while for reward functions it is text formulated in natural language which replaces the [description] placeholder in the template shown in Fig. 14 (e.g., for the task "raisearms-l-hh we generate text "standing with left hand low (at hip height) and right hand high (above head)"). For Huuman-likeness the rater has to choose among four options where they can express preference for either of the two behaviors, or both (a draw), or none of them. We then compute success rate and average positive answer and the total number of replies. The FB-CPR is considered more human like than TD3 in the large majority of cases. FB-CPR is sometimes assessed as human-like by raters, even in tasks when they consider it failed completing the task. Interestingly, while the human-likeness of FB-CPR may come at the cost of lower reward scores, it does not affect the perceived success in accomplishing the assigned goal tasks and FB-CPR has better success rate than TD3 for those tasks.
-inene- by taking the ratio between the
More in detail, per-task success rate scores are presented in Fig. 15 and Fig, 16.
Human-llkanes
06-
Success Rate
06-
Figure 15: Human-likenese likeness and success rate scores of algorithms per goal task sorted by FB FB-CPR performance.
46

---

Published as a conference paper at ICLR 2025
Human- ler.-llleeeesss
Success
00
Figure 16: Human- and success rate scores of algorithms per reward task sorted by FB-CPR performance.
PD CR
ASE
TD3
Figure 17: Example of combination of locomotion and raising tasks (move-ego-0 2-000022-000 10-2-raii rearsssms m- Our FB-CPR (top) agent produces natural human motions while TD3 (bottom) learns high-performing but unnatural behaviors. ASE (middle) has a natural behavior but it is not correctly aligned with the tasks (arms are in the high position not medium).
D.4.2 REWARD-BARE BASED TASKS
We provide a further investigation the performance of our FB-CPR agent on tasks that are i) combination of tasks used for the main evaluation; and ii) highly under-specified.
47

---

Published as a conference paper at ICLR 2025
Success [%)
Naturalness (l.e., human-llkeness) [%]
97.8
100.0
100.0
58.0
60.0
60.0
26.0
40.0
20.0
20.0
0.4
1.6
0.0
TD3
FD-CPR
TD3 > FB-CPR
FB-CPR TD3
Draw
None
Figure 18: Human-evaluation on locomotion combined with arm raising. Left figure reports the percentage of times behavior solved reward-based task (tasks are independently evaluated). Right figure reports the score for numan-likeness by direct comparison of the two algorithms.
The objective i) is evaluate the ability of FB-CPR of composing behaviors. We thus created new category of reward-based tasks by combining locomotion and arm-raising tasks. Specifically, we pair the medium-speen -speed forward locomotion task (with an angle of zero and speed of 2) with all possible arm-raising tasks. Since these two types of tasks have conflicting objectives - locomotion requires movement, while arm-raising rewards stillness we define composite reward function that balances the two. This is achieved by taking weighted average of the individual task rewards, where the weighting varies depending on the specific task combination. Tab. 25 reports the performance of the algorithms these "combined" tasks. We can see that FB-CPR is able to achieve 74% of the performance of TD3 trained on each individual task. Despite the higher performance, even this case, TD3 generates unnatural behaviors. The higher quality of FB-CPR evident in Fig. 17 where we report a few frames of an episode for the task -ereeee-g------00 sea rms-rmmsme -m-m. Similarly, almost the totality (about 98%) of human evaluators rated FB-CPR these tasks.
natural than TD3
The objective of ii) is to evaluate the ability of our model to solve task with human-like bias. To show this, we designed a few reward functions inspired by the way human person would describe a task.
Run. The simplest way to describe running is "move with high speed". Let ex and vy the horizon- tal velocities of the center of mass at the pelvis joint. Then, we define the reward for the task RUNeq as
r(s') 1(v 2 u 2)
Walking with left hand up. This task has two component: walking requires moving with low speed; raising the hand means having the hand z-coordinate above a certain threshold. Then, we define the reward for the task WALK-LAM
r(s') - - (v - 2 < <1.5 zettt wrist. > 1.2
Standing with right foot up. This the most complex task. We define standing being in upright position with the head z-coordinate above a certain threshold and zero velocity. Similar to before, we ask the right ankle to be above a certain threshold. Then, we define the reward for the tasks STAND-RTM (B = 0.5) and STAND-RTH, (3 -1.2 as
r(s') = u(u up 0.9| 1 Phead 1.4 exp (- +2)))) . Fright ankle >
$)
evident to any expert in Reinforcement Learning (RL) that the reward functions in question are not optimal for learning from scratch. These reward functions are too vague, and traditional RL algorithm would likely derive a high-performing policy that deviates significantly from the nat- ural "behavioral" biases. For instance, with TD3, observe completely unnatural behaviors. In stark contrast, FB-CPR manages to address the tasks in a manner that closely resembles human behavior (refer to Fig. 19). Intriguingly, FB-CPR appears to identify the "simplest" policy neces- sary to solve a task. It effectively distinguishes between two different policies, STAND-RIH RTMM, and STAND-RTH even though the policy designed for the higher task would suffice for the medium
48

---

Published as a conference paper at ICLR 2025
Time
RoNeu
WALK-LAMes
STAND-RIMA RYM
STAND RrH
Figure 19: Example of behaviors inferred by FB-CPR from under-specified reward equations.
task, provided that the foot remains above a certain threshold. is also evident the data bias. For example, do not specify the direction of movement in run, just the high speed. FB-CPR recovers a perfect forward movement probably because the majority of run motions in M show this behavior. ASE is not able solve all the tasks.
49

---

Published as a conference paper at ICLR 2025
E ABLATIONS ON BIPEDAL WALKER
Method
Data
Reward
Demonstration Return
Goal Proximity
Return
FB
RND
0.32 = 0.02
0.43 E 0.02
127.38 + 20.51
FB
RND+MMTRAIN
0.60 = 0.03
0.56 + 0.03
211.46 + 17.78
FB+AWAC
MIRAIN
0.51 0.02
0.54
279.90 + 44.07
FB+AWAC
RND+MMTRAIN
0.42
0.43 0.05
249.72 23.92
FB Online
None
0.19 = 0.03
0.02
120.51 = 10.83
FB-CPP
MTRAIN
0.71 + 0.02
0.75 + 0.01
297.17 + 52.14
FB-MPR
MTRAIN
0.77 + 0.02
0.78 + 0.01
258.66 + 43.89
Mean and standard deviation of performance with different prompts. Averaged over 10 random seeds. Higher is better. Normalized returns are normalized w.r.t expert TD3 policy in the same, rewarded task. RND data is generated by RND policy (Burda et al., 2019), while MIRAIN data was generated by rolling out TD3 policies trained for each task separately.
Table 26:
We conduct an ablation study in the Walker domain of dm_control (Tunyasuvunakool et al., 2020) to better understand the value of combining FB with a conditional policy regularization and online training.
Tasks. For this environment only a handful of tasks have been considered in the literature (Laskin et al., 2021). In order to have more significant analysis, we have developed a broader set of tasks. We consider three categories of tasks: run, spin, crawl. In each category, we parameterize speed (or angular momentum for spin) and direction. For instance, to task where the agent receives positive reward by remaining below a certain height while moving backward at speed 1.5. By combining category, speed, and direction, we define 90 tasks. We also create a set of 147 poses by performing a grid sweep over different joint positions and by training TD3 on each pose to prune unstable poses where TD3 does not reach a satisfactory performance.
5) refers
Data. We select a subset of 48 50 expert trajectories that we add dataset M We also run TD3 policies for subset of 122 goals, while using the same 122 states as initial states, thus leading to total of 14884 goal-based trajectories that are added to .MTRAIN- We then build MMTRAIN = Midemo U U.MRRAIN which con- tains demonstrations for a mix of reward- based and goal-reaching policies. For algorithms trained offline, we use either data generated by random network distillation (RND) (Burda et al., 2019)** or combining RND with MTRAIIN- The MTRAIN dataset contains 17,284 rollouts and 1,333,717 transi- tions! while the "RND" dataset contains 5000 episodes of 100 transitions for total of 5,000,000 transitions.
tasks and for each of them we a TD3 policy to obtain
Evaluation. For reward-based evaluation, we use the 42 tasks that were not used to build the demon- stration dataset. For imitation learning, we consider the same 42 tasks and only 1 demonstration is provided. For goal-based evaluation, we use the 25 goals not considered for data generation.
Baselines. For ablation, we compare FB-CPR to the original FB algorithm (Touati et al.. 2023) trained offline, offline FB with davaattag--wiihhte actor critic (AWAC) (Cetin et al., 2024b), FB trained online, and FB-CPR with an unconditional discriminator (i.e discriminator depends solely on the state), that we refer to as FB-MPR (FB with marginal policy regularization).
Results. Table 26 shows the results for each evaluation category averaged over 10 seeds. For reward-based and imitation learning evaluation, we compute the ratio between each algorithm and the TD3/expert's performance for each task and then average it. For goal-reaching evaluation, we report the average proximity. We first notice that training FB online without access to any demon- stration unsupervised dataset leads to the worst performance among all algorithms. This suggests that FB representations collapse due to the lack of useful samples and, in turn, the lack of a good representation prevents the algorithm from performing a good exploration. Offline FB with only RND data achieves a good performance coherently with previous results reported in the literature.
For walker, RND is successful in generating a dataset with good coverage given the low dimensionality of the state-action space. In humanoid, this would not be possible. Notice that goal gaal- trajectories have different lengths as episades truncated upon reaching the goal.
50

---

Published as a conference paper at ICLR 2025
This confirms that once provided with dataset with good coverage. the unsupervised RL train- ing of FB is capable learning a wide range of policies, including some with good performance on downstream tasks. Adding demonstration samples to RND further improves the performance of FB by 15% for reward-hased tasks, 30% for imitation learning, and by 60% for goal-reaching. This shows that carefully curated mix of covering samples and demonstrations can bias FB of- fline training towards learning behaviors that are closer to the data and improve the downstream performance. Nonetheless, the gap to FB-CPR remains significant, suggesting that regularizing the policy learning more explicitly is beneficial. Interestingly, behavior cloning regularization used in FB-AWAC does significantly improve the performance of FB. When trained on MIRAIN, FB- AWAC significantly improves in goal-based problems, but in reward and imitation learning it is only able to match performance of FB with RND. Mixing the two datasets only marginally improves the goal-based performance, while degrading other metrics. Overall FB with online training with a policy regularization emerges as the best strategy across all tasks. Interestingly, the version with unconditional discriminator achieves better performance for reward and demonstration tasks, while is significantly worse for goal reaching problems, where FB-CPR is best. We conjecture that this result is due to the fact that the dataset M is well curated, since trajectories policies and they cover close regions of the state space, whereas in the humanoid case, .M is made of real data where different motions can be very distinct from each other and are very heterogeneous in nature and length. While in the former case just reaching similar states as in M is sufficient to have a good regularization, in the latter stronger adherence the motions is needed.
generated by optimal
F ABLATIONS ON ANTMAZE
We conduct ablation study in the antmaze domains from the recently introduced goal-conditioned RL benchmark (Park et al., 2024a) to better understand the value combining FB with a conditional policy regularization and online training. Antmaze domains involve controlling quadrupedal Ant agent with degrees of freedom.
2G5
medium
large
Figure 20: Layout of antmaze- medium and antmaze-large domains from (Park et al., 2024a)
Data. We stitch datasets for antmaze domains provided in Park al. (2024a), which consist of short goal-reaching demonstrations trajectories. These datasets are designed to challenge agent's stitching ability over subgoals to complete the downstream tasks.
Evaluation. We use the same evaluation protocol employed in Park et al. (2024a). Each domain has 5 downstream tasks. The aim of these tasks is to control the agent to reach target (z, y) location in the given maze. The task is specified by the full state, but only the (I, y) coordinates are set to the target goal, while the remaining state components are randomly generated. For each goal, we evaluate the agent using 50 episodes.
Results. We present a comparison of three methods in Table 27: online FB trained solely on envi- ronment interactions, offline FB with advantage weighting (AWAC) using the offline stitch datasets, and online FB-CPR that utilizes stitch datasets for policy regularization. We report both success rate and proximity (averaged distance to the goal) averaged across 3 models trained with different random seeds. Online FB fails to reach any test goals, achieving zero success rate due to the lack of exploration. In contrast, FB-AWAC achieves decent performance, which is indeed competitive
51

---

Published as a conference paper at ICLR 2025
Algorithm
Antmaze-medium
Antmaze-large
Proximity (L)
Success (t)
Proximity ())
Success (1)
(online) FB
19.71 (0.11)
(0)
25.74 (0.05)
(0)
(offline) FB-AWAC
6.70 (0.4)
0.67 (0.08)
18.00 (1.54)
0.28 (0.05)
(online) FB-CPR
3.19 (0.13)
0.90 (0.1)
7.97 (0.39)
0.53 (0.08)
Table 27: Performance of different algorithms in Antmaze domains (medium and large mazes). We report mean and standard deviation of the performance over three random seeds.
with the non-hierarchical offline goal-conditioned RL algorithms reported in Park et al. (2024a).
Finally, FB-CPR achieves the strongest performance and outperforms the other FB-variants by significant margin, both in success rate and proximity.
G UNDERSTANDING THE BEHAVIORAL LATENT SPACE
In this section, we summarize results from a qualitative investigation aimed at better understanding the structure of the latent space learned by FB-CPR. We recall that the latent space works the same time as state embedding through (s)), a trajectory embedding through ERFn, and policy embedding through TT
G.1 DIMENSIONALITY REDUCTION OF THE BEUIVIORAL LATENT SPACE
We investigate the structure of the latent space learned through FB-CPR by performing dimension- ality reduction via UMAP (Melnnes et al., 2018) on the embeddings 2 coming from two sources: 1) motion embeddings using ERyn and 2) reward embeddings computed via weighted regression. In order to see meaningful structure in the latent space we decide to classify various motions into five categories: jumping, running, walking, crawling, and motions containing headstands cartwheels.
Given these categories we construct a dataset motions by first choosing a single representative motion for each category and subsequently searching for other motions that are sufficiently close to the reference motion as measured be the Earth Mover's Distance (EMD). We chose all motions where the EMD fell below some threshold that was chosen by visual inspection. With this dataset of motions T; {x1..... En of length n we embed the center most subsequence, i.e., {ox : E [[/2]] - 4, [n/2] + 4]} using ERPB. The center subsequence was chosen was most representative of the category whereas other locations usually had more "set up" in preparation for the motion, c.g., walking before performing a headstand.
Reward embeddings were chosen from Appendix C.3.1 to be representative of the motion category. Specifically, we use the following reward functions for each class:
1. Jumping: smpl-jump-1 2
2. Running:
90-4
3. Walking: smpl move-
4. Crawling:
smpl wrw.w-0.5-2--d
5. Headstand: smpl.headstand
Figure 21 depicts both motion and reward embeddings along with illustrative visualizations for each class of behaviors. Interestingly, the motions involving similar activities are accurately clustered in similar regions through the embedding process. Furthermore, even the reward tasks are embedded within the clusters of motions they are closely connected to. This reveals that the training of FB- CPR leads to learning representations that effectively align motions and rewards in the same latent space.
G.2 BEHAVTOR INTERPOLATION
While the analysis in App. G.I shows that the latent space effectively clusters behaviors that are se- mantically similar, we would like to further understand whether it also supports meaningful interpo-
52

---

Published as a conference paper at ICLR 2025
Behavioral Latent Space
Jump
Notion Enbweding
Danord Er broddding
Run
Crawl
Walk
Headstand
Figure 21: UMAP (McInnes et al., 2018) plot of the latent space of FB-CPR with both motion embeddings (circle) and reward embeddings (star). We can see that reward functions are projected clusters that correspond with motions of the same class of behaviors.
lation between any two points. We have first selected a few reward functions that are underspecified enough that can be combined together (c.g., "run" and "raise left hand" tasks could be composed into "run with left hand up"). We make this choice to investigate whether interpolating between the behaviors associated to each reward function would produce a resulting behavior that is the result of the composition of the two original behaviors. More precisely, given the reward functions r1 and r2, we first perform inference to compute 2 and 22 and we then define 2,, = Q21 + (1 a) and we let vary a in [0. 1]. Refer to the supplmentary material for videos illustrating the behaviors that obtained through this protocol for a few pairs of reward functions. In general, not only we observed smooth variation of the behavior as changes, but the interpolated policies often combine the two original tasks, obtaining more complex behaviors such as running with left hand up or moving and spinning the same time.
53

---

Published as a conference paper at ICLR 2025
on full observations
velocities
on observations
amm My
0.6
0.6
2
2
2
0.3
0.3
0.3
-10
-5
-40
-20
-20
-5
20
10-10
10-10
Figure 22: Rollouts of policies learned by different variants of METRA on Humanoid. Each line corresponds to trajectory in y) space generated by a policy T. with 2 uniformly sampled from the unit sphere. (left) The original METRA algorithm trained from scratch (no unlabeled data) with representation é taking as input the full observation vector. (middle) The original METRA algorithm trained from scratch (no unlabeled data) with representation & taking as input only the linear velocities of the robot's ' pelvis along the x.y.z axes. (right) The ASE algorithm trained within the same setting as Table but with METRA replacing DIAYN as the skill discovery component.
Algorithm
Reward (f)
Goal
Tracking
EMD (J)
Tracking
Success (f)
Proximity (1)
Success (1)
Train
Test
Train
Test
METRA
6.37 (1.04)
0 (0)
(0)
9.92 (0. 13)
9.95 (0.18)
0 (.)))
0 (0)
METRA-ASE
37,98 (6.61)
0.30 (0.01)
0.24 (0.05)
2.11 (0.07)
2.12 (0.05)
0.54 (0.04)
0.56 (0.06)
D-AYN-ASE
105.73 (3.82)
(0.37)
0.22 (0:37)
2.00 (0.02)
1.99 (0.02)
0.37 (0.02)
(0.03)
Table 28: Performance of METRA (Park et al., 2024c) and ASE (Peng et al., 2022) with METRA replacing DIAYN as the skill discovery component in the same setting as Table We also include the original ASE algorithm from such table (called DIAYN-ASE) to ease comparison.
H COMPARISON TO UNSUPERVISED SKILL DISCOVERY METHODS
In FB-CPR, we leverage unlabeled datasets scale unsupervised RI. to high-dimensional problems like Humanoid control. The main conjecture is that unlabeled datasets provide a good inductive bias towards the manifold of behaviors of interest (c.g., those that are human-like) -ike), and that this bias is crucial to avoid the "curse of dimensionality" suffered when learning over the (probably intractable) space of all expressible behaviors. On the other hand, there a vast literature on Unsupervised Skill Discovery (USD) which focuses on learning over such full space of behaviors while providing inductive biases through notions of, e.g., curiosity (e.g., Pathak et al., 2017; Rajeswar et al., 2023), coverage (e.g., Burda al., 2019; Liu & Abbeel, 2021), diversity (e.g.. Gregor al., 2016; Eysenbach et al., 2019; Sharma et al., 2020; Park et al., 2022; 2024c).
In this section, we compare METRA (Park et al., 2024c), the current sfate-of-the-an USD method, and show that it fails on our high-dimensional Humanoid control problem unless given extra induc- tive biases through unlabeled data by restricting the set of variables which focus the dis- covery of new behaviors. Given that METRA remains, to our knowledge, the only USD method to discover useful behaviors high-dimensione problems like humanoid and quadruped control, we conjecture that this "negative" result also applies to all existing USD methods.
Implementation and parameters. We implemented METRA following the original code of Park et al. (2024c), with the only difference that we replaced SAC with TD3 as RL optimizer since we used the latter for all algorithms considered in this paper. We also follow Park et al. (2024c) to tune the hyperparameters related to the representation learning component, while for TD3 we use the same parameters and network architectures we found to work well across all baselines tested in this paper. We found the dimension d of the latent space to be the most important parameter, and we found d = 16 to work best after searching over 2,4,8, 16,32,64, 128,256. All parameters are summarized in the following table.
54

---

Published as a conference paper at ICLR 2025
Table 29; Hyperparameters used for METRA pretraining,
Hyperparametter
Value
General training parameters
See Tab.
General prioritization parameters
See Tab. 4
2 update frequency during rollouts
once every 150 sieps
dimension
16
actor network
third column of Tab. 6, output dim = action dim
networks
second column Tah. output dim
< encoder network
fourth column of Tab. 5, output dim 16, hidden layers
Learning tate for actor
10
Learning rate for critic Learning rate for di Constraint slack
10
10-6
10 10-3
Initial Lagrange multiplicr A
30
z distribution
uniform unit sphere
Probahility of relabeling zs
0,8
Polyak coefficient target network update
0.005
Actor penalty coefficient
0.5
Critic penally coellicient
0.5
Inference methods. For goal-based inference, we follow the zero-shot scheme proposed by Park et al. (2024c): when given a goal state g to reach from state s. we set 2 = (Ø(g) ((s))/llo(g)
((s)]l2- Similarly, for tracking we set 2 = (0(g+=)) (())=))))()))///////)/)))))))))) ((+)))/((((++11 - (( o(sx)l2 ll2 at each step of the episode, where 9++1 is the next state in the trajectory to be tracked, while 8t is current agent state. Finally, for reward inference, given a dataset of transitions (s, r)rr) sampled from the train buffer and labeled with the corresponding reward r, we infer through linear regression on top of features (0s') - ((s). This is motivated by the fact that METRA's actor is pretrained to maximize a self- supervupe reward function given by r(s, 2) == (Ø(s') ((s))*= 2. Notice, however, that do not expect this to work well since such a reward, up to discounting, yields telescopic sum which eventually makes the agent care only about the reward received at the end of episode instead of the cumulative sum. Thus we report its performance for completeness.
Results. We test METRA in the same setting Table 1. The results are reported in the first row of Table 28, where we find that METRA achieves near zero performance in all tasks. After a deeper investigation, we found that in all runs, and with all hyperparameters we tested, the agent simply learned to fall on the floor and remain still in different positions, as shown in Figure 22 (left). Interestingly, this happens despite all the objectives, and in particular the "diversity loss" for representation learning, are well optimized during pre-training. This is due to the fact that, from the agent perspective, lying still on the floor in different positions can be regarded as displaying diverse behaviors, and no extra inductive bias would push the agent to learn more complicated skills (e.g., locomotion ones). On the other hand, we believe that METRA manages to learn few of such skills in the Humanoid experiments of Park et al. (2024c) given that it is pretrained on pixel-based observations (instead of proprioception) with a color map on the ground and very small dimension of the latent space (d = 2). This may provide an implicit inductive bias towards locomotion behaviors that make the robot move around the x.y coordinates, which are likely to be the observation variables that can be maximally spread out by the agent's controls. On the other hand, we do not have any such bias in our setup, where each joint has roughly the same "controllability" and the agent thus learns the simplest way to display diverse behaviors.
To verify this last conjecture, we retrained METRA with the same parameters except that we make the representation & only a function of the linear velocities of the robot's pelvis along the three x.y.z directions. Intuitively, this should provide an inductive bias that makes the agent focus on controlling those variables alone, thus learning locomotion behaviors to move around the x.y.z space. This is confirmed in Figure 22 (middle), where we see that the learned skills do not collapse anymore but rather move around different directions of the space.
METRA with ASE regularization. Finally, we tried to combine METRA with the same policy regularization on top of unlabeled data as used by ASE, As we recall that ASE (Peng et al., 2022) combines USD algorithm (DIAYN) with unconditional policy regularization term, we simply replace DIAYN with METRA and keep all other components the same. The results are shown in Table 28, where see that the ASE regularization improves the performance of METRA signifi- cantly on goal reaching and tracking. Moreover, METRA-ASE achieves competitive performance w.r.t. the original DDayN- ASE, improving its success rate in those tasks. Both DIAYN-ASE
55

---

Published as a conference paper at ICLR 2025
and METRA-ASE perform, however, significantly worse than FB-CPR. Finally, note from Fig- ure 22 (right) that METRA-ASE learns to navigate along different directions, though less far than plain METRA trained only the pelvis' velocities. This is likely due to the regularization w.r.. unlabeled data, which makes the agent focus on human-like behaviors, thus avoiding over-actuated movements that would be otherwise learned when naively trying to maximize controls of subset of the observation variables.
56


Reviews:
[{"review_id": "lHjXoBOZ5e", "reviewer": "Reviewer_D81m", "summary": "The paper proposes FB-CPR, a regularizer for unsupervised RL that improves its pre-training performance given a state-only trajectory dataset. FB-CPR is built on prior works of forward-backward (FB) representation in RL & successor measures of state. An FB approximation is trained with bellman update and can used to approximate successor measure, which is used as a zero-shot policy evaluator. Furthermore, using tricks from prior works, the authors further uses a latent vector z to extend the core components (FB & successor measure) to multiple policies. At pre-training time, FB-CPR is used as an regularizer, with a discrimination loss added to original unsupervised RL objective to make sure learned behavior stay close to the distribution in the dataset. FInally, the authors evaluated the merit of the proposed method with one natural application of simulated humanoid control, where mocap data (state-only trajectory dataset) is available. The resulting method outperforms baselines by some metrics, with outstanding performance in staying human-like.", "questions": "What's the dimension of z? Usually, a policy has to be represented by a neural network. I wonder whether a compact latent variable z is expressive enough to represent policies that have combinatorial complexity. In that case, you must be sacrificing something. Could you provide more intuitions and conduct more ablations on the dimension of z?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is sound towards its goal. Intuitively, the discriminator encourages FB_CPR to learn policies that makes it roll out stay close to that in the dataset M, while the Q function itself is approximated by Fz.\n\nThe paper's presentation is clear considering its technical depth.\n\nThe paper is solid with extensive details for reproduction, evaluations, and sufficient mathematical details. The experiments considered sufficient baselines and there are a good amount of ablations for more insights", "weaknesses": "I think the authors explained the necessary details very well in their writing. However, given that the technical depth of FB heavily depends on prior works, I think the authors should definitely provide more intuitions along the way. When reading the prelim section, there are many times that I need to stop and ask myself why is this math transformation okay. A paper shall be relatively self-contained, and shall still allow readers to understand the high-level intuition of each equation without consulting the details of prior works. \n\nOne core assumption of the work is that one can reparameterize the policy dependence with the introduction of a policy embedding z (Eqn 4). I wonder how grounded is this, especially when z has to live in a continuous space. I list a few questions about z in my Questions section.\n\nIt's pretty clear that the paper heavily depends on the prior line of work of FB & state measure. I skimmed the mentioned works in the prelim section, and it seems that most of these works are evaluated on relative toy datasets only e.g. maze. Therefore, it's a big step for the authors to make claims about an application like humanoid control while skipping evidence of FB & state measure methods being general enough for RL benchmarks. This makes readers wonder whether there are hidden limitations of the proposed method. It seems that standard RL benchmarks are out of scope for this project given its unique setting of having some state dataset, but alternative evidence to address my above concern would be appreciated. \n\nFigure 2 is very helpful to one's understanding of the method. However, I believe two simple changes could significantly enhance it: 1. link $F(\\dot,z)^\\top z$ back to $Q^\\pi$ to remind readers of this connection back in prelim, especially because $Q^\\pi=F(\\dot,z)^\\top z$ wasn't even highlighted with its own equation number. 2. Add a post-training / adaptation box that emphasizes all prior training is happening without reward, and is a separate part from downstream abilities.  \n\nAlgorithm 1 is very helpful too, and I believe many readers would be looking for an algorithm box like this in main paper. The authors could highlight it's in the appendix more, or put a abbreviated algo box in main paper called \"Algorithm 1 (informal)\", and link readers to the full version in the appendix in the caption.\n\nI wonder whether TD3 is a competitive baseline at all for \"Naturalness\" - it seems like sequence-based methods like diffuser serve as a stronger baseline here, as RL methods are purely optimized for reward maximization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2STcI2CsM", "reviewer": "Reviewer_whMZ", "summary": "This paper proposes FB-CPR, a variant of FB representations that regularize behaviors to offline trajectory data. Their main idea is to define a regularization reward with a GAN-style discriminator, and add this as a bonus to the original FB reward. Importantly, the authors condition this discriminator on an inferred $z$, making the regularization more targeted. They apply FB-CRP to Humanoid control, showing that it achieves better performance and naturality than other baselines.", "questions": "I'm curious why FB-CPR is only shown on Humanoid control. For example, could the same method be applied to manipulation environments (e.g., D4RL Kitchen or similar human demonstration-based manipulation datasets)?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is of high quality and well-written.\n* To my knowledge, this is one of the few works that scale unsupervised RL up to Humanoid control.\n* The proposed objective is clean and reasonable.\n* The paper presents several ablation studies, which help understand the importance of the individual components of FB-CPR.", "weaknesses": "* One weakness is its limited novelty. The method is largely built upon the previous FB framework, and the only difference between the original FB paper and this work is the additional discriminator term for behavioral regularization. Having a data-regularization term in data-driven RL (i.e., BC, offline RL, motion priors, etc.) is a standard, well-established technique. While FB-CPR's conditioning of the discriminator on inferred $z$ is distinct from standard offline RL regularization techniques, I believe this alone doesn't constitute significant novelty, given how other skill-based offline unsupervised RL works (e.g., HILP (Park et al., 2024), FRE (Frans et al., 2024), etc.) employ similar $z$-inference techniques when applying behavioral regularization, though they don't use explicit discriminators.\n* Another weakness is that the effectiveness of FB-CPR is only shown on Humanoid. While Humanoid control is indeed an important problem, it is unclear whether FB-CPR can generally be applicable to other environments, or if it is only effective for Humanoid.\n* It seems the benefits of FB-CPR mostly come from behavioral regularization, and the contribution of the FB objective seems relatively marginal (Fig. 4, top right). While the authors show that the FB objective helps to some degree, given the significant complexity of the FB algorithm and its marginal effect on performance, I'm not entirely convinced that having the FB component is worthwhile.\n\nDespite the above weaknesses, I believe the contributions of this work outweigh its weaknesses, and thus recommend weak acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNH1RcnCPY", "reviewer": "Reviewer_Gh4R", "summary": "The authors propose to augment the forward-backwards unsupervised RL framework with a policy regularization term that encourages covering the entire set of behaviors present in the training dataset. They do so by augmenting the FB loss with a discriminator that determines whether a state came from the dataset or from the policy. The authors provide extensive experiments on humanoid control problems showing that FB-CPR approaches the performance of policies trained on individual goals and outperforms other multi-task baselines.", "questions": "- On the humanoid experiments, why aren’t there more direct comparisons to FB without the proposed regularization? There are a few ablations in the appendix with direct comparison to FB, but this also feels relevant to the main experiments. \n- How does the proposed regularization affect the nature of the embedding space? Could a nicely regularized latent space enable something like interpolations between skills etc.?\n\nReferences:\n[1] Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: scalable unsupervised RL with metric-aware abstraction. In ICLR. OpenReview.net, 2024b.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors present their method clearly and motivate the problem setting well.\n- The proposed method is a simple addition to the standard FB framework.\n- The proposed implementation allows steering the model to learn useful behaviors by modifying the latent distribution of skills used during training.", "weaknesses": "- The majority of the evaluations are conducted only on the humanoid environment. Although the number of experiments done in this environment is diverse, higher diversity in environments would make the paper even stronger.\n- Although more “human like” behavior (more like the training dataset) might be desirable in a humanoid environment, could the regularization negatively affect performance where the majority of the data is very suboptimal?\n- It feels like something like METRA [1], which is explicitly trained to span a diverse set of behaviors, is also a relevant baseline. It tackles a similar problem that the proposed regularization intends to solve: spanning a diverse set of useful behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzvJFnRFG9", "reviewer": "Reviewer_hx2C", "summary": "This paper proposes an algorithm that pre-trains behavioral foundation models (BFMs) using unlabeled motion capture data for zero-shot generalization to humanoid control tasks. It combines the Forward-Backward (FB) representation with Conditional Policy Regularization (CPR) to solve tasks like motion tracking, goal-reaching, and reward optimization. FB-CPR outperforms existing unsupervised RL algorithms and model-based methods, achieving competitive results compared to task-specific models while also producing more human-like behavior.\n\n**Post-Rebuttal Review**\n\nThanks for the detailed response. I have carefully read your rebuttal and it resolved most of my concerns. I will raise my score accordingly.", "questions": "* It is unclear how the advantages and differences of this method compare to previous approaches, such as AMP[1] and ASE[2], as well as to recent work (like Omnigrasp[3], MaskedMimic[4]), when applied to the demonstration data settings discussed in this paper.\n* I want to know if it is possible to deploy the method on real robots, just like RobotMDM[5] (I feel that this method is difficult to have this scalability).\n* While the paper focuses on motion capture datasets, how would FB-CPR perform when trained on more diverse or noisy datasets, such as uncurated video footage?\n\n\n[1]Peng, Xue Bin, et al. \"Amp: Adversarial motion priors for stylized physics-based character control.\" ACM Transactions on Graphics (ToG) 40.4 (2021): 1-20.\n\n[2]Peng, Xue Bin, et al. \"Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.\" ACM Transactions On Graphics (TOG) 41.4 (2022): 1-17.\n\n[3]Luo, Zhengyi, et al. \"Grasping diverse objects with simulated humanoids.\" arXiv preprint arXiv:2407.11385 (2024).\n\n[4]Tessler, Chen, et al. \"MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting.\" arXiv preprint arXiv:2409.14393 (2024).\n\n[5]SERIFI, AGON, and MARKUS GROSS. \"Robot Motion Diffusion Model: Motion Generation for Robotic Characters.\" (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors address the challenging problem of generalizing to unseen tasks in humanoid whole-body control without the need for task-specific training. The paper provides a lot of details in the appendix, a well-designed and thorough experimental section. The experiments cover a wide range of scenarios, demonstrating the method's effectiveness across various tasks.", "weaknesses": "The problem definition is not very clear, making it difficult to understand at the beginning. The theoretical explanations about FB representation might be a little complicated for readers lacking corresponding background. Some explicit examples or pictures may help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)