Hierarchical World Models as Visual Whole-Body Humanoid Controllers
Authors: Nicklas Hansen
Jyothir S V
Vlad Sobal
Yann LeCun
Xiaolong Wang
Hao Su
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
world model
humanoid
Primary area: reinforcement learning
TL;DR: We propose a hierarchical world model for visual whole-body control of humanoids, which produces highly performant policies as well as motions that are broadly preferred by humans

Abstract: Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans. Code and videos: https://www.nicklashansen.com/rlpuppeteer

Published as a conference paper at ICLR 2025
HIERARCHICAL WORLD MODELS AS VISUAL WHOLE-BODY HUMANOID CONTROLLERS
Nicklas Hansen'
Jyothir S V
Vlad Sobal
Yann LeCun 2.3
Xiaolong Wang" 1x
Hao Su
UC San Diego
New York University Meta AI
Equal advising
Figure I. Visual whole-body control for humanoids. We present Puppeteer, hierarchical world model for humanoid control with visual observations. Our method produces natural and human-like motions without any reward design or skill primitives, and traverses challenging terrain.
ABSTRACT
Whole-body control for humanoids is challenging due to the high-dimennionall na- ture of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose hierarchical world model in which high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in tasks with simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans.
Code and videos: https: hwwww//wwww///////// www.:/wwwwwwaasassee...e. compp/ppeeeccmmmmppppeeeeeeee
INTRODUCTION
Learning a generalist agent in the physical world is long-term goal of many researchers in Al. Among variant agent designs, humanoids stand out as versatile platforms capable of performing wide range of tasks, by integrating whole-body control and perception. However, this is a very challenging problem due to the high- high-dimensional nature of the observation and action spaces, as well as the complex dynamics of bipedal embodiment, and it makes learning successful yet natural whole-body controllers with reinforcement learning (RL) extremely difficult. For example, consider the task shown in Figure I, where humanoid rewarded for forward progress while jumping over gaps. To succeed in this task. humanoid needs to accurately perceive the position and length of oncoming floor gaps, while carefully coordinating full body motions such that it has sufficient momentum and range reach across each gap.
Due to the sheer complexity of such problems, prior work choose to make simplifying assumptions, such using low- oww-iimeniional (privileged) observations and actions (Heess et al., 2017: Peng al., 2018; Wagener et al., 2022; Jiang et al., 2023; Peng et al., 2021), or (learned) skill primitives (Merel al., 2017; 20180 8b; Hasenclever et al., 2020; Peng et al., 2022; Cheng et al., 2024). Most related

---

Published as a conference paper at ICLR 2025
to work, MoCapAct (Wagener et al., 2022) first learn ~260 individual tracking policies via RL, then distill them into a multi-clip tracking policy via imitation learning, and subsequently train high-level RL policy to output goal embeddings for the multi-clip policy to track. While such approaches have been shown to transfer to simple reaching and velocity control tasks from propri- oceptive inputs, we expect to find solution that can perform complex, visual whole-body control tasks while remaining entirely data-driven and relying on as few assumptions as possible, In this pa- per, we propose to directly learn a visual controller for high-dimensional humanoid robot joints via basee RL and an existing large-scale motion capture (MoCap) dataset (CMU, 2003), while requiring several orders of magnitude less interactions to learn new tasks compared to prior work.
We propose data-driven RL method for visual whole-body control that produces natural, human- like motions and can perform diverse tasks. Our approach, dubbed Puppeteer, is hierarchical JEPA-style (LeCun, 2022) world model that consists of two distinct agents: proprioceptive tracking agent that tracks a reference motion via joint-level control, and visual puppeteer agent that learns to perform downstream tasks by synthesizing lower-dimensional reference motions for the tracking agent to track based on visual observations.
Concretely, the two agents are trained independently in two separate stages using the model-based RL algorithm TD-MPC2 (Hansen et al., 2024) as a learning backbone. First, a single tracking world model is (pre) pprinnee to track reference motions from pre-existing human MoCap data (CMU, 2003) re-targeted to a humanoid embodiment (Wagener et al., 2022). It learns a single model to convert any reference kinematic motion physically executable actions. This is departure from previous work that learns multiple low-level models (Merel et al., 2017; Hasenclever et al., 2020; Wagener al., 2022). Importantly, this tracking agent can be saved and reused across all downstream tasks. In the second stage, we train puppeteering world model that takes visual observation as inputs and outputs the reference motion for the tracking agent based on the specified downstream task. The puppeteer agent is trained with online environment interaction using the fixed tracking agent. A key feature of our framework is its striking simplicity: both world models are algorithmically identical (but differ in inputs/outputs) and can be trained using RL. without any bells and whistles. Different from a traditional hierarchical RL setting, our puppeteer agent (high-level policy) outputs geometric locations for a small number of end-effector joints instead of a goal embedding. The tracking agent (low-level policy) is thus only required to learn joint-level physics. This makes the tracking agent easily sharable and generalizable across tasks, leading to an overall small computational footprint.
To evaluate the efficacy of our approach, we propose a new task suite for visual whole-body hu- manoid control with simulated 56-DoF humanoid, which contains total of 8 challenging tasks. We show that our method produces highly performant control policies across all tasks compared to a set of strong model-free and model-based baselines: SAC (Haarnoja al., 2018), Dream- crV3 (Hafner et al., 2023), and TD-MPC2 (Hansen et al., 2024). Furthermore, we find that mo- tions generated by our method are broadly preferred by humans in a user study with 51 partici- pants. We conclude the paper by carefully dissecting how each of our design choices influence results. Code for method and environments is available at https: / //wwww mii nickla waw.ssssssn...... com / rlpuppeteer. Our main contributions can be summarized as follows:
Task suite. We propose new, challenging task suite for visual whole-body humanoid control with a simulated 56-DoF humanoid. The task suite has 8 tasks in total, and poses significant challenge for existing state-of-the-art RL algorithms. At present, no such benchmark exists.
Hierarchical world model. We propose a simple yet highly effective method for high- dimensional continuous control that uses a learned hierarchical world model for planning.
Evaluating "naturalness" of controllers. We develop several metrics for quantifying how natural and human-like like generated motions across tasks in our suite, including human pref- erences from a user study. To the best of our knowledge, no prior work has explicitly evaluated naturalness of learned policies for humanoid control.
Analysis ablations. We carefully ablate each of our design choices, analyze the relative importance of each component, and provide actionable advice for future work in this area.
2 PRELIMINARIES
Problem formulation. We model visual whole-body humanoid control as a reinforcement learn- ing problem governed by an episodic Markov Decision Process (MDP) characterized by the tuple
2

---

Published as a conference paper at ICLR 2025
MoCap tracking (RL) Au
Downstream tasks (RL)
Tracking agent
(qr.cc)
Puppeteering agent
ct)
Tracking World Model
Tracking World Model
Puppeteer World Model
Save Reuse
(Bc,
(cs,
CH
Train puppeteering agent on downstream Figure 2. Approach. We pretrain tracking agent (world model) on human MoCap data using RL; this agent takes proprioceptive information qi and an abstract reference motion (command) c, as input, and synthesizes H low-level actions that tracks the reference motion. We then train a high- level puppeteering agent on downstream tasks via online interaction; this agent takes both state qe and visual information V, input, and outputs commands for the tracking agent to execute.
tracking agent on MoCap
(S, A. T. y, A) where s E S are states, a E A are actions, S: S x A + S is the environment transition (dynamics) function, R: S x A + IR is scalar reward function, y is the discount factor, and A: S + {0, 1 is an episode termination condition. We implicitly consider both proprioceptive information and visual information v as part of states s and will only make the distinction clear when necessary. We aim learn policy S + A that maximizes discounted sum of rewards in expectation: E, 7're ing motions that look "natural". We informally define natural motions as policy rollouts that are human-like, but develop several metrics for measuring the "naturalness" of policies in Section 4.
R(s,, π (s:)) for an episode of length T, while synthesiz-
TD-MPC2. We build upon the model-based reinforcement learning (MBRL) algorithm TD-MPC2 (Hansen et al., 2024), which represents the state-of-the-arr in continuous control and has been shown to outperform alternatives in tasks with nigh-dimensional action spaces (Hansen et al., 2022; 2024; Sferrazza et al., 2024). Specifically, TD-MPC2 learns latent decoder-free world model from vironment interactions and selects actions by planning with the learned model. All components of the world model are learned end-to-end using combination of joint-embedding prediction (Grill et al., 2020), reward prediction, and temporal difference (Sutton, 1998) losses, without decoding raw observations. During inference, TD-MPC2 follows the Model Predictive Control (MPC) framework for local trajectory optimization using Model Predictive Path Integral (MPPI) (Williams et al., 2015) as a derivative-Fe (sampling-baal based) optimizer. To accelerate planning, TD-MPC2 additionally learns model-free policy prior which is used to warm-start the sampling procedure.
A HIERARCHICAL WORLD MODEL FOR HIGH-DIMENSIONAL CONTROL
We aim to learn highly performant and "natural" policies for visual whole-body humanoid control a data-driven manner using hierarchical world models. A key strength of our approach that can synthesize human-like motions without any explicit domain knowledge, reward design, nor skill primitives. While we focus on humanoid control due to their complexity, our approach can in principle be applied to any embodiment. Our method, dubbed Puppeteer, consists of two distinct agents, both of which are implemented as TD-MPC2 world models (Hansen et al., 2024) and trained independently. Figure 2 provides overview of method. The two agents designed as follows:
1. A low-level tracking agent that takes robot proprioceptive state and an abstract command C as input at time f, and uses planning with learned world model to synthesize a sequence of H control actions (ac. a H } that (approximately) obeys the abstract command.
2. A high-level puppeteering agent that takes the same robot proprioceptive state qe as input, as well as (optionally) auxiliary information and modalities such RGB images v, task- relevant information, and uses planning with a learned world model to synthesize a sequence of H high-level abstract commands (Ce, Ctt1,.... Ct+r for the low-level agent execute.
3

---

Published as a conference paper at ICLR 2025
A unique benefit of our approach is that single tracking world model can be (pre)trained and reused across all downstream tasks. This is in contrast to much of prior work that either learn a large number (up to ~2600) of low-level policies (Merel et al., 2017; 2018b; Hasenclever et al., 2020; Wagener et al., 2022), or train policies from scratch on each downstream task (Peng et al., 2018; 2021). The tracking and puppeteering world models are algorithmically identical (but differ in inputs/outputs), and consist of the following 6 components:
Encoder
h((s)
Encodes state into latent embedding
Latent dynamics
z' = d(z, a)
> Predicts next latent state
Reward
f Riz, a)
Predicts reward r of a state transition
Termination
8 D a.
Predicts probability of termination
Terminal value
I Q a)
Predicts discounted sum of rewards
Policy prior
à = p(z)
Predicts an action a" that maximizes Q
where z is a latent state. Because we consider episodic MDPs with termination conditions, we addi- tionally add a termination prediction head (highlighted in Equation 1) that predicts the probability of termination conditioned planning with a world model requires special care and has, to the best of our knowledge, not been explored in prior work; we introduce novel method for this in Section 3.3. In the following, describe the two agents and their interplay in the context of visual Schol--boyy humanoid control.
latent state and action. Use of termination signals in the context of
3.1 LOW-LEVEL TRACKING WORLD MODEL
We first train the low-level tracking world model independently from the high-level agent and any potential downstream tasks. We leverage pre-existing human MoCap data (CMU, 2003) re- targeted to the 56-DoF "CMU Humanoid" embodiment (Tassa et al., 2018) during training of the tracking model, which (as we will later show empirically) implicitly encodes human motion pri- ors. Specifically, we train our tracking world model by sam- pling (st, a,, Tt, St+1: FH) sequences from MoCapAct (Wa- gener et al., 2022), an offline dataset that consists of noisy, sub- optimal rollouts from existing policies trained track reference motions (836 MoCap clips). This is in contrast to prior literature that learn per-clip policies or skill primitives (Heess et al., 2017; Merel et al., 2017; Hasenclever et al., 2020).
Figure 3. MoCap tracking. The low-level tracking agent is trained track relative end- effector (head, hands, feet) positions of sampled refer- ence motions in 3D space.
Observations include humanoid proprioceptive information qr time t, as well as reference motion (command) c to track. During training of the tracking policy, we let c, = (9.+1.. where each qed corresponds relative end- effector (head, hands, feet) positions of the sampled reference motion at future timestep; during downstream tasks, we train the high-level agent to output (via planning) commands e for the low- level agent to track. Figure 3 illustrates our low-dimensional reference; the controllable humanoid tracks end-effector positions a reference motion. We label all transitions using the reward func- tion from Hasenclever et al. (2020). To improve state-action coverage of the tracking world model, we train with a combination of offline data and online interactions, maintaining a separate replay buffer for online interaction data and sampling offline/online data with 50%/50% ratio in each gradient update as in Feng et al. (2023). We find this to be crucial for tracking performance when training a single world model large number of MoCap clips.
3.2 HIGH-LEVEL PUPPETEERING WORLD MODEL
We now consider training a high-level puppeteering world model via online interaction in down- stream tasks. As illustrated in Figure 2, the puppeteering model is trained (using downstream task rewards) to control the tracking model via commands i.e., we redefine commands to now be the action space of the puppeteering agent. The tracking world model remains frozen (no weight updates) throughout this process, which allows to reuse the same tracking model across all down- stream tasks. Because the high-level agent uses planning for action selection, it natively supports temporal abstraction by outputting multiple commands (Ct, Cttly. Ct+ 11) for the low-level agent
4

---

Published as a conference paper at ICLR 2025
stand
walk
run
cdrridor
hurdles
walla
gaps
stai
Figure 4. Tasks. We develop 5 visual whole-body humanoid control tasks with 56-DoF simulated humanoid (bottom), as well as 3 non-visual tasks (top). See Appendix for more details.
to execute; we treat the number of low-level steps per high-level step as a hyperparameter that al- lows to trade strong motion prior (large k) for control granularity (small k). The high-level policy outputs commands at fixed frequency regardless of whether the previous command was achieved.
3.3 PLANNING WITH TERMINATION CONDITIONS
We consider episodic MDPs with termination conditions. In the context of humanoid control, a common such termination condition is non-foot contact with the floor. Use of termination conditions requires special care in the context of world model learning and planning, as both components used simulate (latent) multi-step rollouts. We extend the world model of TD-MPC2 with termination prediction head D, which predicts the probability of termination at each time step. This termination head is trained end-to-end together with all other components of the world model using
Luppeteer (0) = LTD- a CE(),
where 8, 5 are predicted and ground-truth termination signals, respectively, CE is the (binary) cross- entropy loss, and constant coefficient balancing the losses. We additionally truncate TD-targets at terminal states during training. It is similarly necessary to truncate model rollouts and value estimates during planning (at test-time). However, we only have access to predicted termination signals at test-time. which can be noisy and consequently lead to high-variance value estimates for latent rollouts. To mitigate this, we maintain a cumulative weighting (discount) of termination probabilities when rolling the model (capped at 0), such that only a soft truncation is applied.
EXPERIMENTS
Our proposed method holds the promise of strong downstream task performance while still syn- thesizing natural and human-like motions. To evaluate the efficacy of our method, we propose a new task suite for whole-body humanoid control with multi-modal observations (vision and propri- oceptive information) based on the "CMU Humanoid" model from DMControl (Tassa et al., 2018). Our simulated humanoid has 56 fully controllable joints (A E 165)), and includes both head, hands, and feet. We aim to learn highly performant policies in a data-driven manner without the need for embodiment- or task task-specific engineering (e.g., reward design, constraints, or auxiliary objectives), while synthesizing natural and human-like motions. Code for method and environments is available https: /www. n lashansen. com/ puppeteer.
4.1 EXPERIMENTAL DETAILS
Tasks. Our proposed task suite consists of 5 vision-conditionel whole-body locomotion tasks, and an additional 3 tasks without visual input. We provide an overview tasks Figure 4; they are designed with high degree of randomization and include running along corridor, jumping
5

---

Published as a conference paper at ICLR 2025
Stand
Walk
Run
Corridor
500
500
500
200
250
250
250
100
0-
0-
0-
0-
Hurdles
Walls
Gaps
Stairs
200
100
200
200
100
50
100
100
1M
1M
2M
2M
3M
SAC
DreamerV3
TD-MPC2
SAC w/ LL. TD-MPC2
DreamerV3 w/ LL. TD-MPC2
Puppeteer (ours)
Figure 5. Learning curves. Episode return vs. environment steps on all tasks from our proposed task suite. Our method generally matches the return of TD-MPC2 these tasks while producing more natural motions. We only evaluate SAC and DreamerV3 on proprioceptive tasks as they do not achieve any meaningful performance. Average of 10 random seeds; shaded area is 95% Cls.
hurdles and gaps, walking up the stairs, and circumnavigating obstacles (walls). All 5 visual control tasks use reward function that is proportional to the linear forward velocity, while non-visual tasks reward displacement in any direction. Episodes are terminated at timeout (500 steps) or when non-foot joint makes contact with the floor. We empirically observe that the TD-MPC2 baseline degenerates to highly unrealistic behavior without a contact-basee termination condition, and thus modify TD-MPC2 to support termination as described in Section 3.3. See Appendix E for details.
Implementation. We pretrain a single 5M parameter TD-MPC2 world model to track all 836 CMU MoCap (CMU, 2003) reference motions retargeted to the CMU Humanoid model. This contrast to, e.g., MoCapAct (Wagener al., 2022) that trains ~260 individual tracking policies. Our tracking agent is trained for 10M steps using both offline data (noisy rollouts) from MoCapAct (Wagener al., 2022) and online interaction with a new reference motion sampled in each episode. We sample 50% of each batch from the offline dataset, and 50% from the online replay buffer for each gradient update; we did not experiment with other ratios. The puppeteering agent is simi- larly implemented as 5M parameter TD-MPC2 world model, which we train from scratch via online interaction on each downstream task. Observations include a 212-d proprioceptive state vector and 64 x 64 RGB images from a third-person camera. Both agents act at same frequency, i.e., set 1. Training the tracking world model takes approximately 12 days, and training the puppetecring world model takes approximately 4 days, both single NVIDIA GeForce RTX 3090 GPU. CPU and RAM usage is negligible. System requirements detailed in Appendix
Human preference
97.8%
0.4%
1.8%
TD-MPC2
Equal
Ours
Figure 6. Human pref- erence in humanoid mo- tions. Aggregate results from a user study (n = 51) where humans are presented with pairs of motions gener- ated by TD-MPC2 and our method, and are asked to provide their preference.
Baselines. We benchmark our method against state-of-the-art RL al- gorithms for continuous control, including (I) widely used model-free RL method Soft Actor-Criie (SAC) (Haarnoja et al., 2018) which learns a stochastic policy and value function using a maximum entropy RL objective, (2) model-based RL method DreamerV3 (Hafner et al., 2020; 2021; 2023) which simultancously learns world model using generative objective, and model-free policy in the latent space of the learned world model, (3) model-based RL method TD-MPC2 (Hansen et al., 2022; 2024) which learns a self- self-predictive (decoder-free) world model and selects actions by planning with the learned world model, (4) hierarchical baseline that uses the same low-level TD-MPC2 agent our method but trains a SAC policy as the high-level agent, and (5) the same hierarchical baseline but with DreamerV3 as the high-level agent (and TD-MPC2 at the low level). We refrain from making direct comparison MoCapAct (Wagener et al., 2022) and DeepMimic (Peng et al., 2018) as they do not support visual observations and require several orders of magni- tude more environment interactions learn downstream tasks. Both method and baselines

---

Published as a conference paper at ICLR 2025
hurdles +
Ours
ID-MPC2
stairs rs
Ours
TD-MPC2
Figure 7. Qualitative results. Our hierarchical approach, Puppeteer, produces natural human motions, whereas TD-MPC2 trained end-to-end often learns high-performing but unnatural gaits.
the same hyperparameters across all tasks, as TD-MPC2 and DreamerV3 have been shown to be ro- bust to hyperparameters across task suites (Hansen et al., 2024; Hafner al., 2023; Sferrazza al., 2024). For a fair comparison, we experiment with various design choices and hyperparameter con- figurations for SAC and report the best results that we obtained. We provide further implementation details in Appendix D.
4.2
MAIN RESULTS
We first present our main benchmark results, and then analyze and ablate each design choice.
Benchmark results. We evaluate our method, Puppeteez: and baselines on all 8 whole-body humanoid control tasks. Episode return as function of environment steps is shown in Figure 5. We observe that the performance our method is comparable to that of TD-MPC2 across all tasks (except stairs), whereas SAC and DreamerV3 does not achieve any meaningful performance within our computational budget of 3M environment steps; hierarchical DreamerV3 achieves non-trivial yet still poor performance with TD-MPC2 as the low-level agent. As we will soon reveal, TD- MPC2 produces better policies in terms of episode return on the stairy task, but less natural behavior (walking vs. rolling up stairs). We conjecture that this is symptom of reward hacking (Clark& & Amodei, 2016; Skalse et al., 2022). Sample videos are available at https: / /www. nick cickkahhassen com puppptteer....rrrr
"Naturalness" of humanoid controllers. We conduct a user study (n 51) in which humans are shown pairs of short (~10s) clips of policy rollouts from TD-MPC2 and our method, and are asked provide their prefer- ence. Participants are undergraduate and graduate students across multiple universities and disciplines. Results from this study are shown in Figure 6, and Figure 7 shows two sample clips from the study. While both methods perform comparably in terms of downstream task reward, super- majority of participants rate rollouts from our method as more natural than that of TD-MPC2, with only 4% of re- sponses rating them as "equally natural" and 0% rating TD-MPC2 as more natural. This preference especially pronounced in the stairs task, where TD-MPC2 achieves higher asymptotic return (higher forward velocity) but learns to "roll" up stairs as opposed to our method that walks. We also report several quantitative measures of naturalness in Table 1, which strongly support our
Table 1. Proxies for "naturalness". Evaluated on the gaps task. eplen de- notes the average episode length (sur- vival time) at 1M steps and at con- vergence; height is the average torso height (gait) at convergence. Mean and std. across 10 seeds.
oplen IM
oplen
height (cm) "
TD-MPC2
181. 28.1
86.9
Ours
5.2
5.9
96.0

---

Published as a conference paper at ICLR 2025
Pretraining (tracking)
Downstream tasks
Data mixture
MoCap clips
Hierarchical planning
High-level pretraining
61.7
43.0
48.6
40%
50%-
40%
35.1
50%
21.0
20%
1.6
0%
0%
0%
0%
2k1
4h
GkI
sk
1%
planning
1M
Offline
5%
Planning high
Online
25%
Planning low
Scratch (default) Pretrained
Offline + online
100%
Planning low+high
Figure 8. Ablations. Normalized score for various ablations of Puppeteer during pretraining (left) and downstream tasks (right). Pretraining benefits from diverse data, as well both pre- existing (offline) data and online interactions. We also observe that planning is critical to whole- body humanoid control. Mean across 3 seeds; downstream ablations are averaged across 5 tasks.
Generalization (gaps)
50%
Traniing
(6-1, 0.8)
0% 0.0
0.5 Gap length
1.0
0. 1m
0. 4m
O. 8m
2m
Visualization of gap lengths
Puppeteer (ours)
TD-MPC2
Figure 9. Zero-shot generalization to larger gap lengths. (Left) Visualization of gap lengths. Agent is trained gaps 10.1, 0.4 m and evaluated on gaps up 1.2m. (Right) Normalized perfor- mance as a function of gap length in the visual gaps task. Mean of 3 seeds. Our method achieves non-trivial performance on gaps up to 3x the training data. CIs omitted for visual clarity.
study results. These findings underline the importance of a more holistic evaluation of RL policies as opposed to solely relying on rewards. See Appendix A for more results.
4.3 ABLATIONS ANALYSIS
We ablate each design choice of our method, including both the pretraining (tracking) and down- stream task learning stages. Our experimental results are summarized in Figure 8.
Pretraining (tracking). Our method leverages both offline and online data during pretraining of the tracking world model. We ablate this training mixture in two distinct ways: (i) using only offline or online data, and (ii) reducing the number of unique MoCap clips seen during training. Interestingly, find that leveraging both data sources leads to better tracking policies overall. We hypothesize that this because offline data may help in learning to track especially difficult motions such jumping and balancing on leg, while online data improves state-action coverage and thus leads to a more robust world model overall. Similarly, training on more diverse MoCap clips also leads better tracking performance. Training all 836 MoCap clips results the best tracking world model, and we expect tracking to further improve with availability of more MoCap data. See Appendix B for more results.
Downstream tasks. We conduct three ablations that help us better understand the impact of a hierarchical approach to downstream tasks: (i) using learned model-free policy in lieu of planning in either level of the hicrarchy, (ii) pretraining of the high-level agent in addition to the low-level agent, and (ūi) evaluating zero-shot generalization to unseen environment variations (gap length in the gaps task). The first two ablations are shown in Figure 8, and the latter is shown in Figure 9. We find that planning at both levels is critical to effective whole-body humanoid control, which we conjecture is due to the high dimensionality of the problem; this is supported by concurrent studies on nigg-dimmnnsionll continuous control with TD-MPC2 (Hansen et al., 2024; Sferrazza et al., 2024). Next, we pretrain agents the corridor task and independently finetune While the specific environments and motions differ between tasks, we find that our method benefits substantially from finetuning. We conjecture that this is because the need to control low-level
each visual control task.
8

---

Published as a conference paper at ICLR 2025
tracking agent is shared between all high-level agents, irrespective of the downstream task. This is in contrast to contemporary work on pretraining that often pretrains on large-scale out-of-domain data (Hansen et al., 2023b; Xu et al., 2023). Finally, of our method to harder, unseen variations of the gap task. Interestingly, we observe that our method generalizes to gap lengths up to 3x the training data without additional training. In light of these results, we believe that further investigation of the generalization ability of hierarchical world models will be a promising direction for future research.
explore the zero-shot generalization ability
5 RELATED WORK
Learning whole-body controllers for humanoids is a long-standing problem at the intersection of the machine learning and robotics communities. Humanoids are of particular interest to the learning
community because of the high-dimensional nature of the problem (Heess et al., 2017: Merel al., 2017; Peng et al., 2018; Merel et al., 201 8b; Hasenclever et al., 2020; Won et al., 2022; Wagener et al., 2022: Shi et al., 2022; Caggiano et al., 2022: Sferrazza et al., 2024; He et al., 2024; Cheng et al., 2024), and to the robotics community because it is promising morphology for general- purpose robotic agents (Grizzle et al., 2009; Li et al., 2023; BostonDynamicss 2024; Unitree, 2024; Cheng et al., 2024). Prior work predominantly focus on learning control policies for individual tasks using model-free reinforcement learning algorithms, with human MoCap data (CMU, 2003) incor- porated via either adversarial reward terms (Peng al., 2018; 2021; 2022) or learned skill encoders (Heess et al., 2017; Merel et al., 2018b; Hasenclever et al., 2020; Shi et al., 2022; Wagener et al., 2022). While adversarial reward terms can produce natural behavior, this class of methods suffer from poor sample-efficiency as they learn a control policy from scratch for each downstream task. Our work is most similar to the latter class of methods, which enables reuse of the low-level pol- icy and/or skill encoder across tasks. Most related to ours, MoCapAct ( (Wagener et al., 2022) first learn ~660 individual tracking policies via RI., then distill them into multi-clip tracking policy via imitation learning, and subsequently train high-level RI. policy to output goal embeddings for the multi-clip policy to track. Their resulting representation is used to perform simple reaching and velocity control tasks from privileged state information in approx. 150M environment steps. Our method trains a single world model to track the entire MoCap dataset, and is reused to learn a variety of visual whole-body control tasks in I 3M environment steps. Concurrent to our work, Humanoid- Bench (Sferrazza et al., 2024) similarly introduce a whole-body control benchmark using the less expressive Unitree H1 (Unitree, 2024) embodiment. Our contributions differ in two important ways: (1) we develop a method for synthesizing natural human motions with a highly expressive humanoid model while Sferrazza et al. (2024) benchmark existing methods for online RL without regard for naturalness, and (2) HunanoidBench. solcly considers tasks with privileged state information in their experiments (i.e., no visual observations).
World models (and based RL more broadly) are of increasing interest to researchers due to their strong empirical performance in an online RL setting (Ha & Schmidhuber, 2018; Hafner et al., 2023; Hansen et al., 2024), as well as their promise of generalization to structurally similar problem instances (Zhang et al., 2018; Zheng et al., 2022; Lee et al., 2022; Xu et al., 2023; LeCun, 2022: Sobal et al., 2022; Brohan al., 2023). Existing model-based RL algorithms can broadly be categorized into algorithms that select actions by planning with learned world model (Ebert et al., 2018; Schritwieser al., 2020; Ye et al., 2021; SV et al., 2023; Hansen et al., 2024), and algorithms that instead learn a model-free policy using imagined rollouts from the world model (Kaiser et al., 2020; Hafner al., 2023). We build upon the TD-MPC2 (Hansen al., 2024) world model, which uses planning and has been shown to outperform existing algorithms for continuous control (Hansen al., 2023a; Lancaster et al., 2023; Feng et al., 2023; Sferrazza al., 2024). We demonstrate that planning is key to success in the high-dimensional continuous control problems that we consider.
Hierarchical RL offers a framework for subdividing a complex learning problem into more ap- proachable subproblems, often by, e.g., leveraging (learned or manually designed) skill primitives (Pastor et al., 2009; Merel al., 2017: 201 8a; Shi et al., 2022) or facilitating learning over long time horizons via temporal abstractions (Nachum et al., 2019; LeCun, 2022; Hafner et al., 2022; Gumbsch et al., 2023; Chen et al., 2024). Our method, Puppeteer, is also hierarchical nature, but does not rely on skill primitives nor temporal abstraction for task learning. Instead, we learn single low-level world model that can be reused across a variety of downstream tasks, and instead introduce a hierarchy in terms of data sources and input modalities.

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
We demonstrate that Puppeteer consistently produces motions that are considered natural and human-like by human evaluators compared to existing methods for visual RL, in entirely data- driven manner and without any bells and whistles. These results are, to the best of our knowledge, unprecedented in the of whole-body humanoid control. However, we acknowledge that contributions have several limitations that may not be obvious: (i) while our proposed task suite consists of challenging visual whole-body control tasks with detailed humanoid model, tasks pri- marily evaluate the visio-locomotive capabilities of current methods. We expect development of new tasks will be increasingly important as algorithms continue improve, and we hope that the release of our benchmark will help facilitate that. (ii) Our hierarchical approach currently consists of two levels, only one of which is pretrained in the majority of our experiments. Our experiment on high-level pretraining suggests that it can be beneficial to pretrain both levels, but further research on how to pretrain and transfer the full hierarchical world model is warranted.
Ethics statement. 5 participants in our user study are sourced from undergraduate and graduate student populations across multiple universities and disciplines on a volunteer basis. We do not collect personal otherwise identifiable information about participants, and all participants have provided written consent to use of their responses for the purposes of this study. See Appendix F for more information.
Reproducibility statement. Code for method and environments, as well as model checkpoints, is made available at https: //wwww micklashansee Control and MuJoCo for simulation which are publicly available and licensed under the Apache 2.0 license. We leverage the MoCapAct dataset for pretraining which is also publicly available and li- censed under the MIT license. Implementation details and a full list of hyperparameters are provided in Appendix D.
www./ommlpppp.e htpppppppppperrrr. We rely DM-
Acknowledgements. This work was supported, in part, by NSF CCF-21 12665 (TILOS). Nicklas Hansen is supported by NVIDIA Graduate Fellowship. and Vlad Sobal is supported by NSF Award 19922658.
REFERENCES
BostonDynamics. Atlas, 2024. URL www. www.bostondynamics. com/ atass.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Ri-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.155118 2023.
Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite a contact-rich simulation suite for musculoskeletal motor control, 2022.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.025 40.004444, 2024.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Learning fast without a model. International Conference on Learning Representations, 2021.
Randomized ensembled double q-learning:
Xuxin Cheng, Yandong Ji. Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 2024.
Jack Clark and Dario Amodei. Faulty reward functions in the wild. OpenAl Blog, 2016.
Carnegie Mellon University CMU, Carnegic mellon university graphics lab motion capture database, 2003. URL http: //mocap. cs cmu. edu.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. ArXiv, abs/1812.00568, 2018
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Fine- tuning offline world models in the real world. Conference Robot Learning, 2023.
10

---

Published as a conference paper at ICLR 2025
Grill, Florian Strub, Florent Altch'e Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020.
J.W. Grizzle, Jonathan Hurst, Benjamin Morris, Won Park, and Koushil Sreenath. Mabel, new robotic bipedal walker and runner. 2009 American Control Conference, pp. 2030-2036, 2009. doi: 10. 00.119/AACC.2009516055
Christian Gumbsch, Noor Sajid, Georg Martius, and Martin V Butz. Learning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. In The Twelfth Inter- national Conference on Learning Representations, 2023.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pp. 2451-2463. Curran Associates, Inc., 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. ArXiv, abs/181 11208990. 2018.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi, Dream to control: Learn- ing behaviors by latent imagination. ArXiv, abs/1912.0161)) 2020.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. International Conference on Learning Representations, 2021.
Danijar Hafner, Kuang-Husd Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. Advances Neural Information Processing Systems, 35:26091-26104, 2022.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICML, 2022.
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model- reinforcement learning with demonstrations. 2023a.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze. Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for risuo-motor control: Revisiting learning- from- scratch bascline. In International Conference on Machine Learning (ICML), 2023b.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control, 2024.
Leonard Hasenelever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. Comic: Com- plementary task learning & mimicry for reusable skills. In International Conference on Machine Learning. pp. 4105 41 15.. PMLR, 2020.
Tairan Hc. Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning arXiv:2403.0m 044.64466, 2024.
real-time whole-body teleoperation. arXiv preprint
Nicolas Heess, Dhruva Tb, Srinivasan Sriram, Jay Lemmon. Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environ- ments. arXiv preprint arXiv: 1707.02286, 2017.
Zhengyao Jiang, Yingchen Xu, Nolan Wagener, Yicheng Luo, Michael Janner, Edward Grefenstette, Tim Rocktäschel, and Yuandong Tian. H-gap: Humanoid control with a generalist planner. arXiv preprint arXiv:23/2.02669 2023.
11

---

Published as a conference paper at ICLR 2025
Lukasz Kaiser, Mohammad Babacizadeh. Piotr Milos, Blazej Osinski, Roy H. Campbell, K. Czechowski, D. Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, G. Tucker, and Henryk Michalewski. Model-basee reinforcement learning for atari. ArXiv, abs/1903.00374,, 2020.
Patrick Lancaster, Nicklas Hansen, Aravind Rajeswaran, and Vikash Kumar. Modem-v2: Visuo- motor world models for real-world robot manipulation. arXiv preprint, 2023.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. L. Y. Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan S. Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. ArXiv, abs/2205. 15241, 2022.
Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Robust and versatile bipedal jumping control through reinforcement learning. In Kostas E. Bekris, Kris Hauser, Sylvia I.. Herbert, and Jingjin Yu (eds.). Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023, doi: 10.15607/RSS.2023 20T.:/22.200X.0.0.555
URL https:
//doi. org.org//////oooo 10. 15607 RSS 2023. XIX. 052.
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real- time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10994, 2023.
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv: 1rXiv:1707.02201 2017.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nico- las Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. arXiv preprint arXiv:1811.09656, 2018a.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint 1811 2018b.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: 1909. 10618, 2019.
Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics Automation, pp. 763-768, 2009.
doi: 10. 10.1 1w..109/ROBOT.20955555222
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physies-based character skills. ACM Trans. Graph., 37
(4):143:1-143:14 July 2018. ISSN 0730-0301. doi: 10.1145/31975173220333 URL http:
//doi. acm. org/ 1145 /1114////99171111 3201311.
Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-bar based character control. ACM Transactions on Graphics (ToG), 40(4):1 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), -144):1-17, 2022.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839)6604-699 2020.
Carmelo Sferrazza, Dun-Ming Huang. Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-hody locomotion and manipulation, 2024.
12

---

Published as a conference paper at ICLR 2025
Lucy Xiaoyang Shi, Joseph J. Lim, and Youngwoon Lee. Skill-basee model-based reinforcement learning. 2022.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Vlad Sobal, Jyothir SV, Siddhartha Jalagam, Nicolas Carion, Kyunghyun Cho, and Yann Le- Cun. Joint embedding predictive architectures focus on slow features. arXiv preprint arXiv:2211.1083/1 2022.
R. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1998.
Jyothir SV, Siddhartha Jalagam, Yann LeCun, and Vlad Sobal. Gradient-based planning with world models. arXiv preprint arXiv:2512.17227, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, al. Deepmind control suite. Technical report, DeepMind, 2018.
Unitree. HI, 2024. URL www. unituniieee wtw.ttrree........ com/hl.
Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, and Matthew Hausknecht. MoCapAct: A multi-task dataset for simulated humanoid control. In Advances in Neural Information Processing Systems, volume 35, pp. 35418-35431, 2022.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, abs/1509 149, 2015.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. ACM Trans. Graph., 41(4), July 2022. ISSN 0730-0301. doi: 10.1145/3528223. 3530067. URL https:://// /tw./iiiiia// org 10 114.11145555 211111////2222332 .0003333330000 3530067.
Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, and Zhuowen Tu. On the feasibility of cross-task transfer with model-based reinforcement learning. 2023.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// 2020.
github. com/ denidenissnn www.atysptttttrtttttttee
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Picter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, P. Abbeel, Matthew Johnson, and Sergey Levine Solar: Deep structured latent representations for model-hased reinforcement learning. ArXiv, abs/1808.09105, 2018.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In ICML, 2022.
13

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL QUALITATIVE RESULTS
corridor -
Ours
TD-MPC2
gaps
Ours
TD-MPC2
walls -
Ours
-PCC
hurdles
Ours
TD-MPC2
14

---

Published as a conference paper at ICLR 2025
gaps
Ours
TD-MPC2
stairs
Ours
TD-MPC2
1
Ours
TD-MPC2
B ADDITIONAL QUANTITATIVE RESULTS
This section supplements our quantitative evaluations included in the main paper. In particular, Table provides additional performance metrics for our low-level tracking agent, which we define as follows:
Success rate defined the percentage of clips that the tracking policy is able to track without ever exceeding a tracking error threshold of 0.5 at any given time step. Because clips vary greatly in duration, we only evaluate success rate across most 100 steps per clip by truncating longer clips.
Tracking error is defined as the mean distance between all joints and bodies in the humanoid wrt. the corresponding joints and bodies in the reference clip, evaluated on per-time-step basis. Our definition of tracking error is consistent with that of Hasenclever et al. (2020) and Tassa et al. (2018).
CoMic score is the mean episode return achieved by the tracking policy defined by the reward function proposed in CoMic (Hasenclever et al., 2020). We compute the CoMic score over all clips the dataset, truncating clips a maximum of 100 steps to prevent long clips from skewing results.
C SYSTEM REQUIREMENTS
Detailed system requirements for our method and baselines are included Table 3.
15

---

Published as a conference paper at ICLR 2025
Table 2. Low-level tracking quality. We report three additional metrics of tracking quality for the low-level tracking agent: success rate as defined by Luo et al. (2023), average tracking emror across all 836 clips, and CoMic (Hasenclever et al., 2020) reward across all clips.
Success rate (%) t
Tracking error | CoMic score 1
Offline only
6.2
0.503
42.6
5% data
74.4
0.260
45.4
25% data
79.6
0.225
46.3
75% data
87.9
0.202
47.4
Ours
88.3
0.187
48.7
Table System requirements. Training wall-time, inference time, and GPU memory requirements for Puppeteer, TD-MPC2, and SAC on a single NVIDIA GeForce RTX 3090 GPU. Overall, wall- time and system requirements of Puppeteer are mostly comparable to that of TD-MPC2 across both state-based and visual RL. SAC runs approx. 3.6x faster than Puppeteer, but does not achieve any meaningful performance. We exclude replay buffer memory requirements for elarity, but note that IM transitions require 14.1 GB memory for visual RL and 1.0 GB memory for baste RL. We store the replay buffer in GPU memory such that CPU and RAM usage is negligible.
Wall-time
Inference time (ms step)
GPU memory (GB)
(h/ / 1M steps)
Puppeteer
21.8 (vision: 29.0)
88.2
0.6
TD-MPC2
18.6 (vision: 25.2)
50.8
0.5
DreamerV3
50.2
18.7
3.9
SAC
5.9
2.2
0.4
16

---

Published as a conference paper at ICLR 2025
D IMPLEMENTATION DETAILS
MoCap dataset. We use the "small" offline dataset provided by MoCapAct (Wagener et al., 2022), which available https: / /mi crosoft github......... io 1o/MoCapAct. This dataset contains 20 noisy expert rollouts from each of 836 expert policies trained to track individual MoCap clips, totalling (suboptimal) 16,720 trajectories. Trajectories are variable length and are labelled with the CoMiC (Hasenclever et al., 2020) tracking reward which we use throughout this work. We solely use this dataset during (rre)training of the low-level tracking agent: the high-level puppeteering agent trained independently of the tracking agent using only online interaction data and task rewards.
Puppeteer. We base our implementation off of TD-MPC2 and use default design choices and hy- perparameters whenever possible. We experimented with alternative hyperparameters but did not observe any benefit in doing so, All hyperparameters are listed in Table 5, Our approach introduces only two new hyperparameters compared to prior work: loss coefficient for termination prediction (because our task suite has termination conditions; we add this to the TD-MPC2 baseline as well), and the number of low-level steps to take per high-level step (temporal abstraction).
TD-MPC2. We use the official implementation available at https: / com/ but modify the implementation to support multi-modal observations
nick walaasaaannnndmngg and termination conditions as discussed in Section 3. We empirically observe that TD-MPC2 de- generates highly unrealistic behavior without contact-based termination condition.
SAC. We benchmark against the implementation from (Yarats & Kostrikov, 2020) due its strong performance
comm com/
cenisyarats, wettttssytttcchsstt on lower-dimensional DMControl tasks as well as its popularity among the community. We modify the implementation to support early termination. We experiment with a variety of design choices and hyperparameters as we find vanilla SAC to suffer from numerical instabilities on our task suite (presumably due to high-dimensional observation and action spaces), but are unable to achieve non-trivial performance. The ablation in Figure 8 (hierarchical planning) strongly suggests that planning is a key driver of performance in Puppeteer and TD-MPC2, while SAC model-free method incapable of planning. Design choices and hyperparameters that we experimented with are as follows:
Table 4. List of SAC design choices and hyperparameters. We experiment with variety of design choices and hyperparameters, but find that they all fail to achieve non-trivial performance.
Design choice
Values
Number of Q-functions
2,5
TD-target
Default, REDQ (Chen et al., 2021) ReLU, Mish, LayerNorm Mish
Activation
MIP dim
256, 512, 1024
Batch size
256, 512
Learning rate
3 10 1 10-3
DreamerV3, We use the official implementation available https://gitte /gggttttthhg/g. com/ and use the default hyperparameters recommended for proprioceptive DMControl tasks. A key selling point of DreamerV3 is its robustness hyperparameters across tasks (relative to SAC), but we find that DreamerV3 does not achieve any non-trivial performance on our task suite. While D3eamerV3 is a model-based algorithm, it does not use planning, which the ablation in Figure 8 (hierarchical planning) finds to be key driver of performance in Puppeteer and TD-MPC2.
17

---

Published as a conference paper at ICLR 2025
Table 5. List of hyperparameters. We use the same hyperparameters across all tasks, levels (high- level and low-level), and across both Puppeteer and TD-MPC2 when applicable. Hyperparameters unique to Puppeteer are highlighted
Hyperparameter
Value
Planning
Horizon (H)
3
Iterations
Population size
512
Policy prior samples
Number of elites
64
Temperature
0.5
Low-level steps per high-level step
Policy prior
Log std. min. Log std. max.
-10
Replay buffer
Capacity Sampling
1, 000,000000 000 Uniform
Architecture
Encoder dim
256
MLP dim
512
Latent state dim Activation
512
LayerNorm + Mish
Number of Q-functions
Optimization
Update-to-data ratio Batch size Joint-embedding coef.
256
20
Reward prediction coef.
0.1
Value prediction coef.
0.1
Termination prediction coef.
0.1
Temporal coef. (A)
0.5
Q-fn. momentum coef.
0.99
Policy prior entropy coef. Policy prior loss norm.
x 10-4
Moving (5%, 95%) percentiles
Optimizer
Adam
Learning rate
10-4
Encoder learning rate
1 x 10-4
Gradient clip norm
Discount factor
0.97
Seed steps
2,500
18

---

Published as a conference paper at ICLR 2025
E
TASK SUITE
We propose a benchmark for visual whole-body humanoid control based on the "CMU Humanoid" model from DMControl (Tassa et al., 2018). Our simulated humanoid has 56 fully controllable joints (A E IR, and includes both head, hands, and feet. Actions are normalized to be in [-1, 1]. Our task suite consists of rssion-conditioned whole-body locomotion tasks (corridor, hurdles, walls, gaps, stairs), as well as 3 tasks that use proprioceptive information only (stand, walk, run). All 8 tasks are illustrated in Figure 4.
Observations always include proprioceptive information, as well as either visual inputs (high-level agent) or a command (low-level agent). The proprioceptive state vector 212-dimensional and consists of relative joint positions and velocities, body velocimeter and accelerometer, gyro, joint torques, binary touch (contact) sensors, and orientation relative to world z-axis. Visual inputs are raw 64 x 64 RGB images captured by third-person camera (as seen in Figure 4) without any preprocessing steps, and tracking commands are 15-dimensional vectors (corresponding to 5 points in 3D space) with values in [-1,1].
Downstream task reward functions are based on the humanoid reward functions in DMControl with minimal modification to fit our higher DoF embodiment. All 5 visual tasks the same reward function, which is proportional to forward velocity of the humanoid and is bounded to always be non-negative:
R(s) - clip(Iinvelz, [0, Vhurget))
where linvel, is linear velocity along the æ-axiss, and the clip operator bounds the reward value to always be non-negative and at most that of target velocity Uarget which set 6 in all tasks. The 3 proprioceptive tasks use a similar reward function, except that the agent rewarded for velocity in any -dirreccion,, and has an additional term that encourages an upright pose:
(4) where a is a constant coefficient balancing the two reward terms, and headpos_ is the height of the humanoid head in the world frame. The additional height reward term is adopted from the stand, walk, and run run tasks that DMControl implement with a simplified humanoid model (A R221). We find that the TD-MPC2 baseline produces very unrealistic behaviors without the additional reward term, so we choose to keep the term to make comparison more fair.
R(s) = min miinnliieelyyll Uharger) headpos
F
USER STUDY
To compare the "naturalness" of policies learned by our method vs. TD-MPC2, we design user study in which humans are asked to watch short (~10s) pairs clips simulated humanoid motions generated for each of our 5 visual whole-body humanoid control tasks. Each user is presented with 2 such pairs per task, totalling 10 pairs per user. Sample clips used in the user study are available at com/ #lpuppeteer,
https: : // www. nick coickkassle wallaahaasee
well as in Appendix A. Pairs generated by converged Puppeteer and TD-MPC2 agents. We generate 5 rollouts per task for each of two separately trained agents (random seed and 2) using the same method (ie., Puppeteer or TD-MPC2), and select the clips with median episode return for each of the two random seeds. We use clips generated by two unique random seeds to ensure that diversity in behavior due to inter-seed variability is captured in the user study, and we select the median clip to ensure that we neither favor nor disadvantage a method due to outliers. The concrete instructions provided users in the study are as follows:
Instructions
In this study, you will watch pairs of short (~10 seconds) clips of simulated humanoid motions. For each pair, you are asked to determine which of the two clips appear more "natural" and "human-like" to you, i.e., which clip looks more like the behavior of a real human.
Users are then provided with each of the 10 pairs of clips, and prompted to answer questions of the form:
19

---

Published as a conference paper at ICLR 2025
Q4: Which of the following two motions appear more "natural" and "human-like??
LEFT is more natural
RIGHT is more natural
LEFT and RIGHT are equally natural
Figure 10. Screenshot of a question from the user study. Users are shown two clips side side-by-side and are asked to provide their preference.
Q1: Which of the following two motions appear more "natural" and Sluman-like"
1. + LEFT is more natural
2. + RIGHT is more natural
3. LEFT and RIGHT are equally natural
The order of clips selected random for each pair. Aggregate results from the user study are provided in Table 6, and Figure 10 shows a sample question from the user study. Participants are sourced from undergraduate and graduate student populations across multiple universities and disciplines on volunteer basis. We do not collect personal or otherwise identifiable information about participants, and all participants have provided written consent to use of their responses for the purposes of this study.
20

---

Published as a conference paper at ICLR 2025
Table Results from the user study. We summarize results from our user study (n 51) be- low by reporting per-pair aggregate numbers. Higher is better t. Clips generated by our method, Puppeteer, are considered more natural by super-majority of participants.
Pair
TD-MPC2
Equal
Ours
Corridor
Pair 1 Pair
0
0
51
Hurdles
Pair 1
0
51
Pair 2
0
Walls
Pair
1
48
Pair 2
0
0
51
Gaps
Pair
Pair 2
0
51
Stairs
Pair
0
49
Pair 2
1
47
Aggregate
0.4%
1.8%
97.8%
21


Reviews:
[{"review_id": "AKvZS3Lu4W", "reviewer": "Reviewer_HShf", "summary": "The paper proposes a hierarchical world model for whole-body humanoid control based on RL. The framework separates high-level and low-level control, with a high-level puppeteering agent providing commands for a pre-trained low-level tracking agent, which executes detailed joint movements. Key contributions include a task suite for visual humanoid control, a hierarchical control model using RL without pre-defined reward designs, metrics for \"naturalness\" in motion, and thorough analysis through ablation studies and user preference tests.", "questions": "All my questions are listed in the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The hierarchical world model, which integrates high-level visual guidance with low-level proprioceptive control, is novel in its simplicity and efficacy, especially in achieving natural motion without predefined rewards or skill primitives.\n\n2. Puppeteer advances visual whole-body humanoid control by setting new standards for naturalness and efficiency in motion synthesis. The zero-shot generalization to unseen tasks demonstrates the model’s potential for practical application.", "weaknesses": "1. Lack of low-level tracking performance evaluation. There is no evaluation or metrics for the tracking accuracy of success rate. There are several works both from simulated avatars community [1,2] and real-world humanoids [3,4] that evaluate the tracking performance. I am supurised that these works are not mentioned and their metircs are not used for evaluation in this work.\n\n\n[1] Luo, Z., Cao, J., Kitani, K., & Xu, W. (2023). Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10895-10904).\n\n[2] Won, J., Gopinath, D., & Hodgins, J. (2022). Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 41(4), 1-12.\n\n[3] Cheng, X., Ji, Y., Chen, J., Yang, R., Yang, G., & Wang, X. (2024). Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796.\n\n[4] He, T., Luo, Z., Xiao, W., Zhang, C., Kitani, K., Liu, C., & Shi, G. (2024). Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436.\n\n\n2. The lack of interface design discuss. The paper proposes to use high-level controller to generate positions of tracking keypoints. This might be one way for reuse the low-level skills for downstream tasks, but there are many existing designs in prior works [5] [6] that not are compared in this work. To me, the idea of training low-level tracking policy for skills reuse is a long-standing idea, but the interface of this hierarchy matters a lot. I'd love to see more comparison on this.\n\n[5] Tessler, C., Kasten, Y., Guo, Y., Mannor, S., Chechik, G., & Peng, X. B. (2023, July). Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings (pp. 1-9).\n\n[6] Luo, Z., Cao, J., Merel, J., Winkler, A., Huang, J., Kitani, K., & Xu, W. (2023). Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582.\n\n3. The source of naturalness is unclear. The low-level tracking policy might be conditioned on human motion pirors, but if the tracking policy is good enough, it should be able to produce what TD-MPC2 achieves. Also, if the advantage of this paper is sample-efficiency and naturalness, a key baseline here would TD-MPC2+ AMP, which is missing. That being said, all the experimental results make sense to me, but the key comparison experiments are missing somehow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6FD294tdRJ", "reviewer": "Reviewer_C5cF", "summary": "This paper explores the high-dimensional humanoid control from visual observations. Specifically, the proposed approach is based on two RL-trained agent models.  High-level agent generates reference trajectories from visual observations. Low-level agent focuses on tracking these trajectories using current low-dimensional state information. The proposed method demonstrated enhanced natural motion control of a 56-DoF simulated humanoid, outperforming baseline models according to experimental results and a user study.", "questions": "1. How relevant is the metric of \"naturalness\" in real-world humanoid control, and is it sufficient to evaluate humanoid trajectory tracking effectively?\n2. It would be beneficial to include a comparative study on the survival rate or survival time when using the final-trained model\n3. It would be helpful to include baseline experiments focused on \"zero-shot generalization.\"\n4. Please provide details on memory usage, training time, and control (or inference) time across different methods.\n5. Is the low-level tracking effectively transferred to control different humanoid models with varying degrees of freedom?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research addresses a significant and practical challenge in generalist agents: controlling a humanoid agent from visual observations using generalizable world models.\n2. The methodology involves training a low-level agent on trajectory tracking that is adaptable across a range of control tasks, showing promising generalizability.\n3. A high-level agent controls the humanoid from visual observations, a task-specific but broadly applicable approach in real-world scenarios.\n4. A user study validates that the proposed method enables more natural humanoid control, which is preferred by participants.", "weaknesses": "1. The evaluation heavily relies on the \"naturalness\" of movements, which depends on subjective human judgments of what is considered \"human-like.\" This criterion, while important, may not fully evaluate the feasibility of such motions in actual humanoid robots, which face different kinematic and dynamic constraints than humans.\n2. Based on Figure 5, the episodic return of the baseline TD-MPC2 is comparable or superior to the proposed method across most tasks. It would be beneficial to evaluate other performance metrics such as survival rates or survival times on the final-trained model to provide a more comprehensive evaluation.\n3. The paper claims \"Zero-shot generalization to larger gap lengths,\" yet does not compare these results with baseline methods. Including comparative generalization data for the baseline TD-MPC method would strengthen claims of superior generalization.\n4. Minor issue:\na) Resource Efficiency: The two level agents training approach might require significantly more time and resources than single-agent baselines. Comparing memory usage, training duration, and inference times across methods would provide critical insights into the practicality of the proposed method.\nb) Model Reusability: The low-level tracking agent is described as reusable across tasks but it is unclear if this model is applicable only to 56-DoF humanoids or if it can be adapted to different control dimensions.\nc) There is a typo in the Problem Formulation section (page 2). The environment transition function should be denoted as L, not S.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTtHx9saUR", "reviewer": "Reviewer_2bf6", "summary": "Post rebuttal | the authors did a good job in the rebuttal phase to include some necessary baselines.\n\nThe paper improved a lot during the rebuttal phase and now includes some of the necessary baselines to validate the method. When I asked the authors about measures to ensure fairness in the new baseline experimentation, they argued that the considered MBRL algorithms are known to be robust to hyperparameter changes, which is true but not a valid argument for a simple reason: if an algo is robust to hyperparameter choice, doesn't mean that it cannot benefit from tuning. As I believe the proposed method was subject to some tuning during development time (like any ML method is), measures to ensure fairness of comparison would be necessary. Additionally, the baselines added during rebuttal just scratch the surface of hierarchical RL approaches and are themselves made up baselines combining known high-level algorithms with low-level TD-MPC. While these baselines are very important and necessary to understand the method, including standard baselines from the literature would strongly improve the paper and increase its impact. I highly encourage the authors to do that. I will be raising my score but only to a 5, as I still think the paper is not ready for publication at ICLR, despite the good improvements.\n\n---\n\nThe paper proposes \"Puppeteer\" a hierarchical decision-making approach tailored for visual whole-body humanoid control. The proposed method trains two separate world models for high-level and low-level control purposes. The low-level world model is concerned with tracking reference joint-level trajectories produced by the high-level controller. The high-level controller can additionally be conditioned on visual data. Both world models are based on TD-MPC2 which is a sampling-based MPC approach with learned decoder-free world models (with deterministic-only components) and a learned value function for long-horizon value assignment. TD-MPC2 is further extended to include a termination encoder head as is common in other model-based RL methods such as dreamer. The paper claims that the proposed method achieves results that are mostly comparable to TD-MPC2's results, while the plots show significantly worse results than TD-MPC2 in terms of asymptotic performance. The main advantage of the method is that it produces more natural and human-like motion, which was quite well shown in the experiments. The paper also ablates multiple design choices.", "questions": "- the main advantage of this method over TD-MPC2 is the resulting naturalness of the motions. Can the authors elaborate on why they think the proposed method improves this aspect? (here I mean further explaining the reward hacking argument made in the paper and perhaps including other arguments that could make sense)\n- can the authors elaborate on how the low-level policies exactly track the high-level commands? Since the low-level receives a sequence of commands does it keep using $c_t$ until the tracking error is below a threshold, or is it only used for a single step independent of the outcome of applying the one-step action?\n- on the methods side of things, the paper extends td-mpc2 to a hierarchical architecture. Can the authors compare the method to the hierarchical version of Dreamer [1]?\n\n[1] Hafner, Danijar, et al. \"Deep hierarchical planning from pixels.\" Advances in Neural Information Processing Systems 35 (2022)", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- the paper is generally well-written and an enjoyable read, I also liked the figures and plots.\n- the proposed approach is very interesting and promising and is a natural next step to extend the TD-MPC2 framework.\n- the method is evaluated on multiple humanoid tasks including environments with only proprioception as well as others with additional visual observations.\n- the ablations nicely evaluate the role of the different design choices of the method. I especially appreciate the study of the role of planning in the architecture.\n- the baselines include model-free and model-based approaches.", "weaknesses": "- the method section misses a detailed motivation for why hierarchy improves the naturalness of the motions.\n- the method section misses a detailed explanation concerning the exact usage of the high-level commands (see question 2).\n- the paper introduces a hierarchical version of td-mpc2, the baselines however do not include a single hierarchical RL approach, I would at least consider including a hierarchical implementation of dreamer [1].\n- [main weakness] The results of the paper are weak, at least in the current way in which they are presented. While the method is interesting and makes sense, the results show that it significantly underperforms TD-MPC2 but improves the naturalness of the produced motions. That would have been an acceptable tradeoff if the paper could justify why the proposed method improves the naturalness of the motions with intuitions and ideally, some experiments that validate them.\n\n** Minor issues:**\n\n- line 079 end-effector joints --> end-effector links.\n- punctuation is missing in the equations (but I understand that this is a matter of style, so no pressure).\n\nOverall the paper proposes an interesting approach, but it currently fails to showcase the benefits of this approach. I am willing to raise my score if this aspect is properly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hkHhHGBNgB", "reviewer": "Reviewer_EisB", "summary": "The paper presents a novel hierarchical world model Puppeteer designed for visual whole-body humanoid control, which operates without relying on predefined skill primitives, reward designs, or temporal abstractions. This paper also introduces a new task suite consisting of eight challenging tasks for evaluating humanoid control, demonstrating that Puppeteer produces more natural and human-like motions preferred by human evaluators in comparison with model-free and model-based RL baselines.", "questions": "* Although the motion produced by Puppeteer is more natural, why does the humanoid robot always lean forward while moving?\n* Can the proposed method be validated in real-world experiments?\n* Can this framework be generalized to manipulation tasks in HumanoidOlympics?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is well-organized, well-written, and includes clear figures.\n* A well-designed hierarchical control framework (although the idea is not novel) is implemented to control humanoid motion in a more natural way: the high-level agent generates commands given visual observations, and the low-level agent is responsible for executing them.\n* The proposed visual whole-body high-dimensional humanoid control benchmark enrich the evaluation platforms in the area.", "weaknesses": "* The paper primarily evaluates the visio-locomotive capabilities of the humanoid model. It could be better to expand the range of tasks to include more diverse scenarios that test different aspects of humanoid capabilities.\n* More baseline method should be compared, like HumanoidOlympics (https://arxiv.org/pdf/2407.00187). This paper also uses human motion data and reinforcement learning to train natural humanoid motions in various tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)