Hierarchical World Models as Visual Whole-Body Humanoid Controllers

## Abstract

Abstract Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans.

## 1 Introduction

Learning a generalist agent in the physical world is a long-term goal of many researchers in AI. Among variant agent designs, humanoids stand out as versatile platforms capable of performing a wide range of tasks, by integrating whole-body control and perception. However, this is a very challenging problem due to the high-dimensional nature of the observation and action spaces, as well as the complex dynamics of a bipedal embodiment, and it makes learning successful yet natural whole-body controllers with reinforcement learning (RL) extremely difficult. For example, consider the task shown in Figure 1, where a humanoid is rewarded for forward progress while jumping over gaps. To succeed in this task, a humanoid needs to accurately perceive the position and length of oncoming floor gaps, while carefully coordinating full body motions such that it has sufficient momentum and range to reach across each gap.

Due to the sheer complexity of such problems, prior work choose to make simplified assumptions, such as using low-dimensional (privileged) observations and actions (Heess et al., ; Peng et al., ; Wagener et al., ; Jiang et al., ), or (learned) skill primitives (Merel et al., , ; Hasenclever et al., ; Cheng et al., ). While these approaches are able to solve a variety of tasks to some extent, we expect to find a solution that is entirely data-driven and relies on as few assumptions as possible. In this paper, we propose to directly learn a controller on high-dimensional humanoid robot joints, leveraging model-based RL with an existing large-scale motion capture (MoCap) dataset (CMU, ).

We propose a data-driven RL method for visual whole-body control that produces natural, human-like motions and can solve diverse tasks. Our approach, dubbed Puppeteer, is a hierarchical JEPA-style (LeCun, ) world model that consists of two distinct agents: a proprioceptive *tracking* agent that tracks a reference motion via joint-level control, and a visual *puppeteer* agent that learns to perform downstream tasks by synthesizing lower-dimensional reference motions for the tracking agent to track based on visual observations.

Concretely, the two agents are trained independently in two separate stages using the model-based RL algorithm TD-MPC2 (Hansen et al., ) as a learning backbone. First, a *single* tracking world model is (pre)trained to track reference motions from pre-existing human MoCap data (CMU, ) re-targeted to a humanoid embodiment (Wagener et al., ). It learns a single model to convert any reference kinematic motion to physically executable actions. This is a departure from previous work that often learns multiple low-level models (Merel et al., ; Hasenclever et al., ). Importantly, this tracking agent can be saved and reused across all downstream tasks. In the second stage, we train a puppeteering world model that takes visual observation as inputs and outputs the reference motion for the tracking agent based on the specified downstream task. The puppeteer agent is trained with online environment interaction using the fixed tracking agent. A key feature of our framework is its striking simplicity: both world models are algorithmically identical (but differ in inputs/outputs) and can be trained using RL without any bells and whistles. Different from a traditional hierarchical RL setting, our puppeteer agent (high-level policy) outputs geometric locations for a small number of end-effector joints instead of a goal embedding. The tracking agent (low-level policy) is thus only required to learn joint-level physics. This makes the tracking agent easily sharable and generalizable across tasks, leading to an overall small computational footprint.

To evaluate the efficacy of our approach, we propose a new task suite for visual whole-body humanoid control with a simulated 56-DoF humanoid, which contains a total of 8 challenging tasks. We show that our method produces highly performant control policies across all tasks compared to a set of strong model-free and model-based baselines: SAC (Haarnoja et al., ), DreamerV3 (Hafner et al., ), and TD-MPC2 (Hansen et al., ). Furthermore, we find that motions generated by our method are broadly preferred by humans in a user study with $46$ participants. We conclude the paper by carefully dissecting how each of our design choices influence results. Code for method and environments is available at [https://nicklashansen.com/rlpuppeteer](https://nicklashansen.com/rlpuppeteer). Our main contributions can be summarized as follows:

- •
Task suite. We propose a new, challenging task suite for visual whole-body humanoid control with a simulated 56-DoF humanoid. The task suite has 8 tasks in total.
- •
Hierarchical world model. We propose a method for high-dimensional continuous control that leverages a learned hierarchical world model for planning.
- •
Evaluating “naturalness” of controllers. We develop several metrics for quantifying how natural and human-like generated motions are across tasks in our suite, including human preferences from a user study.
- •
Analysis & ablations. We carefully ablate each of our design choices, analyze the relative importance of each component, and provide actionable advice for future work in this area.

## 2 Preliminaries

Problem formulation. We model visual whole-body humanoid control as a reinforcement learning problem governed by an episodic Markov Decision Process (MDP) characterized by the tuple $(\mathcal{S},\mathcal{A},\mathcal{T},R,\gamma,\Delta)$ where $\mathbf{s}\in\mathcal{S}$ are states, $\mathbf{a}\in\mathcal{A}$ are actions, $\mathcal{S}\colon\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S}$ is the environment transition (dynamics) function, $R\colon\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is a scalar reward function, $\gamma$ is the discount factor, and $\Delta\colon\mathcal{S}\mapsto\{0,1\}$ is an episode termination condition. We implicitly consider both proprioceptive information $\mathbf{q}$ and visual information $\mathbf{v}$ as part of states $\mathbf{s}$ and will only make the distinction clear when necessary. We aim to learn a policy $\pi\colon\mathcal{S}\mapsto\mathcal{A}$ that maximizes discounted sum of rewards in expectation: $\mathbb{E}_{\pi}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right],~{}r_{t}=R(\mathbf{
s}_{t},\pi(\mathbf{s}_{t}))$ for an episode of length $T$, while synthesizing motions that look *“natural”*. We informally define natural motions as policy rollouts that are human-like, but develop several metrics for measuring the “naturalness” of policies in Section 4.

TD-MPC2. We build upon the model-based reinforcement learning (MBRL) algorithm TD-MPC2 (Hansen et al., ), which represents the state-of-the-art in continuous control and has been shown to outperform alternatives in tasks with high-dimensional action spaces (Hansen et al., , ; Sferrazza et al., ). Specifically, TD-MPC2 learns a latent decoder-free world model from environment interactions and selects actions by planning with the learned model. All components of the world model are learned end-to-end using a combination of joint-embedding prediction (Grill et al., ), reward prediction, and temporal difference (Sutton, ) losses, *without* decoding raw observations. During inference, TD-MPC2 follows the Model Predictive Control (MPC) framework for local trajectory optimization using Model Predictive Path Integral (MPPI) (Williams et al., ) as a derivative-free (sampling-based) optimizer. To accelerate planning, TD-MPC2 additionally learns a model-free policy prior which is used to warm-start the sampling procedure.

Figure: Figure 2: Approach. We pretrain a tracking agent (world model) on human MoCap data using RL; this agent takes proprioceptive information $\mathbf{q}_{t}$ and an abstract reference motion (command) $\mathbf{c}_{t}$ as input, and synthesizes $H$ low-level actions that tracks the reference motion. We then train a high-level puppeteering agent on downstream tasks via online interaction; this agent takes both state $\mathbf{q}_{t}$ and visual information $\mathbf{v}_{t}$ as input, and outputs commands for the tracking agent to execute.
Refer to caption: x2.png

## 3 A Hierarchical World Model for High-Dimensional Control

We aim to learn highly performant and *“natural”* policies for visual whole-body humanoid control in a data-driven manner using hierarchical world models. A key strength of our approach is that it can synthesize human-like motions without any explicit domain knowledge, reward design, nor skill primitives. While we focus on humanoid control due to their complexity, our approach can in principle be applied to any embodiment. Our method, dubbed Puppeteer, consists of two distinct agents, both of which are implemented as TD-MPC2 world models (Hansen et al., ) and trained independently. Figure 2 provides an overview of our method. The two agents are designed as follows:

- 1.
A low-level *tracking* agent that takes a robot proprioceptive state $\mathbf{q}_{t}$ and an abstract command $\mathbf{c}_{t}$ as input at time $t$, and uses planning with a learned world model to synthesize a sequence of $H$ control actions $\mathbf{\{\mathbf{a}_{t},\mathbf{a}_{t+1},\dots,\mathbf{a}_{t+H}\}}$ that (approximately) obeys the abstract command.
- 2.
A high-level *puppeteering* agent that takes the same robot proprioceptive state $\mathbf{q}_{t}$ as input, as well as (optionally) auxiliary information and modalities such as RGB images $\mathbf{v}_{t}$ or task-relevant information, and uses planning with a learned world model to synthesize a sequence of $H$ high-level abstract commands $\mathbf{\{\mathbf{c}_{t},\mathbf{c}_{t+1},\dots,\mathbf{c}_{t+H}\}}$ for the low-level agent to execute.

A unique benefit of our approach is that *a single tracking world model can be (pre)trained and reused across all downstream tasks*. The tracking and puppeteering world models are algorithmically identical (but differ in inputs/outputs), and consist of the following 6 components:

$$ $\begin{array}[]{lll}\text{Encoder}&\mathbf{z}=h(\mathbf{s})&\color[rgb]{ .5,.5,.5}{\vartriangleright\text{Encodes state into a latent embedding}}\\ \text{Latent dynamics}&\mathbf{z}^{\prime}=d(\mathbf{z},\mathbf{a})&\color[rgb ]{.5,.5,.5}{\vartriangleright\text{Predicts next latent state}}\\ \text{Reward}&\hat{r}=R(\mathbf{z},\mathbf{a})&\color[rgb]{.5,.5,.5}{ \vartriangleright\text{Predicts reward $r$ of a state transition}}\\ \color[rgb]{0.90234375,0.5703125,0.22265625}{\textbf{Termination}}&\color[rgb] {0.90234375,0.5703125,0.22265625}{\hat{\delta}=D(\mathbf{z},\mathbf{a})}& \color[rgb]{0.90234375,0.5703125,0.22265625}{\vartriangleright\textbf{Predicts probability of termination}}\\ \text{Terminal value}&\hat{q}=Q(\mathbf{z},\mathbf{a})&\color[rgb]{.5,.5,.5}{ \vartriangleright\text{Predicts discounted sum of rewards}}\\ \text{Policy prior}&\hat{\mathbf{a}}=p(\mathbf{z})&\color[rgb]{.5,.5,.5}{ \vartriangleright\text{Predicts an action $\mathbf{a}^{*}$ that maximizes $Q$} }\end{array}$ (1) $$

where $\mathbf{z}$ is a latent state. Because we consider episodic MDPs with termination conditions, we additionally add a termination prediction head $D$ (highlighted in Equation 1) that predicts the probability of termination conditioned on a latent state and action. Use of termination signals in the context of planning with a world model requires special care and has, to the best of our knowledge, not been explored in prior work; we introduce a novel method for this in Section 3.3. In the following, we describe the two agents and their interplay in the context of visual whole-body humanoid control. Minor implementation details are provided in Appendix B.

Figure: Figure 3: MoCap tracking. The low-level tracking agent is trained to track relative end-effector (head, hands, feet) positions of sampled reference motions in 3D space.
Refer to caption: x3.png

### 3.1 Low-Level Tracking World Model

We first train the low-level tracking world model independently from the high-level agent and any potential downstream tasks. We leverage pre-existing human MoCap data (CMU, ) re-targeted to the 56-DoF “CMU Humanoid” embodiment (Tassa et al., ) during training of the tracking model, which (as we will later show empirically) implicitly encodes human motion priors. Specifically, we train our tracking world model by sampling $(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1},\dots,r_{H})$ sequences from MoCapAct (Wagener et al., ), an offline dataset that consists of noisy, suboptimal rollouts from existing policies trained to track reference motions (836 MoCap clips). This is in contrast to prior literature that learn per-clip policies or skill primitives (Heess et al., ; Merel et al., ; Hasenclever et al., ).

Observations include humanoid proprioceptive information $\mathbf{q}_{t}$ at time $t$, as well as a reference motion (command) $\mathbf{c}_{t}$ to track. During training of the tracking policy, we let $\mathbf{c}_{t}\doteq(\mathbf{q}^{\textrm{ref}}_{t+1\dots t+H})$ where each $\mathbf{q}^{\textrm{ref}}$ corresponds to relative end-effector (head, hands, feet) positions of the sampled reference motion at a future timestep; during downstream tasks, we train the high-level agent to output (via planning) commands $\mathbf{c}$ for the low-level agent to track. Figure 3 illustrates our low-dimensional reference; the controllable humanoid tracks end-effector positions of a reference motion. We label all transitions using the reward function from Hasenclever et al. (). To improve state-action coverage of the tracking world model, we train with a combination of offline data and online interactions, maintaining a separate replay buffer for online interaction data and sampling offline/online data with a $50\%/50\%$ ratio in each gradient update as in Feng et al. (). We find this to be crucial for tracking performance when training a single world model on a large number of MoCap clips.

### 3.2 High-Level Puppeteering World Model

We now consider training a high-level puppeteering world model via online interaction in downstream tasks. As illustrated in Figure 2, the puppeteering model is trained (using downstream task rewards) to control the tracking model via commands $\mathbf{c}$, *i.e.*, we redefine commands to now be the action space of the puppeteering agent. The tracking world model remains frozen (no weight updates) throughout this process, which allows us to reuse the *same* tracking model across *all* downstream tasks. Because the high-level agent uses planning for action selection, it natively supports temporal abstraction by outputting multiple commands $(\mathbf{c}_{t},\mathbf{c}_{t+1},\dots,\mathbf{c}_{t+H})$ for the low-level agent to execute; we treat the number of low-level steps per high-level step as a hyperparameter $k$ that allows us to trade strong motion prior (large $k$) for control granularity (small $k$).

### 3.3 Planning with Termination Conditions

We consider episodic MDPs with termination conditions. In the context of humanoid control, a common such termination condition is non-foot contact with the floor. Use of termination conditions requires special care in the context of world model learning and planning, as both components are used to simulate (latent) multi-step rollouts. We extend the world model of TD-MPC2 with a termination prediction head $D$, which predicts the probability of termination at each time step. This termination head is trained end-to-end together with all other components of the world model using

$$ $\mathcal{L}_{\textrm{Puppeteer}}(\theta)\doteq\mathcal{L}_{\textrm{TD-MPC2}}( \theta)+\alpha\operatorname{CE}(\hat{\delta},\delta)$ (2) $$

where $\hat{\delta},\delta$ are predicted and ground-truth termination signals, respectively, $\operatorname{CE}$ is the (binary) cross-entropy loss, and $\alpha$ is a constant coefficient balancing the losses. We additionally truncate TD-targets at terminal states during training. It is similarly necessary to truncate model rollouts and value estimates during planning (at test-time). However, we only have access to predicted termination signals at test-time, which can be noisy and consequently lead to high-variance value estimates for latent rollouts. To mitigate this, we maintain a cumulative weighting (discount) of termination probabilities when rolling out the model (capped at 0 0), such that only a *soft* truncation is applied.

Figure: Figure 4: Tasks. We develop 5 visual whole-body humanoid control tasks with a 56-DoF simulated humanoid (bottom), as well as 3 non-visual tasks (top). See Appendix C for more details.
Refer to caption: extracted/5635202/visualizations/stand.png

## 4 Experiments

Our proposed method holds the promise of strong downstream task performance while still synthesizing natural and human-like motions. To evaluate the efficacy of our method, we propose a new task suite for whole-body humanoid control with multi-modal observations (vision and proprioceptive information) based on the “CMU Humanoid” model from DMControl (Tassa et al., ). Our simulated humanoid has 56 fully controllable joints ($\mathcal{A}\in\mathbb{R}^{56}$), and includes both head, hands, and feet. We aim to learn highly performant policies in a data-driven manner without the need for embodiment- or task-specific engineering (*e.g.*, reward design, constraints, or auxiliary objectives), while synthesizing natural and human-like motions. Code for method and environments is available at [https://nicklashansen.com/rlpuppeteer](https://nicklashansen.com/rlpuppeteer).

Figure: Figure 5: Learning curves. Episode return as a function of environment steps on all 8 tasks from our proposed task suite. Our method matches the performance of TD-MPC2 while producing more natural motions. We evaluate SAC and DreamerV3 only on proprioceptive tasks as they do not achieve any meaningful performance. Average of 3 random seeds; shaded area is $95\%$ CIs.
Refer to caption: x4.png

### 4.1 Experimental Details

Tasks. Our proposed task suite consists of 5 vision-conditioned whole-body locomotion tasks, and an additional 3 tasks without visual input. We provide an overview of tasks in Figure 4; they are designed with a high degree of randomization and include running along a corridor, jumping over hurdles and gaps, walking up the stairs, and circumnavigating obstacles (walls). All 5 visual control tasks use a reward function that is proportional to the linear forward velocity, while non-visual tasks reward displacement in any direction. Episodes are terminated at timeout ($500$ steps) or when a non-foot joint makes contact with the floor. We empirically observe that the TD-MPC2 baseline degenerates to highly unrealistic behavior without a contact-based termination condition.

Figure: Figure 6: Human preference in humanoid motions. Aggregate results from a user study ($n=46$) where humans are presented with pairs of motions generated by TD-MPC2 and our method, and are asked to provide their preference.
Refer to caption: x5.png

Implementation. We pretrain a single 5M parameter TD-MPC2 world model to track all 836 CMU MoCap (CMU, ) reference motions retargeted to the CMU Humanoid model. The tracking agent is trained for $10\textrm{M}$ steps using both offline data (noisy rollouts) from MoCapAct (Wagener et al., ) and online interaction with a new reference motion sampled in each episode. We sample $50\%$ of each batch from the offline dataset, and $50\%$ from the online replay buffer for each gradient update; we did not experiment with other ratios. The puppeteering agent is similarly implemented as a 5M parameter TD-MPC2 world model, which we train from scratch via online interaction on each downstream task. Observations include a $212$-d proprioceptive state vector and $64\times 64$ RGB images from a third-person camera. Both agents act at the same frequency, *i.e.*, we set $k=1$. Training the tracking world model takes approximately 12 days, and training the puppeteering world model takes approximately 4 days, both on a single NVIDIA GeForce RTX 3090 GPU.

Baselines. We benchmark our method against state-of-the-art RL algorithms for continuous control, including *(1)* widely used model-free RL method Soft Actor-Critic (SAC) (Haarnoja et al., ) which learns a stochastic policy and value function using a maximum entropy RL objective, *(2)* model-based RL method DreamerV3 (Hafner et al., , , ) which simultaneously learns a world model using a generative objective, and a model-free policy in the latent space of the learned world model, and *(3)* model-based RL method TD-MPC2 (Hansen et al., , ) which learns a self-predictive (decoder-free) world model and selects actions by planning with the learned world model. Both our method and baselines use the same hyperparameters across all tasks. We provide further implementation details in Appendix B.

Figure: Figure 7: Qualitative results. Our hierarchical approach, Puppeteer, produces natural human motions, whereas TD-MPC2 trained end-to-end often learns high-performing but unnatural gaits.
Refer to caption: extracted/5635202/visualizations/0421-hurdles/0.png

### 4.2 Main Results

We first present our main benchmark results, and then analyse and ablate each design choice.

Benchmark results. We evaluate our method, Puppeteer, and baselines on all 8 whole-body humanoid control tasks. Episode return as a function of environment steps is shown in Figure 5. We observe that the performance of our method is comparable to that of TD-MPC2 across all tasks (except *stairs*), whereas SAC and DreamerV3 does not achieve any meaningful performance within our computational budget of 3M environment steps. As we will soon reveal, TD-MPC2 produces better policies in terms of episode return on the *stairs* task, but far less natural behavior; this phenomenon is commonly referred to as *reward hacking* (Clark and Amodei, ; Skalse et al., ).

**Table 1: Proxies for “naturalness”. Evaluated on the *hurdles* task. *eplen* denotes the average episode length over the course of training; *height* is the average torso height (gait) at end of training. Mean and std. across 3 seeds.**
|  | eplen $\uparrow$ | height (cm) $\uparrow$ |
| --- | --- | --- |
| TD-MPC2 | $70.7\pm 5.5$ | $85.9\pm 4.7$ |
| Ours | $\mathbf{100.6\pm 1.0}$ | $\mathbf{96\pm 0.2}$ |

“Naturalness” of humanoid controllers. We conduct a user study ($n=46$) in which humans are shown pairs of short ($\sim$$10\textrm{s}$) clips of policy rollouts from TD-MPC2 and our method, and are asked to provide their preference. Participants are undergraduate and graduate students across multiple universities and disciplines. Results from this study are shown in Figure 6, and Figure 7 shows two sample clips from the study. While both methods perform comparably in terms of downstream task reward, a super-majority of participants rate rollouts from our method as more natural than that of TD-MPC2, with only $4\%$ of responses rating them as “equally natural” and $0\%$ rating TD-MPC2 as more natural. This preference is especially pronounced in the *stairs* task, where TD-MPC2 achieves a higher asymptotic return (higher forward velocity) but learns to “roll” up stairs as opposed to our method that walks. We also report several quantitative measures of naturalness in Table 1, which strongly support our user study results. These findings underline the importance of a more holistic evaluation of RL policies as opposed to solely relying on rewards. See Appendix A for more results.

### 4.3 Ablations & Analysis

Figure: Figure 8: Ablations. Normalized score for various ablations of Puppeteer during pretraining (*left*) and downstream tasks (*right*). Pretraining benefits from diverse data, as well as both pre-existing (offline) data and online interactions. We also observe that planning is critical to whole-body humanoid control. Mean across 3 seeds; downstream ablations are averaged across 5 tasks.
Refer to caption: x6.png

Figure: Figure 9: Zero-shot generalization to larger gap lengths. *(Left)* Visualization of gap lengths. Agent is trained on gaps $[0.1,0.4]$m and evaluated on gaps up to $1.2$m. *(Right)* Normalized performance as a function of gap length in the visual *gaps* task. Mean of 3 seeds. Our method achieves non-trivial performance on gaps up to $3\times$ the training data. CIs omitted for visual clarity.
Refer to caption: extracted/5635202/visualizations/gaps-length/0-1.png

We ablate each design choice of our method, including both the pretraining (tracking) and downstream task learning stages. Our experimental results are summarized in Figure 8.

Pretraining (tracking). Our method leverages both offline and online data during pretraining of the tracking world model. We ablate this training mixture in two distinct ways: *(i)* using only offline or online data, and *(ii)* reducing the number of unique MoCap clips seen during training. Interestingly, we find that leveraging both data sources leads to better tracking policies overall. We hypothesize that this is because offline data may help in learning to track especially difficult motions such as jumping and balancing on one leg, while online data improves state-action coverage and thus leads to a more robust world model overall. Similarly, training on more diverse MoCap clips also leads to better tracking performance. Training on all $836$ MoCap clips results in the best tracking world model, and we expect tracking to further improve with availability of more MoCap data.

Downstream tasks. We conduct three ablations that help us better understand the impact of a hierarchical approach to downstream tasks: *(i)* using a learned model-free policy in lieu of planning in either level of the hierarchy, *(ii)* pretraining of the *high-level* agent in addition to the low-level agent, and *(iii)* evaluating zero-shot generalization to unseen environment variations (gap length in the *gaps* task). The first two ablations are shown in Figure 8, and the latter is shown in Figure 9. We find that planning at both levels is critical to effective whole-body humanoid control, which we conjecture is due to the high dimensionality of the problem; this is supported by concurrent studies on high-dimensional continuous control with TD-MPC2 (Hansen et al., ; Sferrazza et al., ). Next, we pretrain agents on the *corridor* task and independently finetune on each visual control task. While the specific environments and motions differ between tasks, we find that our method benefits substantially from finetuning. We conjecture that this is because the need to control a low-level tracking agent is shared between all high-level agents, irrespective of the downstream task. Finally, we explore the zero-shot generalization ability of our method to harder, unseen variations of the *gap* task. Interestingly, we observe that our method generalizes to gap lengths up to $3\times$ the training data without additional training. In light of these results, we believe that further investigation of the generalization ability of hierarchical world models will be a promising direction for future research.

## 5 Related Work

Learning whole-body controllers for humanoids is a long-standing problem at the intersection of the machine learning and robotics communities. Humanoids are of particular interest to the learning community because of the high-dimensional nature of the problem (Heess et al., ; Merel et al., ; Peng et al., ; Merel et al., ; Hasenclever et al., ; Wagener et al., ; Shi et al., ; Caggiano et al., ; Sferrazza et al., ), and to the robotics community because it is a promising morphology for general-purpose robotic agents (Grizzle et al., ; Li et al., ; BostonDynamics, ; Unitree, ; Cheng et al., ). Prior work predominantly focus on learning control policies for individual tasks using model-free reinforcement learning algorithms, many of which train behavior cloning (BC) policies (Wagener et al., ) and/or skill encoders (Heess et al., ; Merel et al., ; Hasenclever et al., ; Shi et al., ) on human MoCap data (CMU, ) re-targeted to their considered humanoid embodiment. For example, Wagener et al. () learn expert policies for each reference MoCap clip in the dataset, and then subsequently distills them into a single policy via supervised learning. Instead, we train a single tracking world model across all clips in the MoCapAct (Wagener et al., ) dataset, using only noisy rollouts and offline RL to do so. Concurrent work, HumanoidBench (Sferrazza et al., ), similarly introduces a whole-body control benchmark using the less expressive Unitree H1 (Unitree, ) embodiment. Our contributions differ in two important ways: *(1)* we develop a method for synthesizing natural human motions with a highly expressive humanoid model, while Sferrazza et al. () focus on benchmarking existing methods for online RL, and *(2)* our proposed method additionally takes vision as input, whereas HumanoidBench considers tasks with privileged state information in their experiments.

World models (and model-based RL more broadly) are of increasing interest to researchers due to their strong empirical performance in an online RL setting (Ha and Schmidhuber, ; Hafner et al., ; Hansen et al., ), as well as their promise of generalization to structurally similar problem instances (Zhang et al., ; Zheng et al., ; Lee et al., ; LeCun, ; Sobal et al., ; Brohan et al., ). Existing model-based RL algorithms can broadly be categorized into algorithms that select actions by planning with a learned world model (Ebert et al., ; Schrittwieser et al., ; Ye et al., ; SV et al., ; Hansen et al., ), and algorithms that instead learn a model-free policy using imagined rollouts from the world model (Kaiser et al., ; Hafner et al., ). We build upon the TD-MPC2 (Hansen et al., ) world model, which uses planning and has been shown to outperform other existing algorithms for continuous control (Hansen et al., ; Lancaster et al., ; Feng et al., ; Sferrazza et al., ). In this work, we demonstrate that planning is key to success in the high-dimensional continuous control problems that we consider.

Hierarchical RL offers a framework for subdividing a complex learning problem into more approachable subproblems, often by, *e.g.*, leveraging (learned or manually designed) skill primitives (Pastor et al., ; Merel et al., , ; Shi et al., ) or facilitating learning over long time horizons via temporal abstractions (Nachum et al., ; LeCun, ; Hafner et al., ; Gumbsch et al., ; Chen et al., ). Our method, Puppeteer, is also hierarchical in nature, but does not rely on skill primitives nor temporal abstraction for task learning. Instead, we learn a *single* low-level world model that can be reused across a variety of downstream tasks, and instead introduce a hierarchy in terms of data sources and input modalities.

## 6 Conclusion

Puppeteer is a data-driven method for learning highly performant visual control policies for high-dimensional continuous control such as humanoids. We demonstrate that our method consistently produces motions that are considered natural and human-like by human evaluators compared to existing methods, and we find that our method is able to generalize zero-shot to gap lengths that far exceed the training data. These results are, to the best of our knowledge, unprecedented in the area of whole-body humanoid control. However, we acknowledge that our contributions have several limitations that may not be obvious:

- •
While our proposed task suite consists of challenging visual whole-body control tasks with a detailed humanoid model, tasks primarily evaluate the visio-locomotive capabilities of current methods. However, humanoids hold the promise of performing highly dexterous tasks (such as, *e.g.*, manipulating and carrying objects in unstructured environments alongside humans) which is not covered by our existing set of tasks. We expect development of new tasks will be increasingly important as algorithms continue to improve, and we hope that the release of our benchmark will help facilitate that.
- •
Our hierarchical approach currently consists of two levels, only one of which is pretrained in the majority of our experiments. Our experiment on high-level pretraining suggests that it can be beneficial to pretrain both levels, but we expect this conclusion to vary depending on the choice of source and target tasks (Hansen et al., ; Xu et al., ; Lin et al., ). Further research on how to pretrain and transfer the full hierarchical world model is warranted.

## Appendix A Additional Qualitative Results

[Uncaptioned image]: extracted/5635202/visualizations/0421-corridor/0.png

[Uncaptioned image]: extracted/5635202/visualizations/0421-gaps-alt/0.png

## Appendix B Implementation Details

MoCap dataset. We use the “small” offline dataset provided by MoCapAct [Wagener et al., ], which is available at [https://microsoft.github.io/MoCapAct](https://microsoft.github.io/MoCapAct). This dataset contains 20 noisy expert rollouts from each of 836 expert policies trained to track individual MoCap clips, totalling (suboptimal) 16,720 trajectories. Trajectories are variable length and are labelled with the CoMiC [Hasenclever et al., ] tracking reward which we use throughout this work.

Puppeteer. We base our implementation off of TD-MPC2 and use default design choices and hyperparameters whenever possible. All hyperparameters are listed in Table 2. Our approach introduces only two new hyperparameters compared to prior work: loss coefficient for termination prediction (because our task suite has termination conditions; we add this to the TD-MPC2 baseline as well), and the number of low-level steps to take per high-level step (temporal abstraction).

TD-MPC2. We use the official implementation available at [https://github.com/nicklashansen/tdmpc2](https://github.com/nicklashansen/tdmpc2), but modify the implementation to support multi-modal observations and termination conditions as discussed in Section 3.

SAC. We benchmark against the implementation from [https://github.com/denisyarats/pytorch_sac](https://github.com/denisyarats/pytorch_sac) due to its strong performance on lower-dimensional DMControl tasks as well as its popularity among the community. We modify the implementation to support early termination.

DreamerV3. We use the official implementation available at [https://github.com/danijar/dreamerv3](https://github.com/danijar/dreamerv3), and use the default hyperparameters recommended for lower-dimensional DMControl tasks.

**Table 2: List of hyperparameters. We use the same hyperparameters across all tasks, levels (high-level and low-level), and across both Puppeteer and TD-MPC2 when applicable. Hyperparameters unique to Puppeteer are highlighted.**
| Hyperparameter | Value |
| --- | --- |
| Planning |  |
| Horizon ($H$) | $3$ |
| Iterations | $8$ |
| Population size | $512$ |
| Policy prior samples | $24$ |
| Number of elites | $64$ |
| Temperature | $0.5$ |
| Low-level steps per high-level step | $1$ |
| Policy prior |  |
| Log std. min. | $-10$ |
| Log std. max. | $2$ |
| Replay buffer |  |
| Capacity | $1,000,000$ |
| Sampling | Uniform |
| Architecture |  |
| Encoder dim | $256$ |
| MLP dim | $512$ |
| Latent state dim | $512$ |
| Activation | LayerNorm + Mish |
| Number of $Q$-functions | $5$ |
| Optimization |  |
| Update-to-data ratio | $1$ |
| Batch size | $256$ |
| Joint-embedding coef. | $20$ |
| Reward prediction coef. | $0.1$ |
| Value prediction coef. | $0.1$ |
| Termination prediction coef. | $0.1$ |
| Temporal coef. ($\lambda$) | $0.5$ |
| $Q$-fn. momentum coef. | $0.99$ |
| Policy prior entropy coef. | $1\times 10^{-4}$ |
| Policy prior loss norm. | Moving $(5\%,95\%)$ percentiles |
| Optimizer | Adam |
| Learning rate | $3\times 10^{-4}$ |
| Encoder learning rate | $1\times 10^{-4}$ |
| Gradient clip norm | $20$ |
| Discount factor | 0.97 |
| Seed steps | 2,500 |

## Appendix C Task Suite

We propose a benchmark for visual whole-body humanoid control based on the “CMU Humanoid” model from DMControl [Tassa et al., ]. Our simulated humanoid has 56 fully controllable joints ($\mathcal{A}\in\mathbb{R}^{56}$), and includes both head, hands, and feet. Our task suite consists of 5 vision-conditioned whole-body locomotion tasks (corridor, hurdles, walls, gaps, stairs), as well as 3 tasks that use proprioceptive information only (stand, walk, run). All 8 tasks are illustrated in Figure 4.

Observations always include proprioceptive information, as well as either visual inputs (high-level agent) or a command (low-level agent). The proprioceptive state vector is $212$-dimensional and consists of relative joint positions and velocities, body velocimeter and accelerometer, gyro, joint torques, binary touch (contact) sensors, and orientation relative to world $z$-axis. Visual inputs are $64\times 64$ RGB images captured by a third-person camera (as seen in Figure 4), and commands are $15$-dimensional vectors with values in $[-1,1]$.

## Appendix D User Study

To compare the “naturalness” of policies learned by our method vs. TD-MPC2, we design a user study in which humans are asked to watch short ($\sim$$10$s) pairs of clips of simulated humanoid motions generated for each of our 5 visual whole-body humanoid control tasks. Each user is presented with $2$ such pairs per task, totalling $10$ pairs per user. The concrete instructions provided to users are as follows:

Instructions
In this study, you will watch pairs of short ( 10 seconds) clips of simulated humanoid motions. For each pair, you are asked to determine which of the two clips appear more "natural" and "human-like" to you, i.e., which clip looks more like the behavior of a real human.

Users are then provided with each of the $10$ pairs of clips, and prompted to answer questions of the form:

Q1: Which of the following two motions appear more "natural" and "human-like"?

- 1.
$\leftarrow$ LEFT is more natural
- 2.
$\rightarrow$ RIGHT is more natural
- 3.
LEFT and RIGHT are equally natural

The order of clips is selected at random for each pair. Aggregate results from the user study are provided in Table 3, and Figure 10 shows a sample question from the user study. Participants are sourced from undergraduate and graduate student populations across multiple universities and disciplines. We do not collect personal or otherwise identifiable information about participants, and all participants have provided written consent to use of their responses for the purposes of this study.

Figure: Figure 10: Screenshot of a question from the user study. Users are shown two clips side-by-side and are asked to provide their preference.
Refer to caption: extracted/5635202/visualizations/survey-screenshot.png

**Table 3: Results from the user study. We summarize results from our user study ($n=46$) below by reporting per-pair aggregate numbers. Higher is better $\uparrow$. Clips generated by our method, Puppeteer, are considered more natural by a super-majority of participants.**
| Pair | TD-MPC2 | Equal | Ours |
| --- | --- | --- | --- |
| Corridor |  |  |  |
| Pair 1 | 0 0 | 0 0 | $46$ |
| Pair 2 | 0 0 | $2$ | $44$ |
| Hurdles |  |  |  |
| Pair 1 | 0 0 | 0 0 | $46$ |
| Pair 2 | 0 0 | 0 0 | $46$ |
| Walls |  |  |  |
| Pair 1 | $1$ | $2$ | $43$ |
| Pair 2 | 0 0 | 0 0 | $46$ |
| Gaps |  |  |  |
| Pair 1 | 0 0 | 0 0 | $46$ |
| Pair 2 | 0 0 | 0 0 | $46$ |
| Stairs |  |  |  |
| Pair 1 | 0 0 | $2$ | $44$ |
| Pair 2 | $1$ | $3$ | $42$ |
| Aggregate | $0.4\%$ | $2.0\%$ | $97.6\%$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: All my questions are listed in the weakness part.
soundness: 3
strengths: 1. The hierarchical world model, which integrates high-level visual guidance with low-level proprioceptive control, is novel in its simplicity and efficacy, especially in achieving natural motion without predefined rewards or skill primitives.

2. Puppeteer advances visual whole-body humanoid control by setting new standards for naturalness and efficiency in motion synthesis. The zero-shot generalization to unseen tasks demonstrates the model’s potential for practical application.
summary: The paper proposes a hierarchical world model for whole-body humanoid control based on RL. The framework separates high-level and low-level control, with a high-level puppeteering agent providing commands for a pre-trained low-level tracking agent, which executes detailed joint movements. Key contributions include a task suite for visual humanoid control, a hierarchical control model using RL without pre-defined reward designs, metrics for "naturalness" in motion, and thorough analysis through ablation studies and user preference tests.
weaknesses: 1. Lack of low-level tracking performance evaluation. There is no evaluation or metrics for the tracking accuracy of success rate. There are several works both from simulated avatars community [1,2] and real-world humanoids [3,4] that evaluate the tracking performance. I am supurised that these works are not mentioned and their metircs are not used for evaluation in this work.


[1] Luo, Z., Cao, J., Kitani, K., & Xu, W. (2023). Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10895-10904).

[2] Won, J., Gopinath, D., & Hodgins, J. (2022). Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 41(4), 1-12.

[3] Cheng, X., Ji, Y., Chen, J., Yang, R., Yang, G., & Wang, X. (2024). Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796.

[4] He, T., Luo, Z., Xiao, W., Zhang, C., Kitani, K., Liu, C., & Shi, G. (2024). Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436.


2. The lack of interface design discuss. The paper proposes to use high-level controller to generate positions of tracking keypoints. This might be one way for reuse the low-level skills for downstream tasks, but there are many existing designs in prior works [5] [6] that not are compared in this work. To me, the idea of training low-level tracking policy for skills reuse is a long-standing idea, but the interface of this hierarchy matters a lot. I'd love to see more comparison on this.

[5] Tessler, C., Kasten, Y., Guo, Y., Mannor, S., Chechik, G., & Peng, X. B. (2023, July). Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings (pp. 1-9).

[6] Luo, Z., Cao, J., Merel, J., Winkler, A., Huang, J., Kitani, K., & Xu, W. (2023). Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582.

3. The source of naturalness is unclear. The low-level tracking policy might be conditioned on human motion pirors, but if the tracking policy is good enough, it should be able to produce what TD-MPC2 achieves. Also, if the advantage of this paper is sample-efficiency and naturalness, a key baseline here would TD-MPC2+ AMP, which is missing. That being said, all the experimental results make sense to me, but the key comparison experiments are missing somehow.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How relevant is the metric of "naturalness" in real-world humanoid control, and is it sufficient to evaluate humanoid trajectory tracking effectively?
2. It would be beneficial to include a comparative study on the survival rate or survival time when using the final-trained model
3. It would be helpful to include baseline experiments focused on "zero-shot generalization."
4. Please provide details on memory usage, training time, and control (or inference) time across different methods.
5. Is the low-level tracking effectively transferred to control different humanoid models with varying degrees of freedom?
soundness: 3
strengths: 1. The research addresses a significant and practical challenge in generalist agents: controlling a humanoid agent from visual observations using generalizable world models.
2. The methodology involves training a low-level agent on trajectory tracking that is adaptable across a range of control tasks, showing promising generalizability.
3. A high-level agent controls the humanoid from visual observations, a task-specific but broadly applicable approach in real-world scenarios.
4. A user study validates that the proposed method enables more natural humanoid control, which is preferred by participants.
summary: This paper explores the high-dimensional humanoid control from visual observations. Specifically, the proposed approach is based on two RL-trained agent models.  High-level agent generates reference trajectories from visual observations. Low-level agent focuses on tracking these trajectories using current low-dimensional state information. The proposed method demonstrated enhanced natural motion control of a 56-DoF simulated humanoid, outperforming baseline models according to experimental results and a user study.
weaknesses: 1. The evaluation heavily relies on the "naturalness" of movements, which depends on subjective human judgments of what is considered "human-like." This criterion, while important, may not fully evaluate the feasibility of such motions in actual humanoid robots, which face different kinematic and dynamic constraints than humans.
2. Based on Figure 5, the episodic return of the baseline TD-MPC2 is comparable or superior to the proposed method across most tasks. It would be beneficial to evaluate other performance metrics such as survival rates or survival times on the final-trained model to provide a more comprehensive evaluation.
3. The paper claims "Zero-shot generalization to larger gap lengths," yet does not compare these results with baseline methods. Including comparative generalization data for the baseline TD-MPC method would strengthen claims of superior generalization.
4. Minor issue:
a) Resource Efficiency: The two level agents training approach might require significantly more time and resources than single-agent baselines. Comparing memory usage, training duration, and inference times across methods would provide critical insights into the practicality of the proposed method.
b) Model Reusability: The low-level tracking agent is described as reusable across tasks but it is unclear if this model is applicable only to 56-DoF humanoids or if it can be adapted to different control dimensions.
c) There is a typo in the Problem Formulation section (page 2). The environment transition function should be denoted as L, not S.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - the main advantage of this method over TD-MPC2 is the resulting naturalness of the motions. Can the authors elaborate on why they think the proposed method improves this aspect? (here I mean further explaining the reward hacking argument made in the paper and perhaps including other arguments that could make sense)
- can the authors elaborate on how the low-level policies exactly track the high-level commands? Since the low-level receives a sequence of commands does it keep using $c_t$ until the tracking error is below a threshold, or is it only used for a single step independent of the outcome of applying the one-step action?
- on the methods side of things, the paper extends td-mpc2 to a hierarchical architecture. Can the authors compare the method to the hierarchical version of Dreamer [1]?

[1] Hafner, Danijar, et al. "Deep hierarchical planning from pixels." Advances in Neural Information Processing Systems 35 (2022)
soundness: 3
strengths: - the paper is generally well-written and an enjoyable read, I also liked the figures and plots.
- the proposed approach is very interesting and promising and is a natural next step to extend the TD-MPC2 framework.
- the method is evaluated on multiple humanoid tasks including environments with only proprioception as well as others with additional visual observations.
- the ablations nicely evaluate the role of the different design choices of the method. I especially appreciate the study of the role of planning in the architecture.
- the baselines include model-free and model-based approaches.
summary: Post rebuttal | the authors did a good job in the rebuttal phase to include some necessary baselines.

The paper improved a lot during the rebuttal phase and now includes some of the necessary baselines to validate the method. When I asked the authors about measures to ensure fairness in the new baseline experimentation, they argued that the considered MBRL algorithms are known to be robust to hyperparameter changes, which is true but not a valid argument for a simple reason: if an algo is robust to hyperparameter choice, doesn't mean that it cannot benefit from tuning. As I believe the proposed method was subject to some tuning during development time (like any ML method is), measures to ensure fairness of comparison would be necessary. Additionally, the baselines added during rebuttal just scratch the surface of hierarchical RL approaches and are themselves made up baselines combining known high-level algorithms with low-level TD-MPC. While these baselines are very important and necessary to understand the method, including standard baselines from the literature would strongly improve the paper and increase its impact. I highly encourage the authors to do that. I will be raising my score but only to a 5, as I still think the paper is not ready for publication at ICLR, despite the good improvements.

---

The paper proposes "Puppeteer" a hierarchical decision-making approach tailored for visual whole-body humanoid control. The proposed method trains two separate world models for high-level and low-level control purposes. The low-level world model is concerned with tracking reference joint-level trajectories produced by the high-level controller. The high-level controller can additionally be conditioned on visual data. Both world models are based on TD-MPC2 which is a sampling-based MPC approach with learned decoder-free world models (with deterministic-only components) and a learned value function for long-horizon value assignment. TD-MPC2 is further extended to include a termination encoder head as is common in other model-based RL methods such as dreamer. The paper claims that the proposed method achieves results that are mostly comparable to TD-MPC2's results, while the plots show significantly worse results than TD-MPC2 in terms of asymptotic performance. The main advantage of the method is that it produces more natural and human-like motion, which was quite well shown in the experiments. The paper also ablates multiple design choices.
weaknesses: - the method section misses a detailed motivation for why hierarchy improves the naturalness of the motions.
- the method section misses a detailed explanation concerning the exact usage of the high-level commands (see question 2).
- the paper introduces a hierarchical version of td-mpc2, the baselines however do not include a single hierarchical RL approach, I would at least consider including a hierarchical implementation of dreamer [1].
- [main weakness] The results of the paper are weak, at least in the current way in which they are presented. While the method is interesting and makes sense, the results show that it significantly underperforms TD-MPC2 but improves the naturalness of the produced motions. That would have been an acceptable tradeoff if the paper could justify why the proposed method improves the naturalness of the motions with intuitions and ideally, some experiments that validate them.

** Minor issues:**

- line 079 end-effector joints --> end-effector links.
- punctuation is missing in the equations (but I understand that this is a matter of style, so no pressure).

Overall the paper proposes an interesting approach, but it currently fails to showcase the benefits of this approach. I am willing to raise my score if this aspect is properly addressed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Although the motion produced by Puppeteer is more natural, why does the humanoid robot always lean forward while moving?
* Can the proposed method be validated in real-world experiments?
* Can this framework be generalized to manipulation tasks in HumanoidOlympics?
soundness: 3
strengths: * This paper is well-organized, well-written, and includes clear figures.
* A well-designed hierarchical control framework (although the idea is not novel) is implemented to control humanoid motion in a more natural way: the high-level agent generates commands given visual observations, and the low-level agent is responsible for executing them.
* The proposed visual whole-body high-dimensional humanoid control benchmark enrich the evaluation platforms in the area.
summary: The paper presents a novel hierarchical world model Puppeteer designed for visual whole-body humanoid control, which operates without relying on predefined skill primitives, reward designs, or temporal abstractions. This paper also introduces a new task suite consisting of eight challenging tasks for evaluating humanoid control, demonstrating that Puppeteer produces more natural and human-like motions preferred by human evaluators in comparison with model-free and model-based RL baselines.
weaknesses: * The paper primarily evaluates the visio-locomotive capabilities of the humanoid model. It could be better to expand the range of tasks to include more diverse scenarios that test different aspects of humanoid capabilities.
* More baseline method should be compared, like HumanoidOlympics (https://arxiv.org/pdf/2407.00187). This paper also uses human motion data and reinforcement learning to train natural humanoid motions in various tasks.

Meta-review: The paper proposes Puppeteer, a hierarchical reinforcement learning (RL)-based framework for whole-body control of humanoid robots based upon visual observations. The high-level model generates reference commands for a pre-trained low-level policy responsible for trajectory tracking. The experimental analysis emphasizes the "naturalness" of the resulting motion, with performance gains over contemporary baselines.

The paper was reviewed by four referees, who despite attempts by the AC, were unable to come to a consensus on the merits of the paper. Two reviewers (HShf and C5cF) appreciated the broader advancements that the paper provides to humanoid control. Reviewers 2bf6 and EisB found that it is well written and a pleasure to read. Reviewers HShf and EisB appreciated Puppeteer's hierarchical approach, but disagreed on its novelty. Meanwhile, at least two reviewers (C5cf and 2bf6) noted that the results show that the TD-MPC2 baseline is comparable to if not better than the proposed method. Related, several reviewers emphasized the fact that the evaluation focuses on the "naturalness" of the resulting motion, however "naturalness" is subjective and does not afford a standard metric. In their response, the authors note the importance of realizing motions that are natural, which the AC agrees with, but they did not address its subjectivity. Related, Reviewer 2bf6 questions the role of hierarchy in ensuring the naturalness of the resulting behavior, which the authors attribute to pretraining the motions on a large-scale collection of human motion capture data. Additionally, some reviewers identified the need for comparisons to other baselines, while others requested and evaluation of low-level tracking performance, which the authors made an effort to address during the rebuttal period by including additional results. 

The AC recognizes the importance of realizing humanoid motions that are natural and human-like, particularly for settings where robots are expected to interact with people. That said, the paper would benefit from a more concrete discussion of what novel aspects of the method are integral to realizing natural behaviors. THe significance of Puppeteer's contributions in this regard is not clear, particularly in light of the authors' claim that it is due to having pretrained on a large amount of human motion capture data, which one can argue constitutes an incremental contribution.

Decision: Accept (Poster)