H-GAP: Humanoid Control with a Generalist Planner
Authors: zhengyao jiang
Yingchen Xu
Nolan Wagener
Yicheng Luo
Michael Janner
Edward Grefenstette
Tim Rocktäschel
Yuandong Tian
Venue: ICLR 2024 spotlight (2024)
Keywords: Generative Modelling
Humanoid Control
Model Predictive Control
Model-based Reinforcement Learning
Offline Reinforcement Learning
Primary area: generative models
TL;DR: We present Humanoid Generalist Autoencoding Planner (H-GAP), a state-action trajectory generative model capable of adeptly handling downstream control tasks with Model Predictive Control.

Abstract: Humanoid control is an important research challenge offering avenues for integration into human-centric infrastructures and enabling physics-driven humanoid animations.
The daunting challenges in this field stem from the difficulty of optimizing in high-dimensional action spaces and the instability introduced by the bipedal morphology of humanoids. 
However, the extensive collection of human motion-captured data and the derived datasets of humanoid trajectories, such as MoCapAct, paves the way to tackle these challenges. In this context, we present Humanoid Generalist Autoencoding Planner (H-GAP), a state-action trajectory generative model trained on humanoid trajectories derived from human motion-captured data, capable of adeptly handling downstream control tasks with Model Predictive Control (MPC).
For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of motor behaviors. Further, without any learning from online interactions, it can also flexibly transfer these behaviours to solve novel downstream control tasks via planning. Notably, H-GAP excels established MPC baselines with access to the ground truth model, and is superior or comparable to offline RL methods trained for individual tasks.
Finally, we do a series of empirical studies on the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing.

Published as a conference paper at ICLR 2024
H-GAP: HUMANOID CONTROL WITH A GENERALIST PLANNER
Zhengyao Jiang 1,5 Yingchen Xu 1, Nolan Wagener 9, Yicheng Luo Michael Janner 4
Edward Grefenstette : Tim Rocktäschel' Yuandong Tian University College London, 2 Al at Meta, 3 Georgia Institute of Technology, University of California Berkeley," Weeo AI
ABSTRACT
Humanoid control is an important research challenge offering avenues for inte- gration into human-centric infrastructures and enabling physics-driven humanoid animations. The daunting challenges in this field stem from the difficulty of op- timizing in high-dimensional action spaces and the instability introduced by the bipedal morphology of humanoids. However, the extensive collection of human motion-captured data and the derived datasets of humanoid trajectories, such as MoCapAct, paves the way tackle these challenges. In this context, we present Humanoid Generalist Autoencoding Planner (H-GAP), state-action trajectory generative model trained on humanoid trajectories derived from human motion- captured data, capable of adeptly handling downstream control tasks with Model Predictive Control (MPC). For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of mo- tor behaviours. Further, without any learning from online interactions, it can also flexibly transfer these behaviors to solve novel downstream control tasks via planning. Notably, H-GAP excels established MPC baselines that have access to the ground truth dynamics model, and is superior or comparable to offline RL methods trained for individual tasks. Finally, we do a series of empirical stud- ies the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing. Code and videos are available at https: / htt////cxxyygnggcceec gi github. io/hgap/.
INTRODUCTION
Humanoid control stands pivotal realm in the integration into human-centric infrastructures and the generation of physics-driven animations (Peng et al., 2022). Yet, proficient humanoid control challenging due the inherent instability, discontinuity, and high dimensionality of the system. As a result, learning humanoid control policies from scratch not only incurs substantial computa- tional costs but also frequently results in unintended behaviours that do not resemble human-like actions (Heess et al., 2017; Song et al., 2019).
The abundance of motion capture (MoCap) data provides strong bedrock to address humanoid control. MoCap data demonstrates kinematic information about desirable sequences of poses as- sumed by the human body during various motions. By leveraging these demonstrations, becomes possible to learn natural motions that can be used for some task of interest (Merel et al., 2017; Peng et al., 2018; 2021). 1). However, existing methods leveraging the MoCap-derived data either need extra online interactions result in specialized models tailored to particular tasks and reward functions (Merel et al., 2017; 2019; Hasenclever et al., 2020; Peng et al., 2022). Despite generating natu- ral behaviours, such strategy demands extensive online interactions, limiting its applicability in real-world, physical robots. In contrast, offline RL promises policy training without further online interactions, albeit traditionally confined to small-scale datasets and less complex robot morpholo- gies, and offering limited transferability of learned knowledge and skills across tasks (Levine et al., 2020: Prudencio al., 2023; Lange al., 2012; Ernst et al., 2005).
Equal contribution. Correspondence yingchen.xu.. xu.21 @uccl.cc.kk

---

Published as a conference paper at ICLR 2024
codebook
odtaana
reconstructed trijectorioo
jump
Enc
transformer
cartwheel
Doc
imput trajectory
cocos
inttial
sampled lalent yajectorics
stato
intell
prior transformer
trajectory
state
VO-VAE
Prior Transformer
Zero-shot Adaptation (MPC Planning)
Figure 1: Overview of H-GAP. Left: A VQ-VAE that discretizes continuous state-action trajecto- ries. Middle: A Transformer that autoregressively models the prior distribution over latent codes, conditioned on the initial state. Right: Zero-shot adapation to novel tasks via MPC planning with learned Prior Transformer, underscoring H-GAP's versatility generalist model.
This work aims to leverage the state-action trajectories derived from MoCap data for general- purpose humanoid control, without any online interactions. Motivated by the success of large gen- erative models in fields such natural language processing and computer vision, we introduce Humanoid Generalist Autoencoding Planner (H-GAP), a generalist model for humanoid control trained on the large-scale MoCapAct dataset (Wagener al., 2022). Similar Trajectory Autoen- coding Planner (TAP) (Jiang et al., 2023c), H-GAP adeptly models the distribution of state-action sequences, conditioned on the initial state, and can be employed in downstream control tasks using Model Predictive Control (MPC). However, H-GAP differs from TAP in its focus and utility. TAP offline RI. method specialized for specific tasks, while H-GAP is generalist model for hu- manoid control. In terms of modeling, TAP captures state, action, reward, and return while H-GAP only models state and action dynamics. As result, H-GAP places greater emphasis on accurate state prediction than TAP, which uses state prediction only as an auxiliary loss. H-GAP is designed handle vastly larger and more varied datasets, with scales over 1,000 times greater than the single- task data used for TAP. See Fig. for overview of our framework and Appendix A for more detailed comparison between H-GAP and TAP.
Our experiments unveil compelling insights into the performance and scalability of H-GAP We show that H-GAP, as a generalist model, can faithfully represent wide range of motor behaviours from the training data. When deployed for series of downstream control tasks, show that the same H-GAP model either outperforms or is comparable to existing offline RL methods (Wang et al., 2020; Kostrikov al., 2022; Fujimoto & Gu, 2021; Jiang al., 2023c) that train specialist policy for each task, offering more flexible and adaptable solution. Furthermore, H-GAP outperforms conventional MPC strategies, such MPPI (Williams al., 2017), which have access the ground truth model (simulator). This distinction is potentially attributed to H-GAP's incorporation a learned latent action space and robust action prior, which are absent in traditional MPC approaches. In terms of scaling, H-GAP follows the trajectory of most generative models, where prediction accuracy and imitation performance improve proportionately with model size. Nevertheless, our analysis reveals mixed outcomes regarding the model's downstream performance the model's size increases. This discrepancy stems from the larger model's reduced sampling diversity, thereby adversely impacting its steerability for downstream tasks. Furthermore, we discover that H-GAP benefits significantly from larger and more diverse training datasets. This observation provides an incentive for the expansion of human MoCap datasets, encompassing broader range of real-world scenarios, thereby driving further progress in the field.
METHODOLOGY
In this section, explain each component of H-GAP: the discretization of state-action sequences, modelling the prior distribution over latent codes, and the planning process with Model Predictive Control (MPC). We leave more low-level details and hyperparameters in Appendix B.
2

---

Published as a conference paper at ICLR 2024
In general, H-GAP operates state-conditional trajectory generative model. Given an initial state 81 and sequence length T, it allows for the sampling of complete state-action sequences, denoted as in accordance with the conditional probability p(T[s1):
($1, d1, $2, 02,..... ST, ar) p(((s)).
For humamoid control trained on MoCapAct data, the states s are egocentric proprioceptive informa- tion and actions desired joint angles (which are converted to joint torques via pre-defined PD controllers on the humanoid). Both states and actions are continuous and high dimensional, which makes precise generative modelling rather challenging.
2.1 DISCRETIZING STATE-ACTION SEQUENCES
In order to model a highly diverse multimodal distribution of trajectories, discretization is a widely adopted solution (Janner et al., 2021; Reed et al., 2022; Jiang al., 2023c; Schubert et al., 2023; Brohan et al., 2023; Jiang et al., 2023b). In particular, Jiang et al. (2023c) propose an efficient discretization based on Vector- curtti--uad Variational Autoencoders (VQ-VAE) which is particularly powerful for high-dimennional state-action space, We follow a similar approach and learn an encoder and decoder, illustrated on the left component in Fig. 1. Unlike TAP which models state, action, reward and return, our encoder only takes state-action sequences as input:
(sne a1, $2, 02, ................................ ST, ar) (21, 22, (MT/L).
(2)
where L is the transition chunk size and M is the number of latent codes assigned per chunk. Both of them will be described in detail later.
To break it down: the encoder first concatenates the input states and actions into T transition vectors. It then applies sequence model, in this casc, several ID causal convolutional layers followed by causal Transformer decoder block, to obtain 1' transition feature vectors. After that, it divides these transition feature vectors into T/ chunks, each with L transitions, introducing temporal abstraction over the original sequence. A max pooling is then applied to the vectors in the same chunk.
Applying vector quantization directly on these chunk feature vectors can lead to low reconstruction accuracy due their high dimensionality. To remedy this, we use M latent codes to describe each chunk feature vector. We split each chunck feature vector into M code embeddings. resulting MT /T embeddings (I1..... MT/LL) ) in total. These vectors are then mapped into MT/L L latent codes (Z1..... ZMrT/r.) by finding their nearest neighbors in learned codebook e E R KK according to l2 similarity, where K is the size of the discrete latent space and D is the dimensionality of each latent code E
2 Ek: where k arg min ||ا, e,ll-
The decoder predicts the reconstructed trajectories based on the initial state and latent codes: faco($1, 21: 22, 2 MM/LL)
(s1 &2, ....... ST, êT).
(4)
The decoding process can be seen the inverse of the encoding process, except that the initial state is merged into the embeddings of the codes with a linear projection before decoding.
The autoencoder is trained by jointly minimizing the reconstruction loss, the Vector Quantisation (VQ) loss that brings the latent codes closer to the encoder outputs ||sg[z] ell?. as well as the commitment loss that ensures the encoder commits latent code | - sg[e]]]?, where sg stands for the "stop gradient" operator.
2.2 PRIOR OVER LATENT CODES
Following the discretization process, the subsequent step involves modelling the sequences of latent codes in an autoregressive manner using decoder-only causal transformer, frequently referred to as the prior over latent codes within the context of VQ-VAE. The Prior Transformer also conditioned on the initial state 81, which is achieved by adding the state feature to all token embeddings. More specifically, the Prior Transformer models p(z]zze 51), where žςi = (21 52, ......, z5-1) are prefix latent codes correspond to trajectory segment starting from 81. It should be noted that the sequence length MT/L L for modelling can be longer than the original trajectory length due to the mapping of single transition onto multiple latent codes, thereby facilitating more granular representation of the transitions. This critical for H-GAP's effective planning, which will be discussed in Appendix A.
3

---

Published as a conference paper at ICLR 2024
2.3 PLANNING WITH MODEL PREDICTIVE CONTROL
Combining the Prior Transformer and the VQ-VAE, we introduce H-GAP as a state-conditional tra- jectory generative model. This formation allows for the selection of optimal trajectories from those sampled from H-GAP, thereby facilitating a structured approach to planning in complex control tasks, as outlined Algorithm 1.
The strategy for implementing H-GAP within an MPC framework is quite straightforward. Initially, engage top-p sampling-a technique where only the top p% of the most probable samples from given distribution are considered for selection. This is coupled with adopting temperature parameter greater than The elevated temperature ensures that the sampled trajectories exhibit high level of diversity, facilitating more robust planning process with reduced sample set. However, an increased temperature can potentially generate samples with extremely low probability, affecting the modelling accuracy of H-GAP. To circumvent this issue, top-p sampling is employed to filter out these low- www-rbbbbbility, out-of-distribution samples, thereby preserving the reliability of the planning process.
In the context of MPC, an optimal trajectory defined as the one possessing the highest objective score, calculated using simple objective function R(T) = Σ r(s:), where r represents the reward function and t ($1, 01. $2, .2. ... 5T, Š5,ūā) åT) denotes the sampled trajectory.
Algorithm H-GAA Model Predictive Control
Require:
Current state 81, H-GAP prior po(zilzes,8) 81) and decoder Sace. model parameters: T (steps), M (codes per step), L (latent steps), N (number of samples), (temperature), (top p threshold). R(r) (objective function).
1: Initialize trajectory set To = 0
2: for sample number l ..... N do
3:
for iteration = 1,..., MT do -
4: Get probability logits o of prior (2
5:
Adjust the probability with temperature Re
Enja e'SAiT
6:
Adjust the probability with p get (zi, 81),,,
7:
Sample new latent codes ži, according to to"" end for
8:
9:
Decode latent codes т = fdce (21,k, 22, 2M1/LLKs
10: Append the trajectory to the set To = {Ta} UT-1
11: end for
12: Return the optimal trajectory arg maxteT R(TT)
3 EXPERIMENTS
Our experiments aim to answer the following questions: (Q1) How well does H-GAP represent the motor behaviours in the training data? (Q2) How well can H-GAP compose and sequence the learned motion priors to solve novel downstream control tasks? (Q3) How does H-GAP benefit from model and dataset scaling?
Data. We use the MoCapAct dataset (Wagener et al., 2022), which contains over 500k rollouts with a total of 67M environment transitions (corresponding 620 hours in the simulator) from collection of expert MoCap tracking policies for MuJoCo- based simulated humanoid, that can faithfully track 3.5 hours of various recorded motion from CMU MoCap dataset (CMU, 2003). The dataset covers a wide range of human motor behaviours including walking, running, and more complex movements like cart-wheeling.
3.1 IMITATION LEARNING EVALUATION
To evaluate how well H-GAP represents the various motor priors in the training data, we design an imitation learning task where the objective is to match the humanoid poses in some reference trajectories. Similar to the multi-clip tracking task (Hasenclever et al., 2020), the beginning of
4

---

Published as a conference paper at ICLR 2024
Figure 2: MoCap imitation task with simulated humanoid controlled by H-GAP (bronze) and offset reference pose (grey). Conditioned solely on an initial state, the H-GAA agent can faithfully follow the reference trajectories in a rather long horizon.
each episode, the humanoid is initialized pose randomly sampled from reference trajectory. An episode is terminated when the divergence from the reference poses exceeds some predetermined threshold, or when the reference trajectory terminates. Unlike the multi-clip tracking task, we do not provide the future target reference poses to the agent since our focus is on evaluating the learned prior motor distributions solely conditioned on the initial state. To induce imitation behaviour for H-GAP, we simply greedily choose the most likely code given by the prior autoregressively and decode the action. In order to prevent the agent from succeeding just by memorising the dataset, we add Gaussian action noise with a standard deviation of 0.01. As shown in Figure 2, H-GAP faithfully generates a wide range motor behaviours across multiple clips when prompted with the corresponding initial states.
In order to quantitatively evaluate performance, we train behaviour cloming (BC) (Hussein et al., 2017) agent using the entire MoCapAct dataset, which uses 5M parameter MLP its architecture. We compare average episodic clip tracking returns and lengths between H-GAP and BC, utilizing 12 sam- pled clips that showcase diverse motions from the training set. The results, as illustrated in Table 1, unequivocally demonstrate H-GAP's substantial performance advantage over the BC baseline. Higher episodic returns indicate that the generated trajectories closely resemble the reference trajectories, whose initial states are used as prompts. In addition to tracking rewards, we also consider the episode length as an additional surrogate metric for imitation. This is because an episode concludes when the agent's deviation from the reference trajectory surpasses a predefined threshold. The results are not unexpected, MLP BC agent are determined solely by the current states, while the transformer based H-GAP in- corporates elements of past trajectories into its decision-making process. This finding underscores H-GAP's superior ability to model the motor priors present in the datasct, and is in line with obser- vations highlighted in prior research (Janner al., 2021).
Table 1: Imitation Performance
MLP BC
H-GAP
Returns Length
29.38 + 2.43
46.02 # 1.57
49.34
69.11 + 2.62
the outputs of the
3.2 DOWNSTREAM CONTROL EXPERIMENTS
Having verified H-GAP's capability of representing motor priors from the training data, we next in- vestigate how well H-GAP can re-purpose the learned motor priors to solve novel downstream con- trol tasks via planning. First, we elucidate the process of crafting downstream tasks and establishing the benchmarks for comparison, including offline RL and conventional MPC baselines. We com- pare the conceptual and empirical advantages of H-GAP over conventional approachss-hiihlighhee in Table 2 as demonstrates both high modeling accuracy and control performance with the capa- bility for zero-shot adaptation without access ground truth models.
5

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of different methods for humanoid control.
Method
Offliee Actor Critic
TAP
MPPI Bascline
H-GAP
Zero-shot Adaptation
No
No
Yes
Yes
Access to Ground Truth Model
No
No
Yes
No
Madeling Accuracy
Medium
High
Contol Performance
Low
High
Low
High
3.2.1 EXPERIMENT SETUP
Construction of Downstream Tasks: We design a suite of six control tasks: speed, forward, back- ward, shift left, rotate and jump. The reward functions meticulously mapped from the agent's observational parameters, including features like egocentric velocities, egocentric angular velocities, body height, and the global z-axis vector from an egocentric perspective. For the speed task, the re- ward is determined as the norm of the velocities, which does not limit the direction of movement. The forward task rewards are based on the agent's forward velocity, while the backward task focuses on the backward velocity. In the shift left task, rewards are given for achieving velocity towards the left-hand side, and the rotate task allocates rewards based on y-axis angular velocity. The jump task rewards increases in body height, without imposing penalties for reductions.
The initial states for these tasks are sampled from a selection of MoCapAct trajectories, with Gaus- sian noise introduced to the velocity and angular velocity parameters to simulate realistic scenarios. Notably, the state dynamics mirror the conditions encountered MoCapAct environments.
Offine RL Baselines: The offline RL strategies are characterized by their reliance on reward sig- nals. Therefore, for each individual task, we apply the respective reward function to all MoCapAct trajectories, generating reward-labelled dataset specific to that task. This is fair set-up because the offline RL in this case can get access to all the data seen by H-GAP. On the other hand, compared with conventional offline RL settings with rich near-optimal data, this setting is challenging for out- of-distribution tasks like shift left or backward, where the majority of trajectories in the dataset are suboptimal.
Conventional MPC Baseline: To comprehensively evaluate the accuracy and action prior of the H-GAP model, we MPPI (Williams et al., 2017), state-of-the-art MPC method, baseline. This baseline has access to the ground truth dynamics model, thereby establishing an upper bound on modeling accuracy and offering a critical benchmark for evaluating the efficacy of H-GAP's MPC style control. In order to see how helpful the action prior in the MoCapAct dataset will be for the planning, we also tested as variation of MPPI that we denote MPPI In this variation, action noise during planning is parameterised according to the action mean and variance of the actions in the MoCapAct dataset.
3.2.2 EXPERIMENT RESULTS
Here delve deep into the downstream performance summarized in Table 3. We will organize the discussion around H-GAP in this subsection but the new setting and empirical results are also insightful for existing offline RL methods, which we will discuss in Appendix C.
H-GAP versus Offline RL Methods: H-GAP outperforms model-free offline RL methods like CRR (Wang al., 2020), IQL (Kostrikov al., 2022), and TD3-BC (Fujimoto & Gu, 2021) while matching the performance of the model-based offline RL algorithm, TAP. Unlike these offline RL methods, H-GAP is a generalist model. It does not specialize in particular tasks nor does it learn a critic for any long-term value estimation. Despite these apparent restrictions, H-GAP manages to outperform its model-free counterparts, which underscores the effectiveness of planning- algorithms with learned discrete action spaces for humanoid locomotion tasks. This finding aligns with the previous and concurrent works on lower-dimensional continuous control, that 1) planning- based methods are highly effective Hansen et al. (2022a); Authors (2023) and 2) learning a discrete action space improves the control performance for model-free methods Tang & Agrawal (2020); Luo et al. (2023). In addition, as shown in Fig. 3d. H-GAP makes much more accurate state predictions compared to TAP. Its strong sequence modelling abilities make up for its lack of long-term credit assignments. In practical terms, the model's ability handle multiple downstream tasks without

---

Published as a conference paper at ICLR 2024
Table 3: H-GAP and baseline algorithms' performance on downstream tasks. Offline RL methods are trained with seeds. The values after + are standard error of the mean. For H-GAP", discount on rewards are applied.
Offline RL Specialist
MPC Generalist
Task
CRR
IQL
TD3-BC
TAP
MPPI
MPP|,...
H-GAP
H-GAP
speed
63.8
64.9 11.9
61.3 4.4
2.3
36.7
56.1
83.4
97.45
3.81
rotato
-7.5 11.0
9.6 10.8
60.2 12.5
44.0
23.9
14.6
89.5
103.33
1 5.61
jump
1.666
2.2
1.70
forward
27.6 13.6
39.2 18.0
3,15
121.2 10.9
0.9
18.1
51.1
70.06
4.83
-.0
10.7
25.7
27.1
3.1
13.24
backward
34.6 12.2
211.2 16,9
1,9
1.7
10.8
2.46
3.4
2.81 4.14
Mean
7.82
15.72
32,70
43.8
12.50
11.43
40.39
46.18
needing retraining is an advantageous feature. It not only saves computational resources but also provides a viable solution when it's impractical to train separate models for large number of tasks.
H-GAP versus Conventional MPC: When pitted against the MPPI, a method under the MPC cate- gory, H-GAP exhibits superior performance despite MPPI having access to the ground truth model. The underwhelming performance conventional MPC methods can be attributed to their lack of learned action space and strong action prior. The latent action prior learned from mocap data aids in constraining the action space, which historically has been challenging to optimize solely through sampling. Furthermore, H-GAP's constrained action space forces the plan to exhibit natural be- haviours, potentially hinting at encapsulating long-term values. This approach prevents the model from adopting short-sighted reward maximization strategies, such as compromising balance for im- mediate high rewards, thus ensuring more balanced and foresighted optimization strategy. With naive Gaussian action prior. MPPI managed to perform slightly better on tasks that well- represented in the dataset, for example: speed, forward and jump, while still inferior to H-GAP. Further, such a prior is harmful for other tasks which leads to mixed results compared with vanilla MPPI. This shows the complex action prior to humanoid control is not easy to capture with a single mod Gaussian.
3.3
SCALING
Scaling is key to the success of existing large generative models (Kaplan et al., 2020; Henighan et al., 2020; Hoffmann al., 2022: Rombach al., 2022; Podell al., 2023) but has been little studied in humanoid control. This is likely due to the absence of comprehensive datasets like MoCapAct and general-
general-purpose purpose models. H-GAP fills this void, starting an early attempt build foundation model for humanoid control. We therefore investigate H-GAP's scaling properties to provide insights on further research in this direction.
22.00
4.9
(a) Training Loss
(b) Validation Loss
(c) Latent Codes Entropy
(d) State Prediction Error
Figure 3: The graphs show training and validation losses, entropy of learned latent codes, and model prediction accuracy for different model sizes ranging from 6M to 300M parameters. Scaling up model sizes improves model validation set accuracy, which indicates that the larger models are better modelling motor behaviours. However, the prediction accuracy improvement is marginal. The lower entropy of latent codes indicates that larger models generally produce less diverse trajectories.
Model Scaling: We train five different sizes of the H-GAP model, ranging from 6M to 300M parameters, utilizing 8 Nvidia V100 GPUs for each model. This scaling study draws inspiration from the work of Henighan al. (2020), who report consistent test performance improvement

---

Published as a conference paper at ICLR 2024
in autoregressive transformers the number of parameters increases across different modalities. Consistent with these findings, we observe similar trends in our validation loss, particularly noting substantial improvement when increase the model size from 100M to 300M parameters.
The benefits scaling also extend to the imitation performance of H-GAP. Larger models generate more accurate control signals that successfully recover human-like motor skills. However, when apply these scaled models to downstream control tasks, the results are mixed. Contrary to expecta- tions, larger models do not consistently outperform their smaller counterparts. We hypothesize that this inconsistency arises from the larger models' tendency to closely replicate the trajectories present in the training dataset, which adversely affects their steerability in downstream tasks. To investigate this further, we examine the average entropy of latent codes generated by models of different sizes in shown
a case where the initial state distribution consistent with that of the downstream tasks. in Fig. 3c, the results indicate that larger models generally produce latent codes with lower entropy, suggesting decreased diversity when generating control signals. addition, in Fig. 3d, we compared the ability of TAP and H-GAP variations for modelling the state transitions. For an initial state from the downstream task, we sample a trajectory from the model and compare it with simulator roll- outs with the same actions in the trajectory. One the prediction aceuracy. Moreover, we observe degrading performance on downstream control tasks model size scales. The intriguing results suggest that merely scaling up the model size, given the current dataset, does not offer an automatic improvement in downstream task performance even though the large model is better in (self-) sspeevieuee learning metrics.
larger models only marginally improve
1330
1930
Mean Returee
(a) Model scaling: Imitation Tasks
(b) Model scaling: Downstream Tasks
Vee nourr
(c) Data scaling: Imitation Tasks
(d) Data scaling: Downstream Tasks
Figure 4: Scaling properties of H-GAP in terms of imitation and downstream task performances. The first row show results for different model sizes ranging from 6M to 300M parameters. Scaling up model sizes significantly improves imitation performance but surprisingly results in degrading performance in downstream control tasks. The second row show ablations for different datasets ranging from 10% to 100% of the original MoCapAct dataset. Scaling data sizes and diversity improves model's imitation and downstream control performance.
Data Scaling: Given that larger models may suffer from reduced generation diversity, it stands to reason that the size and diversity of the training data could likewise influence downstream task performance. To explore this, we construct three smaller training datasets containing 10%, 30%, and 50% of the original MoCapAct dataset. These subsets range from 7M to 67M transitions. We medium-size (40M) models for all the data scaling experiments.
As depicted in Fig. 4c and Fig. 4d, H-GAP benefits from larger and more diverse training datasets, both in terms of imitation performance and in accomplishing downstream tasks. This finding is especially compelling as it underscores the need for more comprehensive and varied human MoCap datasets improve performance across broader range of real-world scenarios.
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Humanoid Control Simulated humanoids have long been a focal point in robotics research due to their remarkable physical capabilities. However, controlling these humanoids presents substan- tial challenges, primarily stemming from their high-dimensional action spaces and the difficulty of recovering from failure states, While attempts to learn humanoid control from scratch have histor- ically yielded inconsistent, non-human-lin like motions (Heess et al., 2017; Song et al., 2019), recent research efforts have concentrated on acquiring motor primitives or reusable low-level skills derived from motion capture data (Merel et al., 2017; 2019; Hasenclever et al., 2020; Peng et al., 2022; Dou et al., 2023). Despite significant progress, the majority of these methods still necessitate online policy learning for each specific downstream task. In contrast, our proposed method showcases the capability to perform in a variety of novel downstream tasks zero-shot.
Human Motion Generative Models The emergence of large-scale generative models in the fields of natural language processing (Brown et al., 2020), coupled with the abundance of human mo- tion capture data (CMU, 2003; Ionescu et al., 2014), has ignited interest in the development of human motion generative models. Many of these models require paired labeled data comprising both motion and high-level annotations such linguistic descriptions (Jiang al., 2023a; Ahuja & Morency, 2019; Guo et al., 2022; Tevet et al., 2023; Petrovich et al., 2021). While most prior research has centered on modeling state-only trajectories, our approach aims to learn controllable agents by also modeling action distributions.
Model-based Reinforcement Learning Our method closely related to model-based reinforcement learning (RL), a paradigm that involves learning a model of the environment that can be used for planning (Ebert et al., 2018: Zhang et al., 2019; Janner al., 2019; Hafner et al., 2019; Schrit- twieser et al., 2020; Hansen et al., 2022b; Chitnis et al., 2023) and/or policy learning through model- generative rollouts (Pong et al., 2018; Ha Schmidhuber, 2018; Hafner et al., 2020; 2021: 2023). Previous works on this directly usually small Multi-Layer Perceptrons (MIPs) trained exclusively on single-task data. In contrast, our method leverages a single, large-scale transformer- trained on diverse datasets, which can be applied zero-shot to various downstream tasks.
model
Offine Reinforcement Learning In offline reinforcement learning (Levine et al., 2020; Prudencio et al., 2023; Lange et al., 2012; Ernst et al., 2005), the agent learns from fixed offline dataset, aiming to achieve higher rewards than the policy that generates the data. A recent line of work treats offline RL trajectories generative models for other modalities (Janner et al., 2021; Chen et al., 2021; Jiang et al., 2023c; Janner et al., 2022; Reed et al., 2022; Jiang al., 2023b; Schubert et al., 2023), which is most relevant to this work. The major difference between offline RL methods and H-GAP is offline RL have to be trained for specific tasks but H-GAP trained task-unaware. H-GAP also an early attempt to solve challenging 56 DoF humanoid control with offline data, which, as far as we know, unexplored in offline RL and sequence modelling continuous control.
unstructured state-action sequences and models them with various
LIMITATIONS AND FUTURE WORKS
Our study identifies both the strengths and shortcomings of H-GAP, offering directions for future work. On the methodology side, while H-GAP excels in humanoid tasks, its performance may dip in sparse reward settings requiring long-term planning. Potential next steps include incorporating global information of the trajectory in the latent codes, for example, returns or goals, and investi- gating planning mechanisms beyond vanilla MPC, such as hierarchical planning or value learning integration. Also, H-GAP can be less effective in partially observable or stochastic environments. The model's 's optimistic plans, based on latent codes, may not always be realistic. Future work could explore mechanisms for more balanced planning.
With MoCapAct, we find model scaling saturation, where the validation accuracy and imitation per- formance still keep improving but not the downstream tasks. The experiments on different dataset sizes suggest further research could focus on data diversification and volume to forward the devel- opment of the foundation models for humanoid control.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank Meta and the UCL CDI AWS Doctoral Scholarship for supporting this project. We also received insightful input from Siqi Liu and members of the UCL DARK Lab, among others.
REFERENCES
CMU Graphics Lab Motion Capture Database, 2003. URL http::h/tccaapppp cmu edu/.
Chaitanya Ahuja and Louis-Phillppe Morency. Language2Pose: Natural language grounded pose forecasting. In 2019 International Conference on 3D Vision (3DV), pp. 719-728. IEEE, 2019.
Anonymous Authors. Td-mpe2: Scalable, robust world models for continuous control. 2023. URL https::/spenrewwe net/ 2 Forum?id==xhhcstDD.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics transformer for real-world control at scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu (cds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: https: : /ooii .aci.ozz www.oogg.o...ooo.ooooooooo..oo.. 156077.........0.1 /RSS. 2023 XIX. 025.
URL
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vose Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:s::t/.............tt /tt.httss:/ 447c0d66fcbbl 967416 8/ https/////oooeeeensssses. iewerrrps.... tttppsp..ccp...peeeeeeoooee ccc pccp/appr/202// h0sh//ash/hhh// html.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling-
Beygelzimer, Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:/ http://///perreeew net/ net/foouee htrr/forummiis -id-aaa 77APM4439..
Rohan Chitnis, Yingchen Xu, Bobak Hashemi, Lucas Lehnert, Urun Dogan, Zheqing Zhu, and Olivier Delalleau. IQL-TD-MPC: Implicit Q-learning for hierarchical model predictive control. arXiv preprint 2rXi::330.0006677 2023.
Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C-ASE: Learn- ing conditional adversarial skill embeddings for physics-based characters. arXiv preprint arXiv:2309. 1351, 2023.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 18/2.08/2.000 2018.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto and Shixiang Gu. A minimalist approach offline reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://apenrevieee wet.:wiwwwweeeeeeeeeeee......eee wttp://wwwweeeeomommi... Q32U7dzWXpc.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3D human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152-5161, June 2022.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems, 31, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on PMLR, 2019.
Machine Learning, pp.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. In International Conference on Learning Representations, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint A.Xiv:3301.04000 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predic- tive control. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Re- search, pp. 8387-8406. PMLR, 2022a. URL htttt::://// wtttps::/proceeedinggto mlr wttp:nnww.......r.ssse.mmm press/v162/ hansen22a. aa. html.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387-8406. PMLR, 2022b.
Leonard Hasenelever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. CoMic: Com- plementary task learning & mimicry for reusable skills. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4105-4115. PMIR, 13-18 Jul 2020. URL https: http://proeeeeee ngs ml pre.sppesss...... v1 https//v11///hh/eeeeee http/h/eeeeeeeeeeeeeeeeee html.
Nicolas Heess, Dhruva Tb, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours rich environ- ments. arXiv preprint
2017.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. CoRR, abs/2010.14700, 2020. URL hwtps:ttp//.o.....t. www..iiiorrooooo abs /2os//10/000 14701.
Jordan Hoffmann, Sebastian Borgcaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoe, Aurelia Guy. Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203.15555 2022. doi: 10.485550/rXiw.2203.. 15556.............. URL https: /oi. org/10. 0ttg//...8550000 43550 2203 15556.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods, ACM Computing Surveys (CSUR), 5032):1-35,2011 2017.
11

---

Published as a conference paper at ICLR 2024
Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human336M Large scale datasets and predictive methods for 3D human sensing in natural environments. Transactions Pattern Analysis and Machine Intelligence, 36(7):1325-1339) 2014. doi: 10.1 10.1109/TPMM.2211..
IEEE
Michael Janner. Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Michael Jamner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http::////ppeeeeeeee.ee net/forum? (ergge5563ggs0
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. MotionGPT: Human motion as a foreign language. arXiv preprint arXiv:2306. 14795, 2023a.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. In Fortieth International Conference Machine Learning. 2023b.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette. and Yuandong Tian. Efficient planning in compact latent action space. In International Conference on Learning Representations, 2023c.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 025/2001008366 2020. URL ht /ttppp:///axii...///eeee org/ abs/2001 08.008666 61.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q- learning. In International Conference on Learning Representations, 2022. URL https: // openreview. wewnnreviewnnet/fooeeeeee met/foounm 6aa.mmmddddd822nnnnmmm
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Rein- forcement Learning, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action- quantized offline reinforcement learning for robotic skill learning. In 7th Annual Conference on Robot Learning, 2023. URL https:/ wpw.reveeeeeeeeeeeeeeee/e..oe..e net/ forum?
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint 1707 2017.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con- trol. In 7th International Conference on Learning Representationss ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/cpenzeviiee net/forunt id=BJl 6Tjy7
Xue Bin Peng. Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. DeepMimic: Example- guided deep reinforeement learning of physics-bases character skills. ACM Transactions On Graphics (TOG), 1-14, 2018.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: Adversarial motion priors for stylized physics- physics-based character control. ACM Transactions on Graphics (ToG), -02 2021.
12

---

Published as a conference paper at ICLR 2024
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE: Large-scale reusable adversarial skill embeddings for physically simulated characters, ACM Trans. Graph. (4):94:1-94:17, 2022. doi: 10. 10.1145/352822333531100 /tt/o/aaas////iiooo/////// 3528223. 3530110.
Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10985-10995, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis, CoRR, 0ts/2307.01952 2023. doi: 10448550/arxx.22007.09222 URL https://
arwiv. 2307.00000 2197.01952.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep RI. for model-based control. In International Conference on Learning Representations, 2018.
Rafael Figuciredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVE Conference Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685 IEEE, 2022. doi: 10.1109/CVPR5263 201.PRS22682202.000... 2022.01042. URL https:/ /doi. org/ wtgg/1..109///VPP.
2022. 01042.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):04 609, dee 2020. doi: 10.
11500/444100000--305 020 03051-4,
URL https: /ttp///o.oooo// orrg///
Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin A. Riedmiller, Jost Tobias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control. CoRR, abs/2305. 2023. doi: 10.48550/arXiv.23055 10912. URL https:://// /oi./iiioo// orgiioorgg100 48550 /015500/rXivv. 2305 10912.
H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2019.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Phirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Sympo- sium Educational Advances in Artificial Intelligence, EAAI 2020. New York, NY, USA, Febru- ary 7-12, 2020, pp. 5981-5988, AAAI Press, 2020, doi: 10.1609/AAALLV34104.... https: /ttp///oii.o// .or.ii.org//oo 1609/11060 aaai w34104.44.... 6059.
URL
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: http:////peeeeeeeeeeeeeeeeeeeeee net/ forum? wow.:ommmmi..1ssoo2222
13

---

Published as a conference paper at ICLR 2024
Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, and Matthew Hausknecht. MoCapAct: A multi-task dataset for simulated humanoid control. In Advances in Neural Information Processing Systems, volume 35, pp. 14418-35331,22 2022.
Ziyu Wang, Alexander Novikov, Konrad Zołna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems,NuurIPS NeurlPS 2020, Red Hook, NY, USA, 2020. 9781713829546.
Curran Associates Inc. ISBN
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40 (2):344-357, 2017.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. SOLAR: Deep structured representations for model-based reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 7444-74477 -453. PMLR, 2019.
14

---

Published as a conference paper at ICLR 2024
A COMPARISON OF TAP AND H-GAP
Both TAP and H-GAP are trajectory generative models aimed at planning-basee control, but they differ focus and utility. This section highlights the key differences and innovations in H-GAP.
ТАР is an offline RL method, where models are trained for specific tasks with their own reward functions. H-GAP, on the other hand, is a generalist model suitable for a wide range of tasks in humanoid control, offering more versatility. Moreover, H-GAP is designed to handle vastly larger and more varied datasets, with scales over 1,000 times greater than the single-task data used for TAP.
In terms of modelling and planning objectives, TAP models state, action, reward, and returns, while H-GAP only models the state and action dynamies. Accordingly, TAP's planning aims to maximize predicted rewards along with terminal return-to-go. H-GAP focuses on maximizing an objective function based solely on rollouts. As a result, H-GAP places greater emphasis on the accuracy of state prediction than TAP. In TAP, state prediction serves mainly often maps multiple transitions to a single latent code for more efficient planning. In contrast, H- GAP assigns multiple codes individual steps ensure the accuracy of state reconstruction, which its objective function depends on.
auxiliary loss. Therefore, it
H-GAP is not simply extension of TAP within the offline RL context. It is streamlined, general-purpose model for generating humanoid state-action sequences. It avoids unnecessary com- plexities in its architecture, aiming for broad applicability across various tasks.
B FURTHER MODEL SPECIFICATIONS
Mismatched Termination Conditions The MoCapAct dataset and the downstream tasks we con- sider have different termination conditions. which could result in error-prone behaviour if we were to actively predict the end of an episode. Specifically, MoCapAct trajectories terminate either due to timeout when the agent deviates significantly from the reference trajectories. On the other hand, in our suite of downstream tasks, termination occurs when any non-foot body parts touch the opt
ground. To mitigate potential issues arising from these mismatched termination conditions, to only sample complete trajectories during the planning phase. Importantly, the H-GAP model does not actively predict the end of the trajectory. While in theory this could trigger issues for tasks where touching the ground would result in higher rewards, this has not been observed in the suite of tasks we have constructed.
Normalisation When training the VQ-VAE component with Mean Squared Error (MSE) loss, it is crucial to properly normalise the state and action dimensions to prevent large loss bumps, partic- ularly for those dimensions with higher range. In this work, we use the dataset mean and standard deviation for this normalisation.
Learning Rate Anncaling For the annealing of the learning rate, we adopt a similar pattern as proposed by (Janner al., 2021): a linear warm-up phase followed by cosine decay until the learning rate reaches 10% of its initial value. This scheme generally works well for smaller mod- els. However, for the largest models with 300M parameters, some unusual training curves were observed, as shown in Fig. 3a and Fig. 3b. This phenomenon might also be attributed to the learning rate, as a smaller learning rate has been found accelerate the learning of larger models.
Hyperparameter A comprehensive list of hyperparameter settings used in our experiments can be found in Table 5.
C DISCUSSIONS ON BASELINES
CRR and IQL Many offline RL methods have been developed in settings where the dataset con- tains nearly optimal trajectories, often generated by an online learning agent optimising for the same task. This makes simple behaviour cloning of high-quality trajectories in the dataset quite effective for a wide range of benchmarks (Chen et al., 2021). This has also motivated design choices that employ behaviour cloning weighted by the advantage, calculated by critic. However, find
15

---

Published as a conference paper at ICLR 2024
that methods relying such weighted BC designs, such as CRR and IQL, perform poorly the MoCapAct relabelled datasets and their corresponding downstream tasks. This underperformance is especially pronounced for dittrf--ittrrbuttoon tasks like "backward"; 'rotate y'. and 'shift left'. This may be because, after relabelling, the majority of actions in this diverse dataset are irrelevant to the task, making simple behaviour cloning insufficient for guiding the learned policy optimally.
TD3-BC Conversely, TD3-BC, which takes gradients from the critic to directly optimise the target, consistently outperforms CRR and IQL. Remarkably, even when compared with strong sequence modelling methods like TAP and H-GAP, TD3-BC holds its own, particularly on out-of-distribution tasks such as 'rotate_y', 'shift left', and 'backward'.
TAP Among all the offline RL specialist methods, TAP stands out significantly better than the others. In 'forward' tasks, TAP even manages to keep moving forward until timeout, resulting in extremely high returns. This aligns well with findings by Jiang al. (2023c), where TAP's advantages over baselines increase as the state-action dimensionality grows. In their work, the tasks had maximum of 24 degrees of freedom. In our study, we pushed this 56, confirming that the benefits of learned action space are particularly useful for high-dimensional control tasks.
Hyperparameters found in Table 4.
Detailed hyperparameters for the baselines used in our experiments can be
Table 4: Hyperparameters for Baseline Models
Model
Hyperparameter
Value
Discount
0.99
Number of layers
CRR
Number of units
512
Learning rate
1 x 10-4
Training steps
50k
Discount
0.99
Temperature
0.1
0.005
IQL
Expectile
0.9
Learning rate
1 x 10-4
Training steps
50k
Discount
0.99
0.005
TD3-BC
BC
2.5
Learning rate
1 x 10-4
Training steps
50k
Batch size
128
Training steps
380k
Modelling horizon
24
Chunk size L
3
TAP VQVAE
Code per chunk M
1
Number of codes K
512
Learning rate Discount
2 x 10
0.99
Reward weight
1.0
Value weight
1.0
Dimension
512
Number of heads
Number of layers
4
Training steps
380k
TAP Prior
Learning rate
x 10-4
Position embeddings
Absolute
16

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for H-GAP Model
Component and Size
Hyperparameter
Value
Batch size
128
Training steps
10
Modelling horizon T
16
Chunk size
VQ-VAE
Code per chunk M
16
Number of codes
512
Learning rate
3 10
Dimension
384
Number of heads
Number of layers
3
Training steps
5 x 107
6M Prior
Learning rate
10-4
Batch size
1024
Position embeddings
Absolute
Dimension
512
Number of heads
4
Number of layers
4
Training steps
5 x 107
10M Prior
Learning rate
3 10
Batch size
1024
Position embeddings
Absolute
Dimension
768
Number of heads
6
Number of layers
Training steps
5 107
40M Prior
Learning rate
3 10
Batch size
1024
Position embeddings
Absolute
Dimension
1024
Number of heads
8
Number of layers
8
Training steps
5 x 107
100M Prior
Learning rate
3 10 10-4 1024
Batch size
Position embeddings
Absolute
Dimension
1536
Number of heads
12
Number of layers
12
Training steps
10a
300M Prior
Learning rate
3 10-4
Batch size
1024
Position embeddings
Absolute
Table Hyperparameters for MPC Planning
Hyperparameter
Value
Sampling threshold p
0.99
Number of samples N
256
Temperature T
4
17

---

Published as a conference paper at ICLR 2024
D ABLATIONS
We use 100M parameter H-GAP models in all our ablation experiments. We report the mean and standard error over 300 random seeds for downstream control tasks and over 30 seeds for imitation lasks.
Table 7: Codes per Step.
Codes per Step
Imitation Returns
Downstream Returns
8
39.13 1.12
32.84 + 1.29
16
43.58 1.32
35.90 1.18
32
42.24 + 1.33
32.80 + 0.85
Table 8: Training Sequence Length and Planning Horizon
Sequence Length
Horizon
Imitation Returns
Downstream Returns
16
16
43.58 1.32
35.90 + 1.18
24
16
39.13 1.12
32.86 + 1.73
24
24
39.50 1.18
30.84 + 1.93
16
36.74 1.05
28.55 1.61
32
24
36.91 0.96
24.52 + 1.60
32
36.90 1.06
21.36 1.41
64
16
32.49 0.86
23.20 + 1.43
24
31.90 0.92
22.52 1.42
64
32
31.58 0.89
21.91 + 1.34
64
64
30.36 0.87
19.34 1.32
Table 9: Planning Hyperparameters
Top
Temperature
Num Samples
Returns
0.98
4
64
31.87
0.98
4
128
33.53
0.98
4
256
36.39
0.98
8
64
29.18
0.98
8
128
31.87
0.98
8
256
38.39
0.98
16
64
28.89
0.98
16
128
29.58
0.98
16
256
38.17
0.99
4
64
32.17
0.99
4
128
37.29
0.99
4
256
38.81
0.99
8
64
34.52
0.99
8
128
31.33
0.99
8
256
38,06
0.99
16
64
28.28
0.99
16
128
32.20
0.99
16
256
34.36
0.999
4
64
30.44
0.999
4
128
30.19
0,999
4
256
34.54
0.999
8
64
27.92
0,999
8
128
34.14
0.999
8
256
33.60
0,999
16
64
29.40
0.999
16
128
29.44
0.999
16
256
35.11
18


Reviews:
[{"review_id": "iQD8hapGTL", "reviewer": "Reviewer_oGKQ", "summary": "This paper presents a Humanoid Generalist Autoencoding Planner (H-GAP), consisting of a VQ-VAE that maps state-action trajectories into potentially discrete codes based on MoCapAct, a transformer capable of modeling future distributions based on past observations, and an MPC planning algorithm based on multiple trajectories. Extensive experiments show that H-GAP can benefit from larger and more diverse datasets, demonstrating the potential of this line of work.", "questions": "I'm not sure I understand correctly that the whole framework doesn't use a simulator at all. So what's the point of modeling action trajectories here, why not just state trajectories? And if so, why don't we just follow the existing kinematics-based human motion generation?\n\nOverall, the author's response to the concerns is needed to make the final decision. I am happy to increase the rating if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed H-GAP is even simpler than the existing offline RL methods while it is general for different downstream control tasks, meaning that the algorithm does not need access to the simulator and train different high-level policies.\n\nExperimental results also show that the proposed H-GAP can outperform offline RL policy and other traditional MPC algorithms.", "weaknesses": "Looking at the visualization on the website, I'm wondering why there is severe jitter compared to the ground truth sequence. I'm not sure if this is a result of the transformer not modeling temporal smoothness well, or if VQ-VAE doesn't represent state-motion clips well.\n\nI'm still not fully convinced of the ability of the proposed method to model a variety of humanoid motions (e.g., backflips) that have more complex patterns than locomotion. While the paper emphasizes data and model scaling experiments, which I appreciate, train data is a random sampling of the dataset, not of how performance changes when more action categories are added.\n\nI guess a potential limitation is the need to generate multiple trajectories. I would like to know how efficient the proposed method is, what the number of trajectories that typically need to be sampled, and whether these hyperparameters need to be changed to accept more trajectories as the motion database becomes more complex.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UyNcWz36Hl", "reviewer": "Reviewer_kAwr", "summary": "This paper tackles the challenging problem of general humanoid control, which is difficult due to high-dimensional action spaces and instability. The authors propose a Humanoid Generalist Autoencoding Planner (H-GAP) that learns a forward dynamics latent space model from the MocapACT dataset and propose model predictive control for downstream tasks. The forward dynamics model is trained using a transformer over discrete latent codes learned using a VQVAE that employs multiple codes per transition. The effectiveness of H-GAP is demonstrated through experiments on a 56 DoF humanoid across imitation benchmarks as well as analyses of its scaling properties on different model sizes.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Trains a single prior transformer for motion completion in a more principled way to jointly learn a motion prior and forward dynamics model that can be queried autoregressively and used for MPC planning. Introduces multi-code latent space modeling with VQVAE and transformer to accurately capture complex humanoid state-action sequences.\n    \n- Analyzes scaling properties on model sizes up to 300M parameters, revealing limitations in downstream task performance despite accuracy gains.\n    \n- Demonstrates strong imitation learning on a 56 DoF humanoid, validating the model's ability to represent diverse motor behaviors from the MocapAct dataset.", "weaknesses": "- The paper could analyze the coverage of different motion skills by randomly sampling initial states and generating rollouts from the model. This would help quantify the diversity of motions that can be produced [1].\n    \n- For analyzing imitation abilities, the model could be conditioned on a small segment of reference trajectories rather than just initial states. This would better evaluate how well it can leverage larger context sequences. The current approach in Table 1 mainly reveals modeling/memorization capabilities rather than imitation abilities.\n    \n- Additional experiments could be designed to isolate the planning contributions of the model beyond just trajectory modeling. For example, a conditional generation task could be constructed using out-of-distribution initial states in order to see if the model is able to “catch-up” to imitate a trajectory from a different initial polse from the original trajectory.\n    \n- More complex downstream tasks requiring long-term planning could be explored to go beyond trajectory imitation. The current tasks are limited to state distributions from the training data. New tasks like going to specified locations, or following a particular heading and and orientation[1] would require guiding the motion generation process. This could reveal how well the model can piece together behaviors.\n    \n- Overall, evaluating the model on tasks and scenarios that require leveraging the full context sequence in a generative way would provide better insight into the planning abilities. The current experiments focus primarily on trajectory modeling in different flavors. (imitation based Downstream tasks and imitation benchmark)\n    \n\n[1] Dou, Zhiyang et al. “C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters.” *ArXiv* abs/2309.11351 (2023): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xmye6iPuQe", "reviewer": "Reviewer_bwbh", "summary": "This paper incorporates VQ-VAE, a prior transformer, and MPC planning to construct H-GAP, a state-action generative model for humanoid trajectories. Experiments show that H-GAP learns to generate a wide range of human behaviors through imitation learning and flexibly transfers to unseen downstream tasks through planning without retraining.", "questions": "- What are the metrics used in table 1 and table 2?\n- How long does it take to train H-GAP?\n- I would assume that the optimal number of samples $N$ corresponds to the length of the intended trajectory $T$. What is the choice of $N$ for your experiments when $T=M=16$ and $L=4$?\n- Following up from the above question, how long does it for H-GAP to produce the reported results in table 2? How does it compare to the other methods?\n- Does the temperature change throughout algorithm 1? What are the choices of $\\Upsilon$ and $\\rho$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method outperforms MLP-backed behavior cloning in imitation learning by a large margin. Despite being a generalist policy, it also outperforms MPPI in most tasks by large margins in downstream tasks and performs reasonably well against offline RL specialist policies. \n\nWe also observe that the performance of imitation learning increases as the model and data scale up, paving the way to a possible future foundation model for humanoid control. However, we observe a drop in downstream task performance as the model size increases.", "weaknesses": "1. The provided link in the abstract results in 404 not found, and this submission comes without supplementary videos. This, unfortunately, makes it difficult to assess the naturalness of the generated human motion, which is critical to humanoid motion generation. \n2. The paper claims that H-GAP provides a basis for a foundation model for humanoid control. However, only the performance of imitation learning improves with the model scale. The performance of downstream tasks drops for larger models. In my view, this does not constitute a foundation model's basis because supporting downstream tasks is the most important feature of a foundation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)