Universal Humanoid Motion Representations for Physics-Based Control
Authors: Zhengyi Luo
Jinkun Cao
Josh Merel
Alexander Winkler
Jing Huang
Kris M. Kitani
Weipeng Xu
Venue: ICLR 2024 spotlight (2024)
Keywords: humanoid control
motion generation
physics simulation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present a universal motion latent space for physics-based humanoid control.

Abstract: We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.

Published as a conference paper at ICLR 2024
UNIVERSAL HUMANOID MOTION REPRESENTATIONS FOR PHYSICS-BANE CONTROL
Zhengyi Luo! 1.2
Jinkun Cao²
Josh Merel'
Alexander Winkler!
Jing Huang'
Kris Kitani1.2*
Weipeng Xu
Reality Labs Research, Meta; CCarnegie Mellon University
https:://heete luo. gi thub. 10/PUSSE/
ABSTRACT
We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high di- mensionality of humanoids and the inherent difficulties in reinforcement learn- ing, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representa- tion space. To achieve this, we first learn motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our mo- tion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RI, we show that our policies solve tasks using human-like behavior. We demonstrate the ef- fectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.
INTRODUCTION
Physically simulated humanoids have a broad range of applications in animation, gaming, AR/VR, robotics, etc., and have seen tremendous improvement in performance and usability in recent years. Due to advances in motion capture (MoCap) and reinforcement learning (RL), humanoids can now imitate entire datasets of human motion (Luo et al., 2023), perform dazzling animations (Peng et al., 1), and track complex motion from sparse sensors (Winkler et al., 2022). However, each of these tasks requires the careful curation of motion data and training new physics-based policy from scratch. This process can be time-consuming and engineering heavy, as any issue in reward design, dataset curation, learning framework can lead unnatural and jarring motion. Thus, though they can create physically realistic human motion, simulated humanoids remain inapplicable en masse.
One natural way reuse pre-trained motion controllers is to form latent space/skill embedding that can be reused in hierarchical RL. First, task such as motion imitation (Merel et al., 2018; Bohez et al., 2022; Won et al., 2022; Merel al., 2020; Yao et al., 2022) or adversarial learning (Peng et al., 2022; Tessler et al., 2023; Luo et al., 2020a) is used to form a representation space that can be translated into motor action using a trained decoder. Then, for new tasks, a high-level and task-specific policy is trained to generate latent codes as action, allowing efficient sampling from a structured action space and the reuse of previously learned motor skills. The main issue of this approach is the coverage of the learned latent space. Previous methods use small and specialized motion datasets and focus on specific styles of movements like locomotion, boxing, or dancing (Peng et al., 2022; Tessler et al., 2023; Hasenclever et al.; Zhu et al., 2023; Won et al., 2022; Yao et al., 2022). This specialization yields latent spaces that can only produce a narrow range of behaviors
"qqual Advising.

---

Published as a conference paper at ICLR 2024
Figure 1:
We propose to learn a motion representation that can be reused universally by downstream tasks. From left to right: speed, strike target, complex terrain traversal, and VR controller tracking.
preordained by the training data. Attempts to expand the training dataset, such as to CMU MoCap (CMU, 2002), have not yielded satisfactory results (Won et al., 2022; Yao al., 2022). Thus, these motion representations can only be applied to generative tasks such as locomotion and stylized animation. For tasks such as free-form motion tracking, they are limited by their motion latent spaces' coverage of the wide spectrum of possible human motion.
Another way to reuse motor skills is to formulate the low-level controller as motion imitator, with the high-level control signal being full-body kinematic motion. This approach is frequently used in motion tracking methods (Yuan et al., 2022; 2021; Won et al., 2020; Zhang et al., 2023a; Luo al., 2022; 2021; 2023), and trained with supervised learning with paired input (e.g. video and kinematic motion). In generative tasks without paired data, RL is required. However, kinematic motion is a poor sampling space for RL due to its high dimension and lack of physical constraints (e.g. small change in root position can lead to a large jump in motion). To apply this approach to generative tasks, an additional kinematic motion latent space such as MVAE (Zhang et al., 2023a; Ling et al., 2020) or HuMoR (Rempe et al., 2021) is needed. Additionally, when agents must interact with their environment or other agents, using only kinematics interaction dynamics. This makes tasks such as object manipulation and terrain traversal difficult.
motion signal does provide insight into
In this work, demonstrate the feasibility of learning universal motion representation for hu- manoid control. Our proposed Physics-based Universal motion Latent SpacE (PULSE) is akin to foundation model for control where downstream tasks ranging from simple locomotion, complex terrain traversal, to free-form motion tracking can all reuse this representation. To achieve this, we leverage recent advances motion imitation (Luo et al., 2023) and distill comprehensive motor skills into a probabilistic latent space using variational information bottleneck. This formulation ensures that the learned latent space inherits the motor skills of the motion imitator trained on a large-scale dataset. Our key insight lies in using a pretrained imitator for direct online distillation, which is crucial for effective scaling to large datasets.
To enhance our model's expressiveness, oception (humanoid's 's own pose and velocities). Random rollouts from this prior using Gaussian noise lead to diverse and stable humanoid motion. This is essential improve sampling efficiency for downstream tasks since the high-level controller now samples coherent human motion for ex- ploration instead of relying on applying noisy joint torques. The use of this prior in hierarchical RI. speeds up training for the motion tracking task and leads to human-like behavior for generative tasks such complex terrain traversal (Rempe al., 2023) without using adversarial reward.
jointly train learnable prior conditioned on propri-
To summarize, our contributions are as follows: 1) We show that one can form universal humanoid motion latent space by distilling from imitator that can imitate all of the motion from large scale motion dataset. 2) We show that random rollouts from such latent space can lead to human-like motion. 3) We apply this motion representation generative tasks ranging from simple locomotion to complex terrain traversal and show that it can speed up training and generate realistic human--ike behavior when trained with simplistic rewards. When applied motion tracking tasks such as VR controller tracking, our latent space can produce free-from from human motion.
RELATED WORK
Physics-based Humanoid Motion Latent Space. Recent advances often use adversarial learning or motion tracking objective to form reusable motion representation. Among them, ASE (Peng et al., 2022) and CALM (Tessler et al., 2023) use a discriminator or encoder reward to encourage mapping between random noise and realistic behavior. Although effective on small and specialized datasets (such as locomotion and sword strikes), the formed latent space can fail to cover more diverse motor skills. Using explicit motion tracking, the other hand, can potentially form latent space that
2

---

Published as a conference paper at ICLR 2024
covers all the motor skills from the data. ControlVAE (Yao et al., 2022), NPMP (Merel et al., 2018), MrysicsVAE(War et al., 2022) and NCP (Zhu et al., 2023) all follow this methodology. NCP learns the latent space jointly through the imitation task using RL, while ControlVAE and Physics VAE use an additionally learned world model. They show impressive results on tasks like point-goal, maze traversal, boxing, etc, but for tasks such as handling uneven terrains, adaptation layers required (Won et al., 2022). We also use motion tracking as the task to learn motion representation, but form universal latent space that covers 40 hours of MoCap data. We show that the key universal latent space is distilling from a pretrained imitator and jointly learned prior.
Kinematics-basse baaed Human Motion Latent Space. Kinematics-based motion latent space has achieved great success in motion generation and pose estimation. In these approaches, motion representation is learned directly from data (without involving physics simulation) via supervised learning. Some methods obtain latent space by compressing multi-step motion into a single code (Li et al., 2021; Wang et al., 2021; Yuan & Kitani, 2020b; Luo al., 2020b; Petrovich et al., 2021; Guo et al., 2022; Zhang et al., 2023b; Jiang et al., 2023; Lucas et al., 2022). Some, like MVAE (Ling et al., 2020) and HuMoR (Rempe et al., 2021), are autoregressive models that model human motion at the per-frame level and are more applicable to interactive control tasks. While MVAE is primarily geared toward locomotion tasks, HuMoR is trained on AMASS (Mahmood et al., 2019) and can serve universal motion prior. Our formulation is close to HuMoR while involving control dy- namics. Through random sampling, our latent space governed by the laws of physics can generate long and physically plausible motion, while HuMoR can often lead to implausible ones.
Physics-based Humanoid Motion tracking. From DeepMimic (Peng et al., 2018), RL-based mo- tion tracking has gone from imitating single clips to large-scale datasets (Chentanez et al., 2018; Wang et al., 2020; Luo et al., 2023; Fussell et al., 2021). Among them, a mixture of experts (Won al., 2020), differentiable simulation (Ren et al., 2023), and external forces (Yuan & Kitani, 2020a) have been used to improve the quality of motion imitation. Recently, Perpetual Humanoid Con- troller (PHC) (Luo et al., 2023) allows single policy to mimic almost all of AMASS and recover from falls. We improve PHC to track all AMASS and distill its motor skills into a latent space.
Knowledge Transfer and Policy Distillation. In Policy Distillation (Rusu al., 2015), student policy is trained using teacher's collected experiences via supervised learning (SL.. In kickstarting (Schmitt et al., 2018), the student collects its own experiences and is trained with both RL and SL. As we distill motor skills from a pretrained imitator, we also use the student for exploration and the teacher to annotate the collected experiences, similar to DAgger (Ross et al., 2010).
3 PRELIMINARIES
We define the full-body human pose as q. _ (0., P.)), consisting of 3D joint rotation 0, E R//x and position p, E /x3 of all J links on the humanoid, using the 6 Segree-of-freedoor (DOF) rotation representation (Zhou et al., 2019). To describe the movement of human motion, we include veloci- ties Q1.7, where q. (w,, 1h)) consists of angular w, E / and linear velocities v, E R2x3 As notation convention, we use to denote the ground truth kinematic quantities from Motion Capture (MoCap) and normal symbols without accents for values from the physics simulation. A MoCap dataset is called Q We use the terms "track", "mimic", and "imitate" interchangeably.
Goal-conditioned Reinforcement Learning for Humanoid Control. In this work, all tasks use the general framework of goal-conditioned RL. Namely, a goal-conditioned policy is trained to complete tasks such as imitating the reference motion qı:r, following 2D trajectories TI. tracking 3D VR controller trajectories PI-T, etc. The learning task is formulated as Markov Decision Process (MDP) defined by the tuple M = AS.A, T, R, 7) of states, actions, transition dynamics, reward function, and discount factor. Physics simulation determines the state 8, S and transition dynamics T, where a policy computes the action a, For humanoid control tasks, the state 8t contains the proprioception sf and the goal state sy Proprioception is defined as s. A (q., q:), which contains the 3D body pose q, and velocity q.. The goal state of is defined based on the current task. When computing the states of and sf. all values are normalized with respect to the humanoid heading (yaw). Based on the proprioception sf and the goal state sf. we have a reward = R(s sf) for training the policy. We use proximal policy optimization (PPO) (Schulman
et al,, 2017) to maximize discounted reward E
DE -
Our humanoid follows the kinematic
3

---

Published as a conference paper at ICLR 2024
Speed
Skikk Target
Centroller ligat
Temah and Tigeetory
Notion Detabane
Task Input
Rabe: Controlergses
Secoder
Pollyy
Propfixeptoon
Denestiorns
We form our latent space by directly distilling from a pretrained motion imitator that can imitate all of the motion sequences from a large-scale dataset. A variational information bottleneck is used to model the distribution of motor skills conditioned proprioception. After training the latent space model, the decoder D and prior R. are frozen and used for downsteam tasks.
Figure 2:
structure of SMPL (Loper al., 2015) using the mean shape. It has 24 joints, of which 23 actuated, resulting in an action space of a E p23x3, Each degree of freedom is actuated by proportional derivative (PD) controller, and the action a, specifies the PD target.
PHYSICS-BASED UNIVERSAL HUMANOID MOTION LATENT SPACE
To form the universal humanoid motion representation, we first learn to imitate all motion from a large-scale motion dataset (Sec.4.1). Then, we distill the motor skills of this imitator into latent space model using a variational information bottleneck (4.2). Finally, after training this latent space model, we use it as an action space for downstream tasks (4.3). The pipeline visualized F
4.1 PHC+: ONE POLICY TO IMITATE THEM ALL For motion imitation, we follow the Perpetual Humanoid Controller (PHC Luo et al. (2023)) and train& a motion imitator that can recover from fallen states. TP s.. ot s-mmiiic) is formulated as a per-frame motion imitation task, where at each frame, the goal state 8 2--mimic is defined as the difference between proprioception and one frame reference pose i(+1: 8
g-mimic A 0 Prt1
Pt, 6c+1 Ut, W. wi: Q.+1, Pt41). PHC utilizes progressive training strategy, where primitive policy (0)) is first trained on the entire motion dataset Q The success rate regularly evaluated on Q and after it plateaus, sequences in which (0) fails on form (0) and a new primitive p(1) is initialized to learn hard This process continues until Q cannot be learned or contains no motion.
To learn to recover from the fail-state, an additional primitive (P()) is trained. After learning the primitives, a composer is trained to dynamically switch between frozen primitives. PHC achieves a 98.9% success rate on the AMASS training data, and we make small, yet critical modifications to reach 100% to form our imitator, PHC+.
First, extensive analysis of PHC reveals that the MoCap training dataset still contains sequences with severe penetration and discontinuity. These motions doubly impede PHC due to its hard-negative mining procedure, and removing them improves performance. Second, by immediately initializing ((++1) to learn from (+) PHC deprives pP()) of any opportunity to further improve using own
hard
hard negatives. Focusing p(t) on Q ( before instantiating ((t+1) better utilizes (())ss capacity, and coupled with some modernizing architectural changes, we use only three primitives to learn fail-state recovery and achieve success rate of 100%. More details in the supplementary.
4.2 LEARNING MOTION REPRESENTATION VIA ONLINE DISTILLATION
After learning PHC+ Tyuc+ (a, PHC+ sr st mimie) through RL, we learn latent representation via on- line distillation. We adopt an eecoder- structure and use conditional variational information bottleneck to model the distribution of motor skills P [f st g-mimic) produced in ТтиIC+ Although similar to the VAE architecture, our model does not use any reconstruction loss and directly op- timized in the action a, space. Specifically, we have a variational encoder E(z,ss] s a-mimie, that computes the latent code distribution based on current input states and a decoder D D(a, |s,, 21) that produces action. Inspired by HuMoR (Rempe et al., 2021), we employ a learned conditional prior R R(2, (sf) instead of the zero-mean Gaussian prior used in VAEs. The learnable prior R allows
4

---

Published as a conference paper at ICLR 2024
the model to learn different distributions based on proprioception sf. as the action distribution for a person standing still and flipping midair can be significantly different. Formally, we model the encoder and prior distribution as diagonal Gaussian:
E |s. ot g-mimic) = (Z of), ([(z)ls))==N(C = (Z of)-
(1)
During training, we sample latent codes from the encoder distribution žn (z of) and use the decoder D to compute the action. Combining the three networks, we obtain the student policy TPULSH (E,D, R). Together with the learnable prior, the objective can be written as:
log P(aslsp si ) 2 Eg (log ((a, [sf, =1]] DKL (E (E(e s s.-mimic) > (=(((s)))
(2)
using the evidence lower bound. The data term Eg is similar to the reconstruction term in VAEs, and the KI. term encourages the distribution of the latent code to be close to the learnable prior. To optimize this object, the loss function is written as:
L = Leccionn + oeCregu + 3LLLL.
which contains the data term Caction. the learnable prior LKL from Eq.2, and regulization term Lregu- For online distillation, we have Laction = ||a - az In our experiments, we find that unconstrained, the learnable prior can push the latent codes far away from each other even for si that are close. Thus, we introduce an additional regularization term Legu = ll - - 112 that penalizes a large deviation of consecutive latent codes. Intuitively, temporally close transitions should have similar latent representations, providing a smoother and more continuous latent space. fregu serves as weak prior, similar to the AR(1) prior used in NPMP (Merel et al., 2018). In ablations, without which the latent space can be very discontinuous and hard to navigate during RL exploration. To maintain the delicate balance of reconstruction error and KLD, we anneal / gradually.
a 12. similar to the reconstruction term in VAEs.
show that using this prior is crucial for downstream tasks,
For training TUULSE we need pairs of (sf, .5. -.-minic a2s (HHC+, which are obtained by rolling TPULSE in the environment for the motion imitation task and querying TPC*+- While we can opti- mize 7PULSE with the RL objective by treating the decoder D as Gaussian distribution with fixed diagonal covariance matrix (similar to kickstarting (Schmitt et al., 2018)), we find that sampling the latent code together with random exploration sampling for RL introduces too much instability the system. In our experiments, we show that training with RL objectives alone is not sufficient to form a good latent space, and optimizing with RL and supervised objectives combined creates a noisy latent space that is worse than without The same as PHC+'s training process, we perform hard- negative mining and form Quadd for progressive training. When evaluating FPULSE, we use the mean latent code f instead of sampling from N(z, Int of). This progressive training adds the benefit of balancing the data samples of the rarer and harder training motion sequences with the simpler ones.
4.3 HIERARCHICAL CONTROL FOR DOWNSTREAM TASKS
After TPuLSE has converged, we can train a high-level policy Tusk (2. .f) for downstream tasks. The frozen decoder D, together with the simulation, can now be treated as the new dynamcis sys- tem (Haarnoja et al., 2018), where the action is now in the z; space. Each high-level task policy Tisk (2) sp. sf) = N( usk (s], (f]; a lask) represents a Gaussian distribution with fixed diagonal covariance. We test prior, our policy can create realistic human-like behavior for downstream tasks, without relying on any adversarial learning objectives. The full training pipeline can be found in Alg.
suite of generative and motion tasks, and show that sampling from our
Hierarchical Control with Learnable Prior. During high-level policy Task training, it is essential to sample action from the prior distribution induced by R. Intuitively, if PULSE's latent space en- capsulates a broad range of motor skills, randomly sampled codes may not lead to coherent motion. The learaale prior provides good starting point for sampling human-like behavior. To achieve this, we form the action space of Task as the residual action with respect to prior's 's mean μu and compute the PD target a; task for downstream tasks as:
a task D (Ttss (2) task S.. 5 + p4)).
where μu is computed by the prior (0.22) instead of of as we notice that of tends to be rather small. In ablations, we show that not using this residual action formulation for downstream task leads significant drop in performance.
For RL exploration for Tusk. we use fixed variance
5

---

Published as a conference paper at ICLR 2024
Algo 1: Learn PULSE and Downstream Tasks
Function TrainPULSE (E, Q stwc+):
Input: Ground truth motion dataset Q. pectrained PHC+ encoder &, decoder D. and prior R.
while nor converged regeeeeddd
M. 0 initialize sampling memory
while M not jull do
sample motion and initial state from Q
1...T do
8
#e t &f mine
9
Mi of +- E(zi ff a puink)
// encode latent;
// reparamaterizatee trick for sampling latenta;
D(a.la) (a, xx)
decode actiton; // simulation;
12
0c+1 + [(8++1 (Be41/ee (8..).))..))))
into memory M
14
MIC+ -- TTn- (a
(sx) Anmotate collected states using mincc
15
ni of + R(z Compute prior distribution based on propriocetion:
16
R. D +- supervisse update for encodor, decoder, and prior using pairs of (a, at 14, of -4 of) and Eq..
17
return E, D.R. R.
18
Function TrainCownst Tankaakk (D, R, Q, Auakl
19
Input: Pretrained PULSE''s decoder D and prior R, lask definition, and motion dataset for initial state sampling (e.g. Q):
29
while converged
21
M t- @ initialize sampling memory
while not da
sample initial from Q for
24
I...T do
25
z Tiskk Fasala.ee aafaaa/f.a. of (
11 usee pretrained latenz space as action space: If compute prior lazent code;
26
H of - R(z((=))
a D(aa |a #/ch mF)
If decode action using pretrained decoder;
#1+1 --((#4+ 14.41118 1 a
// simulation;
29
Pi <- R(a; 4.
/1 computa reward;
(#c. Pei (ct1) memory
Tisk
update using experiences collected in M
32
5 EXPERIMENTS
Tasks. We study suite of popular downstream tasks used in the prior art. For generative tasks, we study reaching a certain x-directional speed (between 0 m/s and 5m/s), striking a target with the right hand (target initialized between 1.5~5m), reaching 3D point with the right hand (point initialized within meters the humanoid), and following trajectory on complex terrains (slope, stairs, uneven surfaces, and obstacles). For motion tracking, we study a VR controller tracking task, where the humanoid tracks three 6DOF rigidbodies in space (two hand controllers and a headset). Note that the VR controller tracking task can be viewed imitation task, where only partial observation is provided. It is challenging since it requires the latent space to contain motor skills match arbitrary user input in a continuous and streaming fashion. Please refer to the supplement for complete definition of the tasks.
generalization of the full-body motion
Datasets. For training PHC+, PULSE, and VR controller policy, use the cleaned AMASSS train- ing set. For strike and reach tasks, we use the AMASS training set for initial state sampling. For speed and terrain traversal tasks, we follow Pacer (Rempe et al., 2023) to use a subset of AMASS that contains only locomotion for initial state sampling. All methods are trained using the same initial-state sampling strategy. For testing imitation and VR controller tracking policy, we use the AMASS test set, as well real-world dataset (14 sequences, 16 minutes) from QuestSim (Winkler et al., 2022) collected using the Quest headset.
To compare with state-of-the-an latent space models, we train ASE (Peng et al., 2022) and CALM (Tessler et al., 2023) with the officially released code. We train all the latent space models from scratch using the AMASS training set with around I billion samples. Then, the learned latent space to downstream tasks. Both CALM and ASE use a 64-d latent code projected onto a unit sphere, and are designed to be multi-frame latent codes operating at 6 HZ. Since policy produces a per- frame latent code (30 Hz), for a fair comparison, we also test against ASE and CALM operating at 30 Hz. Notice that by design, CALM is trained to perform downstream tasks
Baselines.
apply

---

Published as a conference paper at ICLR 2024
Table I: Motion imitation result (adata cleaning) on AMASS train test (11313 and 138 sequences).
AMASS-Train'
AMASS-Tes*
Method
Suce
Emprre 4
Fase
Fod 4
Suce
Figepge
Pamppe
Faxz +
Fact
PLC
98.9
37.5
26.9
3.3
4.9
97.1%
47.5
40.0
6.8
9.1
PHC+ PULSE
100 %
3.9
36.1
99.8 %
39.2
35.0
3.1
5.2
97.1
54.1
43.5
7.0
10.3
3/Brach
34 Certeder Teckk
Figure 3: (a. b, c, d) Policy trained using motion representation solves tasks with human-like behavior. (e) Our latent space is constrained certain movement styles support free-form tracking. (f) Random sampling from learned prior R leads to human- like movements as well as the recovery from fallen state.
with user-specified skills. However, since we train our generative task without a specific style in mind, we opt out of the style reward and only train CALM with the task reward. For all tasks, we also compare against a policy trained from scratch without latent space or adversarial reward.
Metrics. For generative tasks (reach, speed, strike, and terrain), we show the training curve by visualizing the undiscounted return normalized by the maximum possible reward per episode. For motion imitation and VR controller tracking, we report the popular full body mean per joint position enror for both global Eg-mpine and root-relative Empipe (in mm), and physics-based metrics such as ac- celeration error Face (mm/frame") and velocity eтror Fvel (mm/frame). Following PHC, we measure the success rate Suce (the success rate is deñned as following reference motion 0.5 m, averaged joint, at all points in time). For VR controller tracking, the success rate is measured against only the three body points, but we report full-body Eg-mpire and Empipe when they are available (on AMASS). On the real-world dataset without paired full-body motion, we report controller tracking metrics Eg-miip (mm) which computes the global position error of the three tracked controllers.
Implementation Details. Simulation is conducted Isaac Gym (Makoviychuk et al., 2021), where policy run at Hz and the simulation at 60 Hz. For PHC+ and all downstream tasks, each policy primitive is a 6-layer MLP. Our encoder E, decoder D. prior R, CALM, and ASE three-layer MLPs. The latent space 32d: 2, E 3-32 about half of the humanoid DOF of 69.
5.1 MOTION IMITATION
First, we measure the imitation quality of PHC+ and PULSE after distilling from PHC+. Table shows that after distillation using the variational bottleneck, PULSE retains most of the motor skills generated by PHC+. This is crucial this shows that the learned latent space model (the decoder D) inherits most of the motor skills to perform the majority of the AMASS dataset with high accuracy. Note that without the variational bottleneck, distillation can reach 100% success rate, and the degradation in performance is similar to con-zero reconstruction error of a VAE.
5.2 DOWNSTREAM TASKS
Generative Tasks. We show the qualitative result in Fig.3 for the generative tasks (speed, reach,

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative results on VR Controller tracking. We report result on AMASS and real- world dataset.
AMASS Train"
AMASS- Tess
Real world
Method
Suce
Epappe
Exare
Eax E,
Suce
Equseto:
Enagge
Ex:
Ex
Eyeetge
Ex::
Exc
ASE-30Hz
79.8%
103.0
80.8
37.6%
120.5
83.0
19.5
7/14
W9.0
28.2
20.3
ASE-6Hz
74.2%
113.9
84.9
81.9
57.7
33.3%
136.4
98.7
117.7
80.3
4/14
114.7
115.8
72.9
Caim.
16.6%
130.7
100.8
2.7
7.9
10.1%
122.4
99.4
1.2
12.9
2/14
206.9
2.0
6.6
Calm 6l1x
14.6%
137.6
105.2
58.9
47.7
10.9%
147.6
115.9
82.5
122.3
0/14
98.8%
51.7
47.9
80.4
67.4
8.0
14/14
4.3
6.5
Ours
99.5%
57.8
51.0
3.9
7.1
93.4
88.6
61.7
9.1
14.9
14/14
68.4
5.8
9.0
spoad
06
03-
04
12
02-
0.1
01-
0.0
5--
0.75
3.09
125
camples 300-3000
cumpaie
260-611
acratch
Training curves for each one generative tasks. Using our motion representation improves training speed and performance as the task policy can explore in an informative latent space. Experiments run for 3 times using different random seeds.
Figure 4:
strike, and terrain). From Fig.4 we can see that using our latent space model as the low-level con- troller outperforms all baselines, including the two latent space models and training from scratch. Compared to latent space models (ASE-30Hz, ASE-6Hz, CALM- 30Hz, CALM-6H22 our method consistently provides a better ac- tion space, achieving better normalized return across the board. This shows that the ASE and CALM formulation, though effec- tive in learning motion styles from curated datasets, could not ade- quately capture motor skills large and unorganized dataset like AMASS. In general, 30 Hz models (ASE-30Hz and CALM-30HZ) outperform Hz ones, signaling the need for finer grained control latent space. In general, ASE outperforms CALM, partially due to CALM's motion encoder formulation. Without the style reward, exploration might be hindered for downstream tasks. For all tasks, training from scratch provides the closest normalized return to ours, but this could lead to unnatural behavior, observed in ASE and our qualitative results. Our method also converges faster, even for the challenging terrain traversal task, where the humanoid needs to handle obstacles and stairs that it has not seen during training. Upon inspection, we can see that the humanoid uses human-like behavior for navigating the obstacles without using any style/adversarial reward as guidance. When stepping on stairs quickly, we can see that our humanoid employs motor skills similar to jumping. For detailed qualitative evaluation, see supplement site.
Controllr badding
6.0
02
2411
Figure 5: Success rate com- parison between training from scratch and using our motion rep- resentation during training. Ours converges faster.
VR Controller Tracking. Unlike generative tasks, we can directly measure the success rate and joint position error for this task. In Table 2, we report the motion tracking result in the training. test, and real-world dataset. Continuing the trend of generative tasks, compared to other latent space models, ours significantly improves tracking error. This validates the hypothesis that PULSE can capture a wide range of human motor skills and can handle free-form motion tracking and pose estimation tasks. On the real-world dataset, our method can faithfully track all sequences. Note that our tracking result is comparable to training from scratch, which signifies that our latent space diverse enough that it does not handicap exploration. Since the three-point tracking problem is relatively close to the full-body tracking problem, we expect training from scratch to perform better since it is a specialized tracking policy. Using our latent space also seems to be trading success rate for precise tracking, sometimes sacrificing precision in favor of staying stable. Figure 5 shows the training success rate between training from scratch and using our latent space. We can see that, using our latent space, the policy converges faster.
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation on various strategies learning the motion representation. use the challenging controller tracking task to demonstrate the applicability of the latent space for downstream tasks. Prior: whether to use a learnable prior. Prior-action: whether to use the residual action with respect to the learned prior R. feregu whether to apply fregu. No RL; whether to train PULSE together with the RI. objective.
AMASS
idx
Prior
Prior action
Cuggs
No RL
Suce 1
Exappe
Enpip
Eusc 4
Evdl
x
x
x
36.9%
114.6
79.4
8.1
15.8
45.6%
7.5
x
60.8%
106.8
72.4
9.4
16.0
/
95.1
72.8
x
18.1%
108.6
83,8
11.2
16.4
93.4%
88.6
67.7
9.1
14.9
5.3 RANDOM MOTION GENERATION
As motion is best seen in videos, we show extensive qualitative random sampling and baseline com- parison result in suppl ement site and Fig.3 also shows qualitative sample. Comparing with the imeeatiics-based HuMoR, our method can generate realistic human motion without suf- fering from frequent implausible motion generation. Compared to other physics-based latent space, ASE and CALM, our representation has more coverage and can generate more natural and realistic motion. By varying the variance of the input noise, we can control the generation process and bias it toward smoother or more energetic motion.
5.4 ABLATIONS
In this section, we study the effects of different components of our framework using the challenging VR controller tracking task. During our experiments, we noticed that while the generative down- stream tasks are affected less by the expressiveness of the latent space, the VR controller tracking task is. We train our latent space using the same teacher (PHC+) and train the VR controller tracking policy on the frozen decoder. When comparing Row 1 (R1) vs. R2 as well as R3 vs. R6, we can see that the regularization term Cregu that penalizes a large deviation between consecutive latent codes is effective in providing a more compact latent space for downstream exploration. Without such reg- ularization, the encoder can create a discontinuous latent space that is harder to explore. R6 trains policy that does not use the learnable prior but uses the zero-mean Gaussian AO 1) in VAEs. R2 vs. R6 shows the importance of having a learnable prior, without which downstream sampling could be diffieult. RS R6 shows that using the learnable prior as a residual action described in Eq.4 is essential for exploration, without which the latent space is too hard to sample from (similar the issues with CALM without style reward). R4 vs. R6 shows that mixing the RL training and objective and online supervised distillation has a negative effect.
6 DISCUSSION AND FUTURE WORK
Failure Cases. While PULSE demonstrated that is feasible to Iearn universal latent space model, it far from perfect. First, TUULSE does not reach a 100% imitation success rate, meaning that the information bottleneck is lossy compression of the motor skill generated by PHC+. In our experiments, online distillation can reach 100% success rate using unconstrained latent space (without the variational information bottleneck). The variational formulation enables probabilistic modeling of motor skills, but introduces additional challenges. For downstream tasks, while our method can lead to human-like behavior, for tasks like VR controller tracking, it can lag behind the performance of training from scratch. For motion generation, the humanoid can be stuck in fallen or standing position, although increasing the noise into the system could jolt out of these states.
Future Work. In conclusion, we present PULSE, a universal motion representation for physics- based humanoid control. It can serve generative motion prior for downstream generative and estimation tasks alike, without the use of any additional modification or adaptive layers. Future work includes learning more human- ietereriterpretablee motion representation, incorporating scene informa- tion and buman-objec interactions, as well as articulated fingers.

---

Published as a conference paper at ICLR 2024
Acknowledgement. We thank Zihui Lin for her help in making the plots in this paper. Zhengyi Luo is supported by the Meta Al Mentorship (AIM) program.
REFERENCES
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasen- clever, Yuval Tassa, Emilio Parisotto, Jan Humplik, Tuomas Haarnoja, Roland Hafner, Markus Wulfmeier, Michael Neunert, Ben Moran, Noah Siegel, Andrea Huber, Francesco Romano, Nathan Batchelor, Federico Casarini, Josh Merel, Raia Hadsell, and Nicolas Heess. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXiv:2203.117 17138, 2022.
Nuttapong Chentanez, Matthias Müller, Miles Macklin, Viktor Makoviychuk, and Stefan Jeschke. Physics-based motion capture imitation with deep reinforcement learning. Proceedings MIG 2018: ACM SIGGRAPH Conference on Motion, Interaction, and Games, 2018.
CMU. CMU graphics lab motion capture database. http: htwp//ocppppp//ooo///////o..a... cs. cmu edu/, 2002.
Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. arXiv preprint arXiv:2210.10044 2022.
K Fukushima. Cognitron: a self-organizing multilayered neural network. Biol. Cybern., 20:121- 136, 1975. ISSN 0340- 12...11320000000
Levi Fussell, Kevin Bergamin, and Daniel Holden. Supertrack: motion tracking for physically simulated characters using supervised learning. ACM Trans. Graph., 40:1-13, 2021. ISSN 0730- 0301.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng, Tm2t:
Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. arXiv preprint arXiv:2207.0166 2022.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint wrx1v::804.02808, 2018.
Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. CoMie: Comple- mentary task learning & mimicry for reusable skills. In Hal Daumé li and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learming Research, pp. 4105-4115, PMLR.
Dan Hendrycks and Kevin Gimpel. Gaussian eиror linear units (gelus). arXiv preprint arXiv:1606.08415. 2016.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. arXiv preprint arXiv:2306.14995 2023.
Jiaman Li, Ruben Villegas, Duygu Ceylan, Jimei Yang, Zhengfei Kuang, Hao Li, and Yajie Zhao.
Task-generic hierarchical human motion prior using vaes. arXiv preprini arXiv:2106.044 04004, 2021.
Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Trans. Graph., 39:12, 2020. ISSN 0730-0301
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer, and Yoshua Bengio. Discrete-valued neural communication. arXiv preprint arXiv:2107.02367 2021.
Matthew Loper, Naureen Mahmoody, and Michael J Blackz. Mosh: Motion and shape capture from sparse markers. ACM Trans. Graph., 33, 2014. ISSN 0730-0301,,555-77366
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM Trans. Graph., 34, 2015. ISSN 1537--730
Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, and Grégory Rogez. Posegpt: Quantization- based 3d human motion generation and forecasting. arXiv preprint arXiv: 10542, 2022.
10

---

Published as a conference paper at ICLR 2024
Ying- Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. Carl: Controllable agent with reinforcement learning for quadruped locomotion. arXiv preprint 20HN::2005.03288, 2020a.
Zhengyi Luo, S. Alireza Golestanch, and Kris M. Kitani. 3d human motion estimation via mo- tion compression and refinement. In Proceedings of the Asian Conference Computer Viston (ACCV), November 2020b.
Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Denamtes-regulated kinematic policy for egocentric pose estimation. NeurlPS, 14:2001--55032, 2021. ISSN 1049-5258.
Zhengyi Luo, Shun Iwase, Ye Yuan, and Kris Kitani. Embodied scene-aware human pose estimation. NewrIPS, abs/2206.09106, 2022.
Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. arXiv preprimt arXiv:2305.0644 2023.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. Proceedings of the IEEE International Conference on Computer Vision, 2019-0ctob:5441-5555, 2019. ISSN 1550-5499.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Manfred Otto Heess. Neural probabilistic motor primitives for hu- manoid control. arXiv preprint arXiv: 181111,
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch and carry: Reusable neural controllers for vision-guided whole-body tasks. ACM Trans. Graph., 39, 2020. ISSN 1757000
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines, 2010.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic. ACM Trans. Graph., 37:1-14, 2018. ISSN 0730-0301.
Xue Bin Peng. Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learn- ing composable hierarchical control with multiplicative compositional policies. arXiv preprint 1rXiv:1905.09808, 2019.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Trans. Graph., abs/2104.021800 1-20, 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large- scale reusable adversarial skill embeddings for physically simulated characters. arXiv preprint arXiv:2205.01906 2022.
Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned human motion synthesis with transformer vae. arXiv preprint arXiv:2104.05670, 2021.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. arXiv preprint arXiv:2105.0466 2021.
Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. arXiv preprint 2xx1x22304.0189. 2023.
11

---

Published as a conference paper at ICLR 2024
Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. Diffmimic: Efficient motion mimicking with differentiable physics. arXiv preprint arxiv:2304.03274, 2023.
Stephane Ross, Geoffrey J Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. arXiv preprini arXiv: 10/ 1.0686, 2010.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. arXiv preprini urXiv:2109.11977 2021.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint arXiv:1511.06295. 2015.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and S M Ali Eslami. Kickstarting deep reinforcement learning. arXiv preprint arXiv: 1803v 03833, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06344 2017.
Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY, USA, 2023. Association for Comput- ing Machinery. ISBN 9798400701597.
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. Adv. Neural Inf. Process. Syst., 2017-Decce::337--6366 2017, ISSN 1049-5258.
Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, and Christian Theobalt. Estimating ego- centric 3d human pose in global space. arXiv preprint arXiv:2104. 13454, 2021.
Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler. Unicon: Universal neural controller for physics-based character motion. arXiv preprint arXiv:2011.15119, 2020.
Alexander Winkler, Jungdam Won, and Yuting Ye. Questsim: Human motion tracking from sparse sensors with simulated avatars. arXiv preprint (XXiv22209009391 2022.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. ACM Trans. Graph., 39, 2020. ISSN 0730-
0301,155.-7368.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics- based character controllers using conditional vaes. ACM Trans. Graph., 41:1-12, 2022. ISSN 0730-0301.
Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu, Controlvae: Model-based learning of generative controllers for physics-based characters. arXiv preprint arXiv:2210.06065 2022.
Ye Yuan and Kris Kitani. Residual force control for agile human behavior imitation and extended motion synthesis. arXiv preprint 2tt.::2000007:364 2020a.
Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction.
Lect. Notes Comput. Sci. 12354 LNCS:346-364 2020b. ISSN 0302-9743,1611-33399
Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. CVPR, abs/2104.00683, 2021.
Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. arXiv preprint arXiv:2212.02500, 2022.
Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. Learning physically simulated tennis skills from broadcast videos. ACM Trans. Graph., 42:1-14, 2023a. ISSN 07300000111155773333
12

---

Published as a conference paper at ICLR 2024
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general purpose motion generators. arXiv preprint arXiv:2306.10900 2023b.
general-purpose
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:5738 aam:5538-5776,, 2019. ISSN 1063-6919.
Qingxu Zhu, He Zhang, Mengting Lan, and Lei Han. Neural categorical priors for physics-based character control. arXiv preprint arXiv:2308.0702 072000 2023.
13

---

Published as a conference paper at ICLR 2024
APPENDICES
A Introduction
14
B Details about PHC+
14
B.1 Data Cleaning
14
B.2 Action and Rewards
15
B.3 Model Architecture and Ablations
15
C Details about PULSE
16
C.I Training Procedure
16
C.2 Comparison Training Scratch without Distillation
16
C.3 Comparison to Other Latent Formulation (VQ-VAE, Spherical)
16
C.4 Downstream Tasks
17
A
INTRODUCTION
In this document, we include additional details about our method that are omitted from the main paper due the page limit. In Sec.B, we include additional details about PHC+ and modifica- tions made to imitate all motion from a large-scale dataset. In Sec.C, we include additional details about method, PULSE, such architecture, training details, and downstream task configurations elc.All code and models will be released for research purposes.
Extensive qualitative results provided the project page well in the supplementary zip (the zipped version is of lower video resolution to fit the upload size). As motion is best seen in videos, we highly encourage the readers to view them to better understand the capabilities our method. Specifically, we evaluate motion imitation and fail-state recovery capabilities for PHC+ and PULSE after online distillation and show that PULSE can largely retain the abilities of PHC+. Then, we show long-formed motion generation result sampling from the PULSE' prior and decoder. Sam- pling from PULSE, we can generate long-term, diverse and human-like motions, and we can vary the variance of the input noise to control the behaviors of random generation. We also compare with SOTA kinematics- based method, HuMoR, and show that while HuMoR can generate unnat- ural motions, are regulated by the laws of physics and remain plausible. Compared to SOTA physics-bar basccd latent space(ASE and CALM), our random generation appears more diverse. Finally, show visualization for downstream tasks for both generative and estimation//racking tasks and compare them with SOTA methods.
B DETAILS ABOUT PHC+
B. DATA CLEANING
We perform a failure case analysis and identified two main sources of imitation failure. First, we have dynamic motion, such as cartwheeling and consecutive back flips. Another, often overlooked, is that MoCap sequences can still have a large discontinuity and penetration due to failures in the MoCap optimization procedure or the fitting process (Loper al., 2014). After filtering out human- object interaction data following UHC (Luo et al., 2021), we found additional corrupted sequences PHC's training data that have a large discontinuity or penetration. of the sequences we have identified and removed from the training data. Since we use the random state initialization proposed by DeepMimic (Peng et al., 2018), sampling frames that have large penetration could lead to the humanoid "flying off" from the ground the physics simulation applies a large ground reactionary force. Naively adjusting the height of the sequence based on penetration could lead to floating sequences discontinuity. Frames that have large discontinuities could lead
Fig.6, we visualize some
14

---

Published as a conference paper at ICLR 2024
Figure 6: Visualization of issues in the AMASS dataset. Here show sequences with corrupted poses, large penctration, and discontinuity. the second row, the red and yellow mesh are frame apart 120Hz MoCap.
Table 4: Hyperparameters for PHC+ and Pulse. a: fixed variance for policy. discount factor. e clip range for PPO. a: coefficient for fregu coefficient for LKL.
Method
Batch Size
Learning Rate
a
"ip
samples
PHC+
3072
2 x
0.05
0.99
0.2
0.5
0.3
0.1
0.1
~ 1020
Batch Size
Learning Rate
Lutent size of samples
PULSE
3072
5 x 10-1
0.005
0.01 + 0.001
10P
to imitation failure or humanoid learning bad behavior to anticipate large jumps between frames. We remove these sequences and obtain 11313 training and 138 testing motion suitable for motion imitation training and testing, which we will release with the code and models.
B.2 ACTION AND REWARDS
Our action space, state, and rewards follow the specifications of the PHC paper. Specifically, the ac- tion a, specifies the target for the proportional derivative (PD) controller on each of the 69 actuators. The target joint is set to qf a, and the torque applied at each joint is T' = kPo(a, - q:) kd qq Joint torques capped at 500 N-m. For the motion tracking reward, use:
r, 0.5r
g-imitation
+ 0.55%, amp energy
rt
(5)
r e-imitation
= e 1000 p. + uye - 0 + uye -..1/60 + Wye
0.1 -0.1 -0.1
.g-imitation the motion imitation reward, rimp is the discriminator reward, and ri renegy an where o
energy penalty. y g.iimiiaiion measures the difference between the translation, rotation, linear velocity, and angular velocity of the 23 rigid bodies in humanoid. amp, remp is the Adversarial Motion Prior (AMP) reward (Peng al., 2021), provided by discriminator trained on the AMASS dataset. energy is - 00.0005 Eje joints |μي/ر where μı and wj correspond to the joint torque and the joint angular velocity, respectively. The energy penalty Fu et al. (2022) regulates the policy and prevents high-frequency jitter.
The energy penalty "
B.3 MODEL ARCHITECTURE AND ABLATIONS
All primitives and composers in PHC are a 6 layer MLP with units [2048, 1536, 1024, 1024, 512, 512] and SiLU activation. We find that changing the activation from ReLU (Fukushima, 1975; Nair & Hinton, 2010) to SiLU (Hendrycks & Gimpel, 2016) provides a non-trivial boost in tracking performance. Combining with larger networks (from 3 layer MLP 6 layer), use only three
15

---

Published as a conference paper at ICLR 2024
Table 6: Ablations on training PULSE from scratch using RL (no distillation).
AMASS "rainn
AMASS-
Distill
Suce f
Ex-mpire
Empipe
Eaw
Eyd
Suce
Eg 4
Empipe 4
Eac J
E |
x
72.0%
76.7
52.8
3.5
8.0
32.6%
98.4
79.4
9.9
16.2
99.8 %
39.2
35.0
3.1
5.2
97.1% 1%
54.1
43.5
7.0
10.3
primitives to lear fail-state recovery and achieve success rate of 100%. To study the effect of the new activation function and the progressive training procedure, we perform ablation studies on the training of single primitive P (not the full PHC policy) using the proposed changes.
Each primitive is trained for 3 x 10 samples. From Table 5, we can see that comparing Row (RI) and R3, the new progressive training pro- cedure improves the success rate by large amount, showing that P's capacity is not fully utilized if hard ing each P training. When comparing R2 and R3, we can see that changing the activa- tion function from ReLU to SiLU improves the tracking performance and improves Empipe. Ta- b reports the hyperparameters used for training.
Table 5: Ablations PHC+'s primitive P training. Progressive: refers to whether Qued is updated during the primitive training (rather than waiting until conver- gence and initialize new primitive).
AMASS Tes
formed and updated dur-
Activation
Progressive
Suce
Egretre
Exx
Exct
SiL.1 SILU
92.0%5
43.0
29.2
8.9
98.5%
39,0
28.1
6.7
8.5
C DETAILS ABOUT PULSE
C.I TRAINING PROCEDURE
We train TULLSE using the training procedure used to train primitive p()) in PHC+, where progressively form Quari while training the policy. Since TuLLE and ттиC+ share the same state and action space, we query Trhс+ at training time to perform online distillation. We anneal the 10" samples. Afterward,
coefficient 3 of LRI. from 0.01 0.001 starting from 2.5 10 remains the same. We report our hyperparameters for training TPULSE in Table. 4.
C.2 COMPARISON TO TRAINING SCRATCH WITHOUT DISTILLATION
One of our main contributions for PULSE is the using online distillation learn TrLLSE where the latent space uses knowledge distilled from a trained imitator, PHC+. While prior work like MCP (Peng et al., 2019) demonstrated the possibility of training such policy from scratch (using RL without distillation), imitation objective creates instability during training. We hypothesize that random sampling for the variational bottleneck together with random sampling for RL leads to noisy gradients. In Table 6, we report the result of motion imitation from training from scratch. We can see that the training using RL does not converge to good imitation policy after training for more than x 1010 samples.
find that using the variational information bottleneck together with the
C.3 COMPARISON TO OTHER LATENT FORMULATION VAQ, SPHERICAL.)
In our earlier experiments, we studied other forms of latent space such as a spherical latent space, similar to ASE (Peng et al., 2022), or vector quantized latent space, similar NCP (?). For spher- ical embedding, we use the same encoder-decoder structure in PULSE and use 32-dimensional latent space normalized to the unit sphere. For a vector-quantized motion representation, we follow Liu et al. (2021); Van Den Oord et al. (2017) and use 64-dimensional latent space divided into 8 partitions, using a dictionary size of 64. Dividing the latent space into partitions increases the repre- sentation power combinatorially (Liu et al., 2021) and is more effective than a larger dictionary size. Although through distillation, each of these representations could reach a high imitation success rate and MPJPE (spherical: 100% Suce and 28.1 mpjpe VQ: 99.8 % Succ and 36.5 mpipp))), both lose
16

---

Published as a conference paper at ICLR 2024
the ability to serve generative model: random samples from the latent space do not generate coherent motion. In NPC, an additional prior needs to be learned. The quantized latent space also introduces artifacts, such as high-frequency jitter, since the network is switching between discrete codes. We visualize this artifact in our supp1 ement. site's last section.
C.4 DOWNSTREAM TASKS
Each generative downstream task policy Tiik is a three-layer MLP with units [2048, 1024, 512]. For VR controller tracking, we use a six-layer MI.P of units [2048, 1536, 1024, 1024, 512, 512]. The value function has the same architecture as the policy. All tasks are optimized using PPO. For simpler tasks (speed, reach, strike), we train for 2 10 samples. For the complex terrain traversal task, the policy converges after ~ 1 x 10 samples. The strike, speed, and reach tasks follow the definition in ASE (Peng et al., 2022), while the following trajectory task follows PACER Rempe et al. (2023). VR controller tracking task follows QuestSim Winkler et al. (2022).
Speed. For training the x-direction speed task, the random speed target is sampled between 0 m/s 5m/s (the maximum target speed for running in AMASS is around 5m/s). The goal state is defined of 3.-epeed A (dz, vt)) where de is the target direction and i is the linear velocity the policy should achieve at timestep t. The reward defined as rspeed abs(Ue v²") where v."" 0 the humanoid's root velocity.
as
Strike. For strike, since we do not have sword, substitute it with "strike with hands". The objective is to knock over the target object and is terminated if any body part other than the right hand makes contact with the target. The goal state sf g.-strike & #))) contains the position and orientation 2, as well as the linear and angular velocity æ;)) of the target object in the agent frame. The reward is Tstrike 1 - up . u, where upP is the global up vector and u. is the target's 's up vector.
Reach. For the reach task, a 3D point o is sampled from a 2-meter box centered at (0, 0. 1), and the goal state is of gerech & (ct)). The reward for reaching is the difference between the humanoid's right hand and the desired point position Treach exp(-5 [pp right hand -c 112).
Trajectory Following on Complex Terrains- The humanoid trajectory following on complex terrain task, used in PACER (Rempe et al., 2023), involves controlling humanoid to follow random trajectories through stairs, slopes, uneven surfaces (Rudin et al., 2021), and to avoid obstacles. We follow the setup in P and train policy Tiak (2, s .f. gf-terain)) s.. g-eerran e (o:: Ti:T) where o, represents the height map of the humanoid's surrounding, and 1 is the next 10 time-step's 2D trajectory to follow. The reward is computed as /.lerrain exp(-2) (0) Tz ll) 0.0005 EjE joints ||, 2
where the first term is the trajectory following the reward and the second term an energy penalty. Random trajectories are generated procedurally, with velocity between [0. 3|m/s and accelcration between [0, 2] m/s?. The height map o, is rasterized local height map of size o E 322332x3 which captures a 2m x 2m square centered at the humanoid. We do not consider any shape variation or human-to- tosaann-t--umaaa interaction as in PACER. Different from PACER, which relies on an additional adversarial reward to achieve realistic and human-like behavior, our framework policy does not rely on any additional reward but can still solve this challenging task with human-like movements. We hypothesize that this is result of sampling from the pre-le prior, where human-like motor skills are easier to sample than unnatural ones.
VR Controller Tracking. Tracking VR controllers is the task of inferring full-body human mo- tion from the three 6DOF poses provided by the VR controllers (headset and two hand controllers). Following QuestSim (Winkler et al., 2022), we train this tracking policy using synthetic data. Essen- tially, we treat the humanoid's 's head and hand positions as a proxy for headset and controller posi- tions. One can view the VR controller tracking task as an imitation task, but with only three joints to track, with the goal state being: s A (Ō e 0," "r, P - P, ve 6ي - Ur, Dh'' w 0 ((t-) where the superscript ve refers to selecting only the head and two hands joints. During training, we use the same (full- body) imitation reward train the policy. We use the same progressive training procedure for training the tracking policy.
17


Reviews:
[{"review_id": "HLH7V96PMP", "reviewer": "Reviewer_a8Fw", "summary": "This paper presents a system to train human motion controllers that can imitate large motion datasets and be used efficiently for training tasks such as terrain navigation and vr motion following.\n\nKey contributions:\n\n1. Modification of PHC to learn a better motion controller.\n\n2. A VAE like distillation process to obtain a controller and an action space that can be used efficiently for downstream tasks.", "questions": "1. It will be nice to showcase some scenarios where the benefit of learning the whole AMASS dataset is useful.\n\n2.  What are the motions that PHC cannot handle?\n\n3. I feel like the motion quality produced in the downstream tasks is suboptimal/unnatural. It will be nice to have a metric to measure the motion quality generated in the downstream task for potential future improvement for future work.\n\n4. For the speed task (and other tasks as well, but speed task is one of the results in the ASE paper, so I will focus on this), looks like the motion quality of ASE is really bad, while the original ASE paper has pretty good motion quality (at least visually), any comment on the discrepancy?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A controller that can imitate the whole AMASS dataset.\n\n2. A wide range of tasks to verify the efficiency of the system.", "weaknesses": "1. No comparison with baselines that resemble the proposed method, e.g., Physics-based character controllers using\nconditional VAEs, by Won et al or ControlVAE: Model-based learning of generative controllers for physics-based characters. by Yao et al, which also uses VAEs.\n\n2. The main difference between the proposed system and other systems is that the proposed system is able to scale to the whole AMASS dataset while other systems are mainly doing locomotion or something similar. However, this is not well demonstrated in the downstream tasks, which are mostly just comprised of locomotion tasks in addition to some simple reaching, which the other systems can already do pretty well (maybe less efficiently?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ew2rNZc4UN", "reviewer": "Reviewer_UBzx", "summary": "In the context of human motion representation, the paper proposes a method to create a fundamental representation of humanoid motion that can be used for humanoid control, human motion generation or motion tracking. This representation is created via two main elements: an imitation method and a physics-based learned prior.\nThe distillation is made through a VAE-like architecture that learns the prior R and decoder D that are then used for each downstream task to generate to generate the action.", "questions": "- In section 4.3, a failure case is described using the prior R but it is unclear to me how do we recover from it apart from restarting the task's policy learning? Does it happen often?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written\n- The presented goal of learning a universal human motion latent representation is very interesting, and the effectiveness on relevant downstream tasks is well presented\n- The curation of the mocap training dataset for PHC (along with other modifications) increases robustness and allows fail states recovery\n- The downstream tasks are relevant and show interesting use cases for the learned representation\n- The ablation study is quite compelling", "weaknesses": "- Quantitative results on the VR-controller tracking task are a little bit disappointing compared to Scratch\n- The references section could be cleaned up and harmonised, notably for publication conferences\n- Although the writing is clear, some typos remain (e.g. Guassian)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgMcXcm9Ib", "reviewer": "Reviewer_arMx", "summary": "This work provides a method to learn a universal humanoid motion prior for different downstream tasks, the prior is designed to cover a wide range of motions. The learned latent space can be used for long, stable and diverse motion generation, and also for solving tasks with natural motions. The latent is learnt by first training an imitator controller to imitate a wide range of human motions, and then learn the latent space by distilling the learned motion imitator. Results show the learned controller could generate a wide range of human motion, and outperform baselines on downstream tasks by a large margin.", "questions": "1. Is the proposed method robust against methodology and dynamics changes? It would be interesting to see these results and potentially enable appling the proposed method to humanoid robots.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is well-motivated, addressing the problem that motion imitator can hardly be used for downstream tasks (especially when interaction with the environment is required) and learned motion prior can not cover a wide range of human motions.\n2. Method is well demonstrated with details, and most designs are well-explained. Clear ablation study also shows the effectiveness of different components. Additional implementation details and hyper-parameters are provided in the supplementary materials for the community to reproduce the results. \n3. The learned humanoid motion prior showed impressive motion imitating performance over a wide range of human motion, and also showed promising results when applying to diverse downstream tasks, including motion tracking and locomotion over complex terrains, et al.\n4. The supplement videos make the difference between proposed method and baseline more comprehensive.", "weaknesses": "1. One of this work’s claims is that previous work has limited coverage of the learned latent space that can not cover the wide spectrum of possible human motion. But in Table 1, there is no comparison with ASE or CALM or Imitate&Repurpose on motion imitation performance. Though ASE, or CALM might be a bit hard to compare, Imitate&Repurpose should be reasonable to compare with. I’m not expecting it to perform better, just for completeness.\n2. Though PHC+ exhibit great performance on motion imitation, it’s comparison with PHC might be a bit unfair, since one of the modification fo PHC+ is “removing” some hard-negative in the dataset, provide motion imitation result on modified dataset might be beneficial. \n3. Some content is a bit hard to read: In Figure 3, it’s quite hard to see the human motion in the Figure 3(e) row, and the title for each row is really hard to see. (Minor issue)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)