Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior
Authors: Ruihan Yang
Zhuoqun Chen
Jianhan Ma
Chongyi Zheng
Yiyu Chen
Quan Nguyen
Xiaolong Wang
Venue: CoRL 2024 (2024)
Keywords: Legged Robots
Imitation Learning
Agile Locomotion
TL;DR: Our system learns a multiple agile locomotion skills with a single instructable motion prior from a diverse reference motion dataset.

Abstract: The agility of animals, particularly in complex activities such as running, turning, jumping, and backflipping, stands as an exemplar for robotic system design. Transferring this suite of behaviors to legged robotic systems introduces essential inquiries: How can a robot be trained to learn multiple locomotion behaviors simultaneously? How can the robot execute these tasks with a smooth transition? How to integrate these skills for wide-range applications? This paper introduces the Versatile Instructable Motion prior (VIM) – a Reinforcement Learning framework designed to incorporate a range of agile locomotion tasks suitable for advanced robotic applications. Our framework enables legged robots to learn diverse agile low-level skills by imitating animal motions and manually designed motions. Our Functionality reward guides the robot's ability to adopt varied skills, and our Stylization reward ensures that robot motions align with reference motions. Our evaluations of the VIM framework span both simulation environments and real-world deployment. To the best of our knowledge, this is the first work that allows a robot to concurrently learn diverse agile locomotion skills using a single learning-based controller in the real world.

Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior
Ruihan Yang", Zhuoqun Chen Jianhan Ma Chongyi Zheng", Yiyu Chen Quan Nguyen 3 Xiaolong Wang'
UC San Diego "CMU USC
Real-Robot Trajectory. Our robot demonstrates diverse agile locomotion skills, including running. jumping, and back-flipping in real using a single motion prior and without fine-tuning.
Figure 1:
The agility animals, particularly in complex activities such running, urning, jumping, jumpngg and backflipping, stands as exemplar for robotic system design. Transferring this suite of behaviors to legged robotic systems introduces essential inquiries: How can robot learn multiple locomotion behaviors simultaneously? How can the robot execute these tasks with a smooth transition? How to integrate these skills for wide-range applications? This paper introduces the Versatile In- structable Motion prior (VIM) Reinforcement Learning framework designed incorporate range of agile locomotion tasks suitable for advanced robotic applications. Our framework enables legged robots to learn diverse agile low-level skills by imitating animal motions and manually designed motions. Our Function- ality reward guides the robot's ability to adopt varied skills, and our Stylization reward ensures that robot motions align with reference motions. Our evaluations of the VIM framework span both simulation and the real world, Our framework allows a robot to concurrently learn diverse agile locomotion skills using a single leasning-based controller in the real world. Videos can be found on our website: https httgg//eeeaaaaaannaie
Abstract:
imi tator. github. io
Keywords: Legged Robots, Imitation Learning, Agile Locomotion
Introduction
Researchers have been studying for years equipping legged robots with agility comparable to that of natural quadrupeds. Picture golden retriever gracefully maneuvering in park: darting, leaping over obstacles, and pursuing a thrown ball. These tasks, effortlessly performed by many animals, remain challenging for contemporary legged robots. To accomplish such tasks, robots need not only master individual agile locomotion skills like running and jumping, but also the capacity to adaptively select and configure these skills based on sensory inputs. The inherent ability of quadrupeds to smoothly execute diverse locomotion skills across varied tasks inspires our pursuit of a control system with
Equua Contributions
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Diverse Reference Motions
Versatile Instructable
Imitation Learning
Single
Pollcy
Figure 2: Our system learns single instructable motion prior, from diverse reference motion dataset.
Table 1: Comparison of Skill Learning Framework. (More Details in Appendix D)
Tracking Tracking
Style
Agility
Control Skills to Learn
Skalls
Sources
Information
Privileged
Deployment
Peng et al. [4]
AMP [22]
X
WASABI [23]
[18]
Motion Imitation [17]
VIM
a general locomotion motion prior that includes these skills. We introduce a novel RL framework, Versatile Instructable Motion prior (VIM) aiming to endow legged robots with spectrum of reusable agile locomotion skills by integrating existing agile locomotion knowledge.
Agile gaits[1, 2, 3] for legged robots have been sculpted using model-based or optimization methods at the price of demanding significant engineering input and precise state estimation. Imitation-based controllers are also proposed to learn from motion sequences from animals [4] or optimization methods [5]. Recent works [6, 7, 8, 9, 10, 11, 12, 13, 14] also incoporate perception for legged robots. Despite encouraging results, most of these works focus on building single controller from scratch, though much of the learned locomotion skills could be shared across tasks. Recent works build reconfigurable low-level motion priors [15, 16, 17, 18, 19, 20] for downstream applications, but fail to make the best use of existing skills to learn diverse locomotion skills with high agility.
In this work, we focus on building low-level motion prior to utilize existing locomotion skills in nature and previous optimization methods, and learn multiple highly agile locomotion skills simultaneously, as Figure 2. We utilize motion sequences to offer a consistent representation of diverse agile locomotion skills. Our motion prior extracts and assimilates range of locomotion skills from reference motions, effectively mirroring their dynamics. These references comprise motion capture (mocap) sequences from quadrupeds, synthetic motion sequences complementing mocap data, and optimized motion traeectories. We translate varied reference motion clips into a unified latent command space, guiding the motion prior to recreate locomotion skills based on these latent commands and the robot's state. For legged robots, locomotion skill the ability to produce specific trajectory. We classify this into two aspects: Functionality and Style. Functionality involves fundamental movement objectives, like moving forward at set speed, while Style focuses how robot accomplishes task, for example, two robots could run at the same speed, but with different gait. Teaching both aspects simultaneously is challenging [21]. We use three feedback: objective performance metrics, qualitative assessments, and detailed kinematic guidance. This structured approach helps the robot master basic functional objectives before refining its locomotion gaits.
By incorporating diverse reference motions and our reward design, our VIM learns diverse agile locomotion skills and makes them available for intricate downstream tasks. We evaluate method in the simulation and real world, as Figure 1. Our method significantly outperforms baselines in terms of final performance and sample efficiency.
Related Work
Classical legged locomotion controllers [24, 25, 26, 27, 1, 28] based on model-based methods [29, 30, 31, 32, 33, 34, 35] and trajectory optimization [36, 3] have shown promising results in diverse tasks with high levels of agility. Nonetheless, these methods normally come with considerable engineering work for the specific task, high computation requirements during deployment, or fragility to complex dynamics. Learning- based methods [6, 37, 13, 38, 39, 40, 41]
Blind Legged Locomotion:

---

Jon Eiyle fawnnd
Triitto Tae
Instuctab Motion Prior
Evot
Vise
Initaton
Reward
Figure 3: VIM and Reward: Our reference motion encoder maps reference motions into latent skill space and low-level policy output motor command. Vie is the low-level critic for RL training. Our reward encourages the robot track the root trjectory and the joint motion of the reference motion.
controllers are proposed to offer robust and lightweight controllers for deployment at the cost of offline computation. Peng et al [42] developed a controller producing non-agile life-like gaits by imitating animals. Though previous works offer robust or agile locomotion controllers across complex environments, these works focus on finishing a single task at time without reusing previous experience. Peng al [22] leverage reference motions prior knowledge when directly addressing specific tasks. Li et al [23] obtain an agile locomotion skill from single partial rough demonstration including only robot root trajectory. Smith et al. [43] utilize existing locomotion skills to solve specific downstream tasks. Hoeller et al [44] built multiple individual locomotion skills and utilized them for agile navigation. Vollenweider et al. [45] utilize multiple AMP [22] to develop a controller to solve a fixed task set. In this paper, our motion prior captures diverse agile locomotion skills from reference motions including mocap trajectories and trajectories generated by trajectory optimization. and provides them for intricate future downstream tasks.
Due to the low sample efficiency and considerable effort required for reward engineering of RI., low-level skill pre-training has drawn growing attention. Singh et [15] utilize a flow- model to build an actionable motion prior with motion sequences generated by scripts. More recent works [16, 17, 18, 19, 46, 20, 47] focus on building low-level motion prior for downstream tasks but fail to include diverse highly agile locomotion skills. Luo al [48, 49] developed unified motion prior for simulated humanoid robot. Peng et al [18] develop simulation- based low-level motion prior entirely through unsupervised methods, yet they do not assure the acquisition of agile skills from the reference motion dataset (Additional discussion in Appendix C). In this work, we build motion prior with reference motions consisting of mocap sequences, synthesized motion sequences, and trajectories from optimization methods and learn multiple highly agile locomotion skills with a single controller. Comprehensive comparison provided in Table
Motion Priors:
3 Learn Versatile Instructable Motion Prior (VIM)
Building Versatile Instructable Motion prior (VIM), as shown in Figure 3, involves: constructing a reference motion dataset, and training the motion prior with an imitation- based reward system.
Our dataset includes reference motions for various locomotion skills such cantering, turning, backflips, and jumps. Reference motions are from: (a) Mocap data [50] of quadrupeds; (b) synthesized motions from generative model[50] to enhance diversity; (c) motions from trajectory optimization methods. While mocap and synthesized motions provide extensive data, not all are feasible for the robot. Thus, trajectory-optimize moves like jumps and backflips. The detailed motion list in the Appendix A. To address differences between animals and our robot, we retarget mocap and synthesized sequences as per Peng al. [4]. Each trajectory noted as (sff,... stf"), , where si the reference robot state ith timestep, and T'' the length of the reference motion. We denote the dataset D {(5Øf s N Each frame includes the robot's pose, velocity, foot position, foot height, and joint angle and velocity without motor commands. Privileged information like robot pose and velocity is used only in simulation, and policies do not require it in the real world.
Reference motion dataset:
motions are included for complex
3.1 Motion Prior Structure
Our motion prior consists of a reference motion encoder and a low-level policy. Reference motion encoder maps varying reference motions into condensed latent skill space, and low-level policy utilizes our imitation reward and reproduces the robot motion given a latent command.
3

---

Reference motion encoder: Our reference motion encoder Eee(-) maps segments of reference motion to latent commands a latent skill space that outlines the robot's prospective move- seef seef seef ste
ment. These segments are expressed as şe = {s++1, S choose set s/o ster
St42: ft45: $1410= st+30)- Specifically, we
s/f stef (+2: 8t45 to provide immediate desired future motion, (+110 4430 to provide de- sired motion over longer time-span. We model the latent command as Gaussian distribution (( (((((st))) from which we draw sample at each interval to guide the low-level pol- latent skill space, our training integrates an information bottleneck [51, 52] objective TAR. where the prior follows an auto-regressive model [53]. Specif- ically, given the sampled latent command for the previous time step 2t-1, we minimize the KL divergence between the current latent Gaussian distribution and a Gaussian prior parameterized by 1- LAR(S ( 1)) 3KL. (Ni of) ) I| (( (1- (1)) where a = 0.95 the sealar controlling the effect of correlation, 33 is the coefficient balancing regularization.
icy. To maintain a temporal-consister
Low-level policy training: Our low-level policy Tow takes latent command 2, and robot's pro- prioceptive state and outputs motor commands ag for the robot, where s, is encoded with proprioception encoder Eprop- We train low-level policy and reference motion encoder using PPO [54] in an end-to-end manner. We introduce learnable motion embeddings for the critic (Vow in Figure 3) to distinguish reference motions. Episodes initiate with random frames of the dataset and terminate when the root pose tracking error is too large or the episode length is beyond the maximum length.
3.2 Imitation Reward for Functionality and Style
Given the formulation of our motion prior, the robot learns diverse agile locomotion skills with our imitation reward and reward scheduling mechanics. Our reward offers consistent guidance, ensuring the robot captures both the functionality and style inherent to the reference motion.
Learning Skill Functionality: To mirror the functionality of the reference motion, we translate the root pose discrepancy between agent trajectories and reference motion into a reward. The functionality reward l'mme includes tracking rewards for robot root position "fus and orientation rau Recognizing the distinct importance of vertical movement in agile tasks, the root position tracking is further split into rewards for vertical pos-z and horizontal movements pos-xy
Tfunc ($t, spof)
Junc * fune + wopeyy prpssyy pos-z -
The formulation of our functionality rewards is provided as follows, similar to previous wor oo
ooi (St, stef)
cxp
10 |
(2), Fuue " stef) exp [-20||s !|
x; (
T "fune (st, ster) exp
(-80 || x xp) where q, q and x, are the root orientation and position from the robot and reference motion respectively. Unlike previous work [4], we emphasize root height in our reward, crucial for mastering agile locomotion skills such as backflips and jumps.
Learning Skill Style: Capturing the style of a reference motion, in addition to its functionality, expands the application of the locomotion skills by meeting criteria such energy efficiency, and robot safety. Drawing inspiration from how humans learn [55, 56]- starting by emulating the broader style before focusing intricate joint movements our robot first mimics the broader locomotion style with an adversarial style reward and later refines its technique with a joint angle tracking reward.
Adversarial Stylization Reward:
We train discriminators D,. i = 1..N for all N reference motions separately to distinguish robot transitions from the transition of reference motion [22, 45] and use the output to provide high-level feedback to the agent. Our discriminator is trained with:
argmin Dj
E (aL.5e+1)
E (Di(se, 8441) - 1) of (11.8111)
(Di 8141) +1)
where d.M (sc. Sp4 1) and d (sy, 11) denote the state transition pair distribution of the dataset, and the state transition pair distribution generated by the policy for ith reference motion respectively. For each reference motion, the likelihood from the discriminator then converted to a reward with: pady St+1)
1 * (1 D(st, 1t+1)) Initially, our adversarial stylization reward provides dense reward and enables the robot to learn credible gait, but it can not provide more detailed instructions the training proceeds, which leads to mode collapse and unstable training.

---

Table 2: Evaluation of Motion Prior in Simulation: We compare Horizontal and Vertical Root Position (Root Pos (XY), Root Pos (Height)), Root Orientation (Root Ori), Joint Angle, and End Effector Position (EE Pos) tracking errors and RL objectives of all methods. Our methods outperform all baselines in terms of smaller tracking errors higher episodic returns, and longer episode lengths. GAIL baseline shows smaller root position tracking error since it can't follow the reference motion leading to early termination of the episode.
Tracking, Error I
RL Objectives f Episode Length
Method
Root Pos (XY) (ir")
Root Pos (Heigh)) Root Oci (rad')
Joint Angle (rad") )
EE Pos (m²)
Episode Return
(in")
VIM (wla Scheduling)
1.24:0.05
a.Bla
6.11as.00
0,08.0
0 0.03
14.313a:1.4
166.783::22.222
1.28.1.. 07
0.1.10
0.1
0.0510.04
13.963.11 ass
179.
Motion Imitation
1 1.39 oo
0.0077=0.00
0.11#s.00
0.25
0. Da..ce
9. 9.536=2..0 .43
183.3933 1 614
GAIL WASABI
1.0ds0.s
0.03ess.0
6.13ec.00
0.17sou
0.09aa..o
3.586
54.7281:75.06
0.31 ca
0.08eoo. as
0.13:a0.00
4.14=1.00
0:21#0.07
0.71
22.822
VIM (wlo Func Reward)
1024
1.0
0.11.c..06
0.61_
0.02
10.66.1222
115. 19.117.77
(wlo Style Reward)
1-49a0
0.00a0..
0.12aa.0
5.14
0:20a0..o
6.28266
109.76:t10.67
Joint Angle Tracking Reward: On the contrary, joint angle tracking reward [57, 17] instruction for the robot to mimic the gait of reference motion, while the reward signal is small in scale during the initial training stage when the joint angle is away from joint target. Similar to root pose tracking reward, our joint angle tracking reward has the following formulation:
provides stable
"o ( exp
-5 Σ
q
<<l"')
exp
(20 Σ |0 <fl1')
+ exp
-20
Σ
6
where q/ q/ are the joint angle of robot and reference motion, cf éf are the end-effector positions of robot and reference motion, h/ h/ are the end-effector height of robot and reference motion.
Stylization Reward Scheduling: To learn the style quickly and stably, we propose to use both adversarial stylization reward and joint angle tracking reward with a balanced scheduling mechanism. Considering the discriminator as a "coach", We utilize the mean ad adversarial reward as an indication of how the coach is satisfied with the current performance. When it's not satisfied with the current perfor- mance of the robot, provides detailed instructions for the robot to learn. Specifically, our stylization reward follows: (sty $t =
wadd
style
* *tyl
+ w Wayle * joint style + wtyle (1
Ty $1+1)) * roint
#
With this formulation, our stylization reward provides dense rewards at the beginning of training, enabling the robot to quickly catch the essence of different agile locomotion skills, and provides detailed instruction as the training proceeds, enabling the robot to refine its gait.
3.3 Solving Downstream Tasks with Motion Prior:
For intricate tasks like jumping over gaps, start- ing from scratch challenging due to the need for agile locomotion skills and the intensive en- gineering balance rewards and regularize mo- tion. Using low-level motion prior, robots can immediately use existing skills and focus on high-level strategies. For each distinct down- stream task, we train a high-level policy Thigh (As shown in Figure 4) takes the high-level ob-
Ruber Soaa Faskur
Actime
Syrap
Low
Arcks
TTink
Intoraaionn
Downstream Task
Highioee Later Cormand
Figure 4: Solving High-level Tasks with Our Motion
Prior. Our high-level policy outputs high-level latent command for low-level policy.
servation Onigh, and proprioception of the robot and outputs latent command for low-level motion prior utilize the existing skills: de = Tow (Tnugh(onn...)) Eppp()))
Additional implementation details about observation/action space, reférrence/rropriccepiio encoder, low-Level/high-lleel policy, and value network can be found in the Appendix F.
Experiments
We evaluate our system in simulation and real-world, comparing with prior work for low-level skill learning and various high-level tasks. Our robot demonstrates life-like agility in the real world.
5

---

Figure 5: Real World Backflip Trajectory: Each row represents single trajectory (From top to bottom: Reference Motion, VIM, GAII,, Motion Imitation). Trajectories are shown from left right.
Table 3: Evaluation Motion Prior Real: We collect representative metrics for different skills with
corresponding metrics from reference motion. N/A denotes completely failed skills in real.
Metrics
Unit
VIM
Motion Imitation
GAIL
Reference Motion
Height (Jump While Running)
(m)
0.50.0.003
0.4210.01
0.4110.04
0.53.10.00
Height (Jump Forward)
(m)
0.44#0.01
0.42+6.01
0.27+0.006
0.59=+0.006
Height (Jump Forward (Syn))
(m)
0.52+0.01
N/A
N/A
0.55=20-007
Height (Backflip)
(m
0.62++001
0.4940.01
N/A
D. 0.60=00.006
Distance (Jump While Running)
(m
0.48110.8
0.35.10.02
0.40.0.003
0.56.10.08
Distance (Jump Forward)
(m
0.76+20.5
0.40+0.01
0.10+0.002
0.82:2.008
Distance (Jump Forward (Syn))
(m)
0.4911004
N/A
N/A
0.54:200007
Linear Velocity (Pace)
(m/s)
0.76+0.01
0.97+0.07
0.50=0.02
0.72 05
Linear Velocity (Canter)
(m/s)
1.4910.15
N/A
N/A
3.87
Linear Velocity (Walk)
(m/s)
0.90=20.4
0.96:2000
0.53+ 58
0.97=20-2
Linear Velocity (Trot)
(m/s)
1.33 17
1.05 02
0.93 01
1.16:20.1 12
Angular Velocity (Left Turn)
(rad/s)
1.71=0.04
0.00+6.00
0.91+a.04
1.01
Angular Velocity (Right Tum)
(rad/s)
0.8120002
0.6210.02
0.63
0.41
Joint Angle Tracking Error
(rad? (joint)
0.10
0.27:0.16
0.22
Follswing Coumadd
Jate Forward
Follwwigg Commznn Jama Forward
Jalp Kall Ransing
Tum
Tudiffe
Laversan
Semgless
Sumplees
Suapple
PPO
AMP
HRL (w AMP)
VIM
Figure 6: Latent Skill Space t-SNE. We visualize the latent embedding for varying motion segments.
Figure 7:
High-level Tasks Evaluation in Simulation: Solid
line and shaded area denote the mean and std across random seeds. Our system outperforms all baselines.
4.1 Evaluation of Learned Low-level Motion Priors
We benchmark our method against three representative baselines: Motion Imitation [4, 17, 20] baseline represents a thread of recent works whose imitation rewards are defined solely with errors between current robot states and the corresponding reference states. Generative Adversarial Imitation Learning (GAIL) baseline represents a thread recent work [18], whose imitation reward solely provided by the discriminator trained distinguish trajectories generated by the policy from the ground truth reference motions. WASABI baseline represents modified version of WASABI [23] for our setting. Each method trains for x 10 samples across 3 random seeds. Both method and the Motion Imitation baseline adopt identical reward scales for all motion error-tracking rewards.
Baselines:

---

Table 4: High-level Tasks in Real World: We compare Following Conmand + Jump Forward
policies of methods in real. N/A denotes completely failed skills in real. Our methods outperform all baselines in real for most metrics.
Metries
Unit
Ours
AMP
PPO
HRL
Max Linear Velocity
(m/s)
1. 1.
1.74
1.75
1.70=0.08
Max Angular Velocity (Left)
(rad/s)
1.78440004
1.07=0.00
2. 0.245
0.00
Max Angular Velocity (Right)
(rad/s)
2.03
0.83=0.09
1.75
0.95=0.37 0.37
Jump Distance
(m)
0.50
0.00=0.00
N/A
N/A
Jump Height
(m)
0.50=o.02
0.38=0.01
N/A
N/A
Simulation Evaluation: In the simulation, we measure average imitation tracking errors, episode returns, and trajectory lengths across random seeds. As listed in Table 2, the tracking error of root pose represents the ability of the robot to reproduce the locomotion skill, and the tracking error of joint angle and end effector position represents the ability of the robot to mimic the style of reference motion. Our method achieves similar root pose tracking error as the motion imitation baseline with a much smaller joint angle tracking error. This shows that our method strikes a balance between functionality and style, superior to the motion imitation baseline that focuses mainly on functionality. Meanwhile, the GAIL baseline failed to learn the functionality of the reference motions leading to short episode length and the least episode return. We surmise that the GAIL baseline's inadequacy arises from the adversarial reward does not offer temporally consistent guidance throughout skill learning and the mode collapse issue inherent in adversarial training hinders the robot from mastering highly agile skills, such backfipping. The poor performance of the Motion Imitation baseline may stem from the challenges of balancing different terms and selecting suitable hyperparameters when concurrently learning multiple agile locomotion skills.
We provide the ablation study over the reward term and the scheduling mechanism shown in Table 2. We found that without Functionality reward, the learned controller could not robustly track the reference motion resulting in smaller Episode Return and, shorter Episode Length on the other hand, removing Style reward results in significantly higher Joint Angle and End-Effector tracking error. Comparing VIM with and without stylization reward scheduling, we find the former exhibits enhanced style tracking performance, underscoring the value of stylization reward scheduling in refining robot gait tracking.
Ablation Study of Learned Motion Prior:
We evaluated learned agile locomotion skills in the real world using specific metrics tailored to different skills, as detailed in Table 3. We repeated our experiment three times per skill per method per seed since our real-world experiment. For Jump While Running/ Jump Forward/ Jump Forward (Syn) / /Backflip, we measured jumping height and distance. For Pace Bacce/CCatee//aa////aa velocity. Results show our method retains most of the reference motion functionality. The only significant deviation observed in Canter due differences between animal and robot capabilities, quadrupeds use tendons to achieve higher running speeds, which our robot lacks. Despite similar root pose tracking eπrоrs in simulations, real-world metrics like jumping height, distance, and velocity tracking error, indicating that mirroring reference motion style improves sim2real transfer for natural gaits. The GAIL baseline struggled with real-world locomotion skills. Figure visually compares real-world trajectories, showing our method's superiority in capturing both motion functionality and style. Due to poor simulation performance, the WASABI baseline was not evaluated the real world.
Real World Evaluation:
and Left /urn/Right Turn, we measured linear and angular
method outperforms the Motion Imitation baseline in
We visualize the learned latent skill space in Figure 6 by vi- sualizing the latent embedding corresponding to motion segments in our reference motion dataset via t-SNE [58]. We find that different skills are separated into different regions with clear bound- aries. Our reference motion encoder also clusters the skills with similar semantic meaning together: embeddings from Left Turn /arn/Right Rig Turn sequence are close, which enables the smooth tran- sition between different skills. embeddings from Jump While Running Jump Forward Jump Forward (Syn) sequence clustered together. These observations suggest that our system learned a smooth and semantically meaningful latent skill space for solving high-level tasks.
Latent Skill Space Visualization:
7

---

4.2 Evaluation on High-level Tasks
To evaluate how our method leverages learned agile locomotion skills for high-level tasks, we designed set of tasks and tested our method against baselines in simulation and the real world.
High-level Tasks & Observation: Our tasks include: Following Command: directing the robot to move with specific linear and angular velocities. Linear velocity commands range from 0 ~ 2 m/s, and angular velocity commands range from -2 ~ 2 rads/s. In our motion prior, the robot is trained to move and turn at the reference motion's 's speed. Hence, to follow a command precisely, the high-level policy needs to smoothly interpolate between different speeds. Jump Forward: directing the robot to jump while running. We have adapted a subset of jumping rewards from CAJun [59] to evaluate policy interpolation between jumping and running motions within a fixed timeframe. Foll alllowiing Command + Jump Forwaree rd: directing the robot either jump forward or adjust to changing commanded speeds. To optimize episode return, the robot should not only use the agile locomotion skills from the reference motion dataset but also develop unobserved skills like executing sharp turns. Detailed high-level observations for different tasks are provided in the Appendix F.
Baselines: Given the baseline's subpar performance low-level motion prior training, we compare our system with three representative baselines without pre-trained low-level controllers: PPO [54]: Controllers trained exclusively on high-level task rewards. AMP [22]: Utilizes reference motion for styling reward in adversarial imitation learning and learns high- tasks while mimicking reference motions. Hierarchical Reinforcement Learning (HRL) from Jain et al. [60]: Learns high-level policy sending latent commands to low-level controller, resembling works that decompose tasks into sub-probe pooblloolems [61, 62, 63, 64, 65, 66, 67]. For fair comparison, we removed the trajectory generator in [60], used PPO for AMP and HRI., and used full reference motion for AMP and HRI with AMP.
Evaluation in Simulation & Real World: We trained all methods on each high-level task for 4 x 10 samples with 3 random seeds. Simulation results are detailed in Figure 7, and real-world results are provided in Table 4. Real-world Following Commands trajectory is also provided in Appendix I. For the Following Command task, all methods mastered basic locomotion, but ours excelled in efficiency and smooth transitions between diverse linear and angular velocities. In the Jump Forward and Followl ng Command + Jump Forwar rd tasks, which required advanced jumping abilities, baselines struggled. They either moved forward continuously, remained grounded when prompted to jump, toppled avoid energy consumption penalties. In contrast, our system seamlessly integrated jumping and running actions, achieving the highest episode return. Despite having comprehensive reference motion dataset, baselines couldn't 't harness the skills effectively. This likely stems from the difficulty of deriving agile locomotion skills using only adversarial stylization rewards, similar to the GAIL baseline's 's poor performance in low-level training.
Limitations
Our current system exhibits several limitations: 1) Safety is not assured with our current method. Introducing safety constraints during training could mitigate this issue; 2) The robot's limited capacity restricts its ability to fully replicate certain motion capture data, like cantering- Upgrading the hardware could address this limitation; 3) Currently, our system does not incorporate dynamics information. This could be improved by integrating adaptation techniques during deployment; 4) The system's 's low-level motion priors and high-level policies currently lack perceptual information.
6 Conclusion
In this paper, we propose Versatile Instructable Motion prior (VIM) which learns agile locomotion skills from diverse reference motions with a single motion prior. Our simulation and real-worlde results show that our VIM captures both the functionality and style of locomotion skills from reference motions. Our VIM also provides a temporally consistent and compact latent skill space representing different locomotion skills for high-level tasks. With agile locomotion skills in our VIM, complex High-level tasks can be solved efficiently with minimum human effort.
8

---

Acknowledgement
This work was supported. in part, by NSF CCF-2112665 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CD) NSF ACI-1541349 CC*DNI Pacific Research Platform.
References
[1] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2245 -2522. IEEE, 2018.
[2] C. Nguyen, L. Bao, and Q. Nguyen. Continuous jumping for legged robots stepping stones via trajectory optimization and model predictive control. In 2022 IEEE ó1 Conference on Decision and Control (CDC), pages 93-99. IEEE, 2022.
[3] Q. Nguyen, M. Powell, B. Katz, J. Di Carlo, and Kim. Optimized jumping on the mit cheetah 3 robot. In 2019 International Conference Robotics and Automation (ICRA), pages 7448-7454. IEEE, 2019.
[4] X. B. Peng, E. Coumans, T. Zhang, T.W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 07 2020. doi: 105:10.15607/RSS..000 21T..22200002066
[5] Y. Fuchioka, Z. Xie, and M. van de Panne. Opt-mimic: Imitation of traetorie for dynamic quadruped behaviors, 2022. URL httpss//aaravv..rrr/aa//////////
trajectories 01247.
[6] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. 6th Annual Conference on Robot Learning, 2022.
[7] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Conference Computer Vision and Pattern Recognition 2023, 2023. URL https:// openreview net. forum' d=JYyWCcmWDDS
[8] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning uision-guided quadrupedal locomotion end-to-end with cross-modal transformers. International Conference on Learning Representations, 2022. URL https: http:::///ppeeevieww net/ dar.ummmmddaanooooooooo 6AB.
[9] C. Imai, M. Zhang, Y. Zhang, M. Kierebinski, R. Yang. Y. Qin, and X. Wang. Vision saidnn-guided quadrupedal locomotion in the wild with multi-modal delay randomization. In 2022 IEEE/RSJ
international conference on intelligent robots and systems (IROS), 2021.
[10] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, Coumans, S. Ha. J. Tan, and T. Zhang locomloocomoiion Learning walk complex terrains with vision. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1291-1302. PMLR, 08 11 Nov 2022. URL https: http:://pocceeddeee .ml..ng..........mmm.........m.m wtt.::/s.............mm. / v1 64 yu22a a. html.
[11] Kareer, N. Yokoyama, D. Batra, Ha, and J. Truong. Vinl: Visual navigation and locomotion obstacles. arXiv preprint arXiv:2210.14777 2022.
[12] Z. Zhuang, 7. Fu, Wang, C. G. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In 7th Annual Conference on Robot Learning, 2023, URL https: httpttttt//peeeeeee //pp://eiee.....e.e net/ forum? 1 1d= 5eTE.
[13] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci Robot. 7662):eab22822, Jan. 2022.

---

[14] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Leaninng to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[15] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinchart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning, 2020.
[16] L. Hasenclever, F. Pardo, R. Hadsell, N. Heess, and J. Merel. CoMic: Complementary task learning &amp; mimicry for reusable skills. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4105-4115, PMIR, 13-18 Jul 2020. URL https:/ proceedings.. .wwiinngsmmm.......mmm
httpinnns......psssse...e www.s/resssvve.e. htt.::/p..ese////aaaaeeeeeeeer/a http///awweeeeeeeeeeeeeeeeeeeeee html.
[17] Bohez, S. Tunyasuvunakool, Brakel, Sadeghi, L., Hasenclever, Y. Tassa, E. Parisotto,
J. Humplik, T. Haarnoja, R. Hafner, M. Wulfmeier, M. Neunert, Moran, N. Siegel, A. Huber, F. Romano, N. Batchelor, F. Casarini, J. Merel, R. Hadsell, and N. Heess. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors, 2022. URL
https: aaw.iv.vvvv.. httpp////aarar.oooo//////22....2 http:/ababb/22...133333
[18] X. B. Peng. Y. Guo, L. Halper, S. Levine, and S. Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), July 2022.
[19] J. Juravsky, Y. Guo, S. Fidler, and X. B. Peng. Padl: Language- directed physics-basee bassdd character control. In SIGGRAPH Asia 2022 Conference Papers, SA '22, New York, NY, USA, 2022. Association for Computing Machinery, ISBN 9781450394703.
1tte:0..145//5500933555555 URL. htwps:ttp//.oo / doii wooi..rgg/000 10l 1111/1555/3566.1. 3555391.
[20] L. Han, Q. Zhu, J. Sheng, C. Zhang, T. Li, Y. Zhang, H. Zhang, Y. Liu, C. Zhou, R. Zhao. J. Li, Y. Zhang, R. Wang, W. Chi, X. Li, Y. Zhu, L.. Xiang, X. Teng, and Z. Zhang. Lifelike agility and play on quadrupedal robots using reinforcement learning and generative pre-trained models, 2023.
[21] J. Xu, Tian, Ma, D. Rus, S. Sueda, and W. Matusik. Prediction-guins guided multi-objectivv reinforcement learning for continuous robot control. Proceedings of the 37th International Conference on Machine Learning. 2020.
[22] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based based character control. ACM Trans. Graph., 40(4), July 2021. doi: 145/ 3450626.3459670. URL http: /ooi acm. org 1 /1144/3445555 3450626..6666 34506663335667600 3459670.
[23] C. Li. M. Vlastelica, Blaes, J. Frey, Grimminger, and Martius. Learning agile skills via adversarial imitation of rough partial demonstrations, 2022.
[24] H. Geyer, A. Seyfarth, and R. Blickhan. Positive force feedback in bouncing gaits? Proceedings of the Royal Society of London. Series B: Biological Sciences, 270(1559) 2003.
[25] K. Yin, Loken, and M. Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), 26(3): 2007.
[26] N. Torkos and M. van de Panne. Footprint-based quadruped motion synthesis. Proceedings of the Graphics Interface 1998 Conference, June 18-20, 1998, Vancouver, BC, Canada, pages 151- 160, June 1998. URL httpt:::/ggappccsineeeeeeee gil 9989999999 19.9. pdf.
[27] H. Miura and Shimoyama. Dynamic walk of biped. The International Journal of Robotics Research, 3(2):60-74, 1984.
[28] M. H. Raibert. Hopping in legged systems-modeling and simulation for the two-dimensional one-legged case. IEEE Transactions Systems, Man, and Cybernetics, SMC- (MC-14(3:4511-663 1984.
10

---

[29] J. D. Carlo, P. M. Wensing. B. Katz, G. Bledt, and S. Kim. Dynamic locomotion in the MIT cheetah through convex model-predictive control. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-5, 2018, pages 1- IEEE, 2018. 109.11000//0..0001... IROS. 2018. 8594448.
URL https: ptttps://doi.oom http:///ooooooro./.ooo/o
[30] C. Gchring. S. Coros, M. Hutter, M. Blösch, M. A. Hoepflinger, and R. Siegwart. Control of dynamic gaits for a quadrupedal robot. In 2013 IEEE International Conference Robotics and Automation, Karlsruhe, Germany, May 6-10, 2013, pages 3287-3292. IEEE, 2013, doi: doi:10.1109/ ICRA.2013.6631
URL https::/p///... doi. org/10. 1tt//10.11000 1109/ ICRA. 2013. 66 6631035
[31] Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and Kim. Dynamic locomotion in the mit cheetah through convex model-predictive control. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-9. IEEE, 2018.
[32] Y. Ding, A. Pandala, and H.-W. Park. Real-time model predictive control for versatile dynamic motions in quadrupedal robots. In 2019 International Conference on Robotics and Automation (ICRA), pages 8484-8490. IEEE, 2019.
[33] G. Bledt and S. Kim. Extracting legged locomotion heuristies with regularized predictive control. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 406-412. IEEE, 2020.
[34] R. Grandia, Farshidian, A. Dosovitskiy, R. Ranftl, and M. Hutter. Frequency-aware model predictive control. IEEE Robotics and Automation Letters, 42):1517-1524, 2019.
[35] Y. Sun, W. L. Ubellacker, W.-L. Ma, X. Zhang, C. Wang, N. V. Csomay-Shanklin, M. Tomizuka, K. Sreenath, and A. D. Ames. Online learning of unknown dynamics for model-based controllers in legged locomotion. IEEE Robotics and Automation Letters (RA-L), 2021.
[36] J. Carius, R. Ranftl, V. Koltun, and M. Hutter. Trajectory optimization for legged robots with slipping motions. IEEE Robotics and Automation Letters, 4(3):301333000, 2019. doi: 10.1109/LRA.2099 2923977....
[37] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robot. Robotics: Science and Systems, 2021.
[38] J.-P. Sleiman, F. Farshidian, and M. Hutter. Versatile multicontact planning and control for legged loco-manipulation. Science Robotics, 8(81):aad55014, 2023. doi: 1attt10.112/ccs 12//s//obboissss.. adg5014. URL https: //w:://www////////////////.wt wwwp::sseeenccc.cc.ee...o http://abb......//.......1. wtt.1111///orooii..c.oooo adg5014.
10.
[39] X. Cheng, A. Kumar, and D. Pathak. Legs manipulator: Pushing quadrupedal agility beyond locomotion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[40] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning a unified policy for manipulation and locomotion. Conference on Robot Learning (CoRL), 2022.
[41] Z. Li, X. Peng, Abbeel, S. Levine, G. Berseth, and K. Sreenath. Robust and versatile bipedal jumping control through reinforcement learning, 2023.
[42] P. Fankhauser, M. Bloesch, C. Gehring. M. Hutter, and R. Siegwart. Robot-centric elevation mapping with uncertainty estimates. In International Conference on Climbing and Walking Robots (CLAWAR), 2014.
[43] L. Smith, J. C. Kew, T. Li, L. Luu, X. B. Peng. S. Ha, Tan, and S. Levine. Learning and adapting agile locomotion skills by transferring experience. arXiv preprint arXiv: 0044:4230,.9883, 2023.
11

---

[44] D. Hoeller, N. Rudin, D. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots, 2023. URL https: / /aaaiiiivv...... orgg 2obs/23066 11000.148877 14874.
[45] E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning, 2022.
[46] B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
URL www.t:tt///oott htttss:/oppereeviee net/ /
[47] T. Li, R. Calandra, D. Pathak, Y. Tian, F. Meier, and A. Rai. Planning in learned latent action spaces for generalizable legged locomotion, 2021. URL https: ptctps://arxiv..or/ abs/2008 11867.
[48] Z. Luo, J. Cao, J. Merel, A. Winkler, J. Huang, K. M. Kitani, and W. Xu. Universal hu- manoid motion representations for physics-basee based control. In The Twelfth International Con- ference on Learning Representations, 2024. URL https:://///.. http:///pppeevveeeeee wtt.iiiwwweeeeeeeeeeeeeeeeeeeeee forum? dd=0rod8Px002.
[49] Z. Luo, J. Cao, A. W. Winkler, K. Kitani, and W. Xu. Perpetual humanoid control for real-time simulated avatars. In International Conference on Computer Vision (ICCV), 2023.
[50] H. Zhang, S. Starke, T. Komura, and J. Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG), 37(4):1-11, 2018
[51] N. TISHBY. The information bottleneck method. Computing Research Repository (CoRR), 2000.
[52] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
[53] S. Bohez, S. Tunyasuvunakool, P. Brakel, F. Sadeghi, L. Hasenclever, Y. Tassa, E. Parisotto, J. Humplik, T. Haarnoja, R. Hafner, et al. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXiv:2203.17138, 2022.
[54] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: h 1707.06347, 2017.
[55] M. Fitts and M. I. Posner. Human Performance. Brooks/Cole Publishing Co., Belmont, CA, 1967.
[56] N. A. Bernstein. Dexterity and its Development. Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1996.
[57] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics- based character skills. ACM Trans. Graph., 37(4):143:1- 143:14, July 2018. ISSN 0730-0301, 1oi: 1at:10.114/3197511111 11411//1995117322011111 URL http: : /doi. acm. org 10. . 10111111119511 .2001311.
[58] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[59] Y. Yang, G. Shi, X. Meng, W. Yu, T. Zhang, Tan, and B. Boots. Cajun: Continuous adaptive jumping using learned centroidal controller. arXiv preprint 2xXiv:2306609557 2023.
[60] D. Jain, K. Caluwaerts, and A. Iscen. From pixels to legs: Hierarchical learning of quadruped locomotion. In Conference on Robol Learning, pages 91-102, PMLR, 2021.
[61] R. S. Suttona, D. Precup, and S. Singha. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181-211, 1999.
12

---

[62] T. G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion, Journal of artificial intelligence research, 13:227-303, 2000.
[63] R. Parr and S. Russell. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems, 10, 1997.
[64] O. Nachum, S. S. Gu, H. Lee, and Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[65] J. Gehring, G. Synnaeve, A. Krause, and N. Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems, 14:11553-1,564 2021.
[66] M. Klissarov and D. Precup. Flexible option learning. Advances in Neural Information Processing Systems, 34:4632-4646 2021.
[67] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning, pages 1025-1037. PMLR, 2020.
[68] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
13

---

Appendix
A Reference Motion Dataset
Our reference motions (I1 reference motions in total) come from motion capture of animal motion, trajectory optimization method, and synthesized data with generative model. The length of our reference motion ranges from 32 to 500. During training, we repeat the reference motions cyclically to fit the length of the episode.
Table Reference Motion Dataset:
Name
(Mocap)
Trot (Mocap)
Jump while Running (Mocap)
Right (Mocap)
Tum (Mocap)
(Mocap)
Mation length
500
32
500
Skill Name
Jump Forward (Synthetic)
Left Tum (Synthatic)
Backfip (Optimization)
Jump Forward (Optimization)
Canter (Mocap)
Motion length
500
500
129
120
64
B Performance Across Different Reference Motions
In our framework, when the root pose of the simulated robot diverges much from the reference root pose, terminate the episode, described in Sec 3.1 In this case, the episode length is good indicator of whether the learned policy could follow the reference motions. As shown in Fig. 8, the performance of our framework varies when imitating different reference motions. When imitating relatively steady motions like Walk (Mocap), Pace (Mocap), Left Turn (Mocap), the learned controller could track the motion for longer period. When imitating relatively agile motions, especially with high moving speed, such as Canter (Mocap), Jump While Running (Mocup), the performance of our system drops. This phenomenon is rooted in the methodology disparities between our robot and real animals.
Acerage Epicode Langths when Instaking Difierent Raference Motion
Epttod
333
(Moce)
(Pece:
(Pec4)
(Poc4))
((oci))
J.P0 PUKKnn
(Dolm Ielion
Ossmmattion
Figure 8: Performance for different reference motions: We provide the average episode length of the learned motion prior when it imitates different reference motions.
C
Additional Discussion about ASE
We would like to clarify the significant differences between method and the ASE bascline in the following aspects:
Skill Control and Learning: Unlike ASE, which learns motor skills from transition dataset in an unsupervised manner without controlling the outcome, our method intentionally learns specific motor skills from our reference motion dataset. This controlled learning approach ensures that critical skills, such as the jump motion are effectively acquired. This capability crucial for constructing a motion prior tailored for varied high-level tasks. As shown ASE video and demonstration even though there jump motion in their dataset. ASE failed to learn it.
Long-Tere Skill Acquisition: ASE focuses learning motor skills the transition level, limiting its ability to lear complex, long-term motor skills like backflipping, which require extended motion sequences. Our method, however, leverages combination of adversarial
14

---

styling and dense tracking rewards, providing structured supervision for acquiring long-tenm term motor skills, including agile locomotion abilities like backflipping and jumping forward. This sequence-level modeling is essential for the effective learning of complex locomotion skills.
Performance Evaluation: Since ASE learns different motor skills in an unsupervised manner, it's difficult to evaluate the performance of the learned low-level controller (There is no evaluation or benchmark for the low-level controller in ASE paper). While our method learns to imitate different locomotion skills in the dataset at the sequence level, we could directly benchmark the tracking eπror to evaluate the quality of the learned low-level controller. Benchmark over the low-level controller also important if we want to build backbone low-level skills for diverse potential high-level tasks.
D Additional Discussion over Skill Learning Frameworks
In this section, we provide further discussion on the existing skill-leaming framework:
Function Tracking:
The resulting controller of skill-leaming framework can accurately track the movement of the robot's 's base.
Skill Tracking: The resulting controller can faithfully replicate the joint movement patterns of the robot.
Agility:
The controller is capable producing highly agile locomotion skills. Since there is no universally accepted definition of "agile", in our work, we consider a skill "agile" if involves the robot leaving the ground, such as in backflip or jump, or running/turning at high speed.
Given fixed set of reference motions, the resulting controller can reliably reproduce specific skills. Unsupervised methods like ASE do not guarantee performance on any particular skill in the dataset.
Control Skills to Learn:
Multiple Skills: The resulting controller is capable of performing a variety of different
skills.
Diverse Sources:
The resulting controller learns different skills from various sources.
Reusable: The resulting controller can be reused for tasks beyond reproducing the reference motion's skills.
No Privileged Information: (such as the robot's velocity or position in the world frame) during deployment.
The resulting controller does not require privileged information
Real Deployment: The proposed framework is validated in real-world scenarios.
Additional discussion on the performance of ASE/AMP-based methods in agile locomotion skills: ASE struggles to capture agile locomotion skills for the following reasons:
Difficulty in Learning Agile Skills: Agile locomotion skills, such as jumping and back- flipping, are inherently more challenging to learn compared to other locomotion skills like walking or trotting. Due the well-known mode-collapse issue in the generative adver- sarial learning paradigm, it is particularly difficult for generative adversarial methods (like ASF/AMP) to discover and learn these complex skills in an unsupervised manner.
Limitations of Transition-Leeee Learning: As discussed in Appendix C, ASE/AMP performs adversarial learning at the transition level, focusing on the current and previous states of the robot (as shown in Formula (3) in ASE[18]). However, skills that require a longer sequence of actions, such as backflipping or jumping, are difficult to learn with this approach. For example, executing backflip involves multiple stages: sitting down, lifting the front legs, pushing off with the rear legs, adjusting the pose in mid-air, and landing, Similarly, jumping and rumning require coordinated stages of movement. Transition-level supervision lacks the long-term guidance needed to learn these complex, agile skills.
15

---

Table 6: Evaluation of Motion Prior in Simulation for single Reference motion: We compare Horizontal Vertical Root Position (Root Pos (XY), Root Pos (Height)), Root Orientation (Root Ori), Joint Angle, and End Effector Position (EE Pos) tracking errors and RL objectives of all methods.
Tracking Error
RL Objectives
Method
Root Pos (XY)
Root Pos (Height)
Root Ori (rad")
Joint Angle (rad")
FR Pos (mr²)
Episode Return
Episode Length
(mr²)
(nn²)
Jump Forward (Optimization)
VIM
1.1640.82
0.01+0011
0.10ss
0.0Me
38.
422.
Motion Imitation
1. 1.190044
0.00g0:00
0.13
1.21
0.12 12
26. 11
325. 08 100
GAIL (Single Skill AMP)
2,00
0.Menoo
0.10es
0.92
0.03eam
11.71a9.0
161 38e8s.0
Jump
While Running (Mocap)
VIM Metion Imitation GAIL (Single Skill AMP)
1. 18580.0
=.01
0.09eavss
1.63e0.18
0.06e001
13.36ie7.1
172 90
1000000
0.00ga.co
0.09
3.04.0.00
10.93
163. 3G
2. 2919.0..84
0.0Menox
0.17
2. 28
0.11sau02
4.61=2.54
120. 18
(Mocap)
VIM Motion Imitation GAIL (Singe Skill AMP)
1:21+0.82
0.00eenco
0.088e004
0.18+0.08
0.01=3.00
21. 52212..
213. 10
0.00g0.co
0.1000
0.17
0.01+2.00
17.62a8.
174 884888577
1.76as077
0.00eeaco
D.08
0.61
0.03eaos
14.87
159. 49e00.9
Left Tum (Mocap)
VIM
0.07=00.0 0s
0.00ee0co
0.16
0.15aa000
0.01
31 3d:6.a
299, 13
Motion Imitation GAIL (Single Skill AMP)
0. 0.11
0.00g0.00
0. 14
0.00g0.41
0:03
36. 314
383. (Ge 1e6.ta6...
0.15ee0.1
0.00sa.00
0.17s00s
0.28a0.10
0.01s000
75.10.20
208.96s112.88
Limited Input for Discriminator: ASE/AMP discriminators only consider joint angles as input, making more challenging for the robot to learn agile locomotion skills that involve significant changes in the robot's position and orientation.
E Single Skill Comparison in Simulation
We conducted additional evaluations focusing on single skill learning, where all methods are required to learn single reference motion using identical hyperparameters. For these experiments, removed the motion embedding from the critic since there is only one skill to learn. We selected Jump Forward ( (Optimization), and Jump while Running (Mocap) as representative skills for agile locomotion, and Trot (Mocap) and Left Turn (Mocup) as representative skills for normal locomotion. Each method was trained with 2 x 10th samples per skill across three random seeds.
In general, as shown in Table single-skill tracking tends deliver better results, in terms of longer episode length and higher episode return (representing the overall performance). for the specified skill because the task is easier to learn and more samples are dedicated to that particular skill. (In our low-level motion prior training stage, all skills share the total number of samples.) For agile locomotion skills, our method outperforms both GAIL (Single Skill AMP) in terms of better tracking of the robot's root movement and the motion imitation baseline by achieving smaller joint tracking errors. For normal locomotion skills, all methods deliver reasonable results for both Trot (Mocap) and Left Turn (Mocap). However, GAIL (Single Skill AMP) exhibits slightly higher joint tracking error for Trot, which we attribute to the lack of temporal alignment in the adversarial reward. Although the GAIL baseline can faithfully reproduce the skill, it shows a slightly higher tracking error. It's important to note that the results in Table 6 should not be directly compared with those in Table 2 as longer episodes tend to accumulate more errors, leading to larger tracking errors, and the experiment setting is not identical.
F Implementation Details
Our low-level observation includes joint angles, joint velocities, gravity vector in the robot frame, and the previously executed actions. Our controller outputs target joint angles for 12 joints of our robot in 25hz. The target joint angles are converted to torque command with PD controller where KP=40, KD=1.0.
Observation & Action Space:
High-level Observation For Following Command task, high-level observation includes the target linear velocity and target angular velocity. For Jump Forward task, our high-level
16

---

observation includes the target jumping forward velocity and the normalized phase information in the jumping forward cycle. For Fol lowing Command + Jump Forward rd task, our high-level observation includes the high-level observation for both Following Command and Jump Forward tasks as well as an additional binary command indicating whether following command or jump forward current time-step
Reference Encoder Eres & Proprioception Encoder Eprop: Our reference encoder proprioception encoder are both two-layer MLP with [256] hidden units, mapping the reference motion segment into a 64 dimensional latent distribution and proprioception into a 64-dim rohot state feature, respectively.
Low-Ievel Policy Now and Value Network Viow: Our low-level policy is three-layer MLP with [256, 128] hidden units, mapping the robot state feature and latent command to 12-dim robot target joint angles. Our low-level value network shares the same structure while taking motion embedding as additional input, and output 1-dim value for RL training. Our learnable motion embedding is 64-dim vector for each reference motion.
High-level Policy Thigh and Value Network Vhigh: Our high-level is formulated as a three-layer MLP with [256, 128] hidden units, mapping proprioception information and high-level task informa- tion high-level latent command for low-level motion prior. Our high-level value network shares the same structure. High-level task information depends on specific task. Additional implementation details are provided in the supplementary materials
Reward Coefficients: In our experiment, we use w woi func
pos--u wy 0.1875, w W func
w
Pos-= func
1.5,
wadv 1, Joint 0.5.
style
Other Rewards: To smooth the robot trajectory, we also include energy penalty Tenergy, and action
smooth reward Fcction V'eneryy = le - 3 * >; IT, x úsl where 7; is the the motor torques applied to the ith joint, and the d is the joint velocity for the ith joint. Taction = le 2 Σ. la af f-1 where at and at 1 are the action from policy for ith joint at current timestep and the previous timestep.
Simulation Setup:
We utilize IsaacGym[68] to simulate 4096 robots in parallel and our simulation runs in 200 Hz. During motion prior training, for each robot, we uniformly sample reference motion from the dataset for it to track.
G High Level Task Reward
Our high level jumpping reward is adapted from CAJun [59] with the following terms.
'jump= = 2 (2 Probot 211)/2 + 5 (Base Height - 0.6) max(
Σ
Cy 4,
fEfect
Σ
|| c - 2 +1*
Σ Sefoet
||ey Łyll min(h) 0.16))0.1666
fEfeet
Here our desired foot contact ēy at each step is binary value generated by the task generator as in CAJun [59] with value 0 for no contact, and value for contact, similar for the actual contact ey, and hf is the foot height over the ground.
Our high level command following reward is defined as follows.
Ffollowing cmd
1.5 exp
[[Vommaand
Vrobal 11 1/0255 1.5 * exp (| Wrobox 1| 10.25
Base Height Target Height||
Here the Gcommand is the commanded target linear forward velocity, Urobat is the current forward velocity of the robot. The Woommand is the commanded target angular velocity, Whober is the current angular velocity of the robot.
We also include energy penalty and action smooth reward as shown in the other rewards in Appendix F.
17

---

H
Additional Low-level Skill Comparison
In addition to the low-level skill comparison in Figure 5, we provide another low-level skill compari- son in Figure Our controller learned to jump forward in the air with natural gait, while baselines failed to leave the ground failed to move forward
Figure 9: Real World Jump Forward Trajectory Comparison: Each represents single trajectory (From top to bottom: Reference Motion, VIM, GAIL, Motion Imitation). Trajectorics are shown from right to left.
I High-level Policy Visualization
To better understand of the performance of our high-level policy, provide Following Command trajectory in Figure 10. Though our low-level controller only learns turn with specific angular velocity, our high-level could track different angular velocity in the real world.
Figure 10: Real World high-level Following Commands trajectory: Our high-level Following Command policy can track wide-range linear and angular velocity commands even for velocities absent in the reference motion dataset, indicating high-level policy can manipulate the motion prior for High-level tasks. The trajectory shown from left to right, from top to down.
Detailed Observation Space
We provide more detailed observation space for motion prior. Our Unitree Al robot has 12 joints, corresponding to 12 Degrees of Freedom (DoF), and positional control for the 12 DOF (KP = 40 and KD 1.0). Specifically, the proprioceptive state of the robot contains:
18

---

Joint Angle IR 12x3 contains joint rotations for all joints (12D) for the past three control step.
Joint Velocity 1233 contains joint velocities for all joints (12D) for the past three control step.
Previous Action R 12x3 contains positional command for all joints (12D) for the past three control
step.
Projected Gravity 3x3 contains the projected gravity in the robot frame, representing the orientation the robot for the past three control steps.
Foot position 33x4x3 contains the robot foot positions in the robot frame, 3 dim per foot per timestep for the past three control steps
Note that, our discriminators used for adversarial reward only use the joint angle transition for training and reward calculation.
We also provide additional high-level observation for the example high-level tasks used. For Following Command task, provide target linear velocity and target angular velocity high-level observation. For Jumping Forward task, since the robot is tasked to jump forward in a fixed frequency, we provide normalized temporal phase as high-level observation.
K Domain Randomization
Here we provide our hyperparameters related to domain randomization for better sim2real transfer and shared by all methods.
Parameter
Range
Added Mass for the base
[-1,
Friction
[0.1, 1.3]
Restitution Range
[0, 1.0]
COM shift
[-0.05, 0.05]
Motor Strength ratio
10.7,
KP randomize ratio
[0.8, 1.5]
Kd randomize ratio
[0.5, 1.5]
Proprioception noise
[0, 0.01]
Action noise
[0, 0.05]
L RL Training Details
Here we provide hyperparameters related to RL training and shared by all methods.
Hyperparameter
Value
Max Episode Length
500
Non-linearity
ELU
Policy initialization
Standard Gaussian
# of samples per iteration
4096 48
Discount factor
.99
Parallel Environment
4096
Optimization epochs
# of batches
16
Clip parameter
0.1
Policy network learning rate
3e-4
Value network learning rate
3e-4
Discriminator learning rate
1e-5
Entropy
0.001
Optimizer
Adam
3 for latent regularization
le-5
19

---

M Additional Results Regarding Stylization Tracking
Though adversarial training is generally unstable, we found it relatively stable during our training. We provide the training log our discriminator and the average adversarial reward across epochs in Figure 11. Specifically, We applied the following techniques to stabilize the adversarial training.
We clipped the gradient of the discriminator to have the maximum norm of 0.5
We applied gradient penalties during the training of the discriminator, following AMP [22]
We think our motion imitation reward also helped stabilize the adversarial training since our joint tracking and end-effector position tracking reward provide fine-grained instruction for policy training. We didn't 't observe specific latent skill space collapse during our training, we think this is for the following reasons:
We provide motion embedding for the value function to distinguish different reference motions during training, as shown in Figure 3 in the manuscript. With motion embedding, the value function in our method could learn to distinguish different skills easily.
Our tracking reward terms provided dense instruction for the controller to generate different behaviors, which further regularized the latent skill space during training.
Discriminetor Loss Curve
Average Adversarial Reward Curve
Discriminator Loss
Adversaria Rewarrd
380
Epoch
Epoch
Figure 11: Adversarial Training Log
To study the training behavior in more detail, we visualized the episode return for the joint angle tracking term with the average adversarial reward in Figure 12. We found that in the first 1000 epoch, the episode return of joint tracking increased swiftly corresponding to the rapidly decreasing period of average adversarial reward. After 1000 epochs, the increasing rate of the episode return of joint tracking drops. We think this phenomenon corresponds to our claim in the manuscript that the model transits from learning the overall motion, where the episode return of joint tracking boost, towards learning the fine-grained behavior using the joint angle tracking reward, where the episode return of joint tracking grows slowly.
Episode Return (Joint Angle Tracking] Curve
Average Adversarial Rewaid Curve
Episode naturn Ubint Angle Tracking)
Adversanaal Rewand
Epoch
Epoch
Figure 12: Average Adversarial Reward with Episode Return (Joint Angle Tracking)
20


Reviews:
[{"review_id": "BWi9bKjda5", "reviewer": "Reviewer_wE5L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a technique to train single policy for a collection of skills on a legged robot. The approach consists of two types of reward: 1. functionality and 2) style reward. The functionality reward tracks the root orientation and position(z). The style reward has two components that 1) tracks the joint angle and 2) captures the style using N discriminators (1 for each skills) to capture the distribution of state transitions (similar to AMP). The authors claim that their method works better than vanilla PPO, AMP and HRL and provide experiments in sim and real to justify their claims", "summary_of_recommendation": "The paper seem to advocate for a mix of AMP like style reward and tracking rewards to train skills. But the current experiments and arguments presented in the paper do not make a convincing case.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8iSm9JnhDZ", "reviewer": "Reviewer_M6Lk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a system that can train a robot to imitate multiple reference motions as well as performing transitions between them.", "summary_of_recommendation": "I think this is a good attempt at generating diverse agile skills for quadruped robot. Although the potential of the system is fully explored, e.g, solving more intersting high level tasks, get nicer looking low level skills.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. \"To the best of our knowledge, this is the first work that allows a robot to concurrently learn diverse agile locomotion skills using a single learning-based controller in the real world.\". I think \"Lifelike Agility and Play on Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models\" also learns diverse locomotion skills with a single policy, and has been on arxiv since August 2023.\n\n2. Some concerns I listed in the review sections.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pswxYIHj94", "reviewer": "Reviewer_wyry", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work addresses the challenge of learning a single reusable quadrupedal robot motion prior by imitating diverse reference motion trajectories. The proposed Versatile Instructable Motion (VIM) framework achieves this by learning to project reference trajectories into a latent space. By varying the latent vector, diverse motions can be generated through a learned low-level policy. The framework is trained using both a functionality reward and a style reward. Validation of the method was conducted in both simulation and hardware. The results demonstrate that the proposed method outperforms various baseline methods and successfully generates diverse motions on an A1 robot.", "summary_of_recommendation": "I am towards a weak acceptance due to the overall quality of the hardware valuations as well as the reliable of the proposed method.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1.Weakness No. 3 is my main concern. The authors should focus on justifying the benefit of learning a single policy versus multiple sub-policies.\n2. I am also interested in how the proposed method compares to baseline methods in learning a single reference trajectory.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel algorithm for learning a single policy that can imitate a wide range of motor skills. While reviewers enjoy the results, there exist several concerns, such as (1) novelty against the baselines, (2) poor quality of the baseline approaches, and (3) missing technical details. We encourage the authors to discuss these issues in the rebuttal period.

During the rebuttal, the authors provided a lot of complementary experimental results against the baselines. It resolved a lot of concerns from the reviewers, and one updated his/her recommendation to WA. Based on these positive feedback, we recommend this paper to CoRL this year.
Decision: Accept