Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior

Authors: ruihan  yang
zhuoqun  chen
jianhan  ma
chongyi  zheng
yiyu  chen
quan  nguyen
xiaolong  wang
u c san
  cmu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-14 16:58:33

Abstract: The agility of animals, particularly in complex activities such as running, turning, jumping, and backflipping, stands as an exemplar for robotic system design. Transferring this suite of behaviors to legged robotic systems introduces essential inquiries: How can a robot be trained to learn multiple locomotion behaviors simultaneously? How can the robot execute these tasks with a smooth transition? How to integrate these skills for wide-range applications? This paper introduces the Versatile Instructable Motion prior (VIM) – a Reinforcement Learning framework designed to incorporate a range of agile locomotion tasks suitable for advanced robotic applications. Our framework enables legged robots to learn diverse agile low-level skills by imitating animal motions and manually designed motions. Our Functionality reward guides the robot's ability to adopt varied skills, and our Stylization reward ensures that robot motions align with reference motions. Our evaluations of the VIM framework span both simulation environments and real-world deployment. To the best of our knowledge, this is the first work that allows a robot to concurrently learn diverse agile locomotion skills using a single learning-based controller in the real world.

Main paper: 
 introduction: 
 Researchers have been studying for years equipping legged robots with agility comparable to that of natural quadrupeds. Picture a golden retriever gracefully maneuvering in a park: darting, leaping over obstacles, and pursuing a thrown ball. These tasks, effortlessly performed by many animals, remain challenging for contemporary legged robots. To accomplish such tasks, robots need not only master individual agile locomotion skills like running and jumping, but also the capacity to adaptively select and configure these skills based on sensory inputs. The inherent ability of quadrupeds to smoothly execute diverse locomotion skills across varied tasks inspires our pursuit of a control system with ✓ ✓ ✓ ✓AMP [22] ✓ ✓ ✓ ✓ WASABI [23] ✓ ✓ ✓ ✓ ASE [18] ✓ ✓ ✓ ✓Motion Imitation [17] ✓✓ ✓ ✓ ✓ ✓ ✓ VIM ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓a general locomotion motion prior that includes these skills. We introduce a novel RL framework, Versatile Instructable Motion prior (VIM) aiming to endow legged robots with a spectrum of reusable agile locomotion skills by integrating existing agile locomotion knowledge.Agile gaits [1, 2, 3] for legged robots have been sculpted using model-based or optimization methods at the price of demanding significant engineering input and precise state estimation. Imitation-based controllers are also proposed to learn from motion sequences from animals [4] or optimization methods [5] . Recent works [6, 7, 8, 9, 10, 11, 12, 13, 14 ] also incoporate perception for legged robots. Despite encouraging results, most of these works focus on building a single controller from scratch, even though much of the learned locomotion skills could be shared across tasks. Recent works build reconfigurable low-level motion priors [15, 16, 17, 18, 19, 20] for downstream applications, but fail to make the best use of existing skills to learn diverse locomotion skills with high agility.In this work, we focus on building low-level motion prior to utilize existing locomotion skills in nature and previous optimization methods, and learn multiple highly agile locomotion skills simultaneously, as in Figure 2 . We utilize motion sequences to offer a consistent representation of diverse agile locomotion skills. Our motion prior extracts and assimilates a range of locomotion skills from reference motions, effectively mirroring their dynamics. These references comprise motion capture (mocap) sequences from quadrupeds, synthetic motion sequences complementing mocap data, and optimized motion trajectories. We translate varied reference motion clips into a unified latent command space, guiding the motion prior to recreate locomotion skills based on these latent commands and the robot's state. For legged robots, a locomotion skill is the ability to produce a specific trajectory. We classify this into two aspects: Functionality and Style. Functionality involves fundamental movement objectives, like moving forward at a set speed, while Style focuses on how a robot accomplishes a task, for example, two robots could run at the same speed, but with different gait. Teaching both aspects simultaneously is challenging [21] . We use three feedback: objective performance metrics, qualitative assessments, and detailed kinematic guidance. This structured approach helps the robot master basic functional objectives before refining its locomotion gaits.By incorporating diverse reference motions and our reward design, our VIM learns diverse agile locomotion skills and makes them available for intricate downstream tasks. We evaluate our method in the simulation and real world, as Figure 1 . Our method significantly outperforms baselines in terms of final performance and sample efficiency. 
related work: 
 Blind Legged Locomotion: Classical legged locomotion controllers [24, 25, 26, 27, 1, 28] based on model-based methods [29, 30, 31, 32, 33, 34, 35] and trajectory optimization [36, 3] have shown promising results in diverse tasks with high levels of agility. Nonetheless, these methods normally come with considerable engineering work for the specific task, high computation requirements during deployment, or fragility to complex dynamics. Learning-based methods [6, 37, 13, 38, 39, 40, 41 controllers are proposed to offer robust and lightweight controllers for deployment at the cost of offline computation. Peng et al [42] developed a controller producing non-agile life-like gaits by imitating animals. Though previous works offer robust or agile locomotion controllers across complex environments, these works focus on finishing a single task at a time without reusing previous experience. Peng et al [22] leverage reference motions as prior knowledge when directly addressing specific tasks. Li et al [23] obtain an agile locomotion skill from a single partial rough demonstration including only robot root trajectory. Smith et al. [43] utilize existing locomotion skills to solve specific downstream tasks. Hoeller et al [44] built multiple individual locomotion skills and utilized them for agile navigation. Vollenweider et al. [45] utilize multiple AMP [22] to develop a controller to solve a fixed task set. In this paper, our motion prior captures diverse agile locomotion skills from reference motions including mocap trajectories and trajectories generated by trajectory optimization, and provides them for intricate future downstream tasks.Motion Priors: Due to the low sample efficiency and considerable effort required for reward engineering of RL, low-level skill pre-training has drawn growing attention. Singh et al [15] utilize a flow-based model to build an actionable motion prior with motion sequences generated by scripts. More recent works [16, 17, 18, 19, 46, 20, 47] focus on building low-level motion prior for downstream tasks but fail to include diverse highly agile locomotion skills. Luo et al [48, 49] developed unified motion prior for simulated humanoid robot. Peng et al [18] develop a simulationbased low-level motion prior entirely through unsupervised methods, yet they do not assure the acquisition of agile skills from the reference motion dataset (Additional discussion in Appendix C). In this work, we build motion prior with reference motions consisting of mocap sequences, synthesized motion sequences, and trajectories from optimization methods and learn multiple highly agile locomotion skills with a single controller. Comprehensive comparison is provided in Table 1 . 
learn versatile instructable motion prior (vim): 
 Building Versatile Instructable Motion prior (VIM), as shown in Figure 3 , involves: constructing a reference motion dataset, and training the motion prior with an imitation-based reward system.Reference motion dataset: Our dataset includes N reference motions for various locomotion skills such as cantering, turning, backflips, and jumps. Reference motions are from: (a) Mocap data [50] of quadrupeds; (b) synthesized motions from a generative model [50] to enhance diversity; (c) motions from trajectory optimization methods. While mocap and synthesized motions provide extensive data, not all are feasible for the robot. Thus, trajectory-optimized motions are included for complex moves like jumps and backflips. The detailed motion list is in the Appendix A. To address differences between animals and our robot, we retarget mocap and synthesized sequences as per Peng et al. [4] . Each trajectory is noted as(s ref 0 , • • • , s ref T ), where s ref i is the reference robot state at ith timestep, and T is the length of the reference motion. We denote the dataset asD = {(s ref 0 , • • • , s ref T ) i } N i=1. Each frame includes the robot's pose, velocity, foot position, foot height, and joint angle and velocity without motor commands. Privileged information like robot pose and velocity is used only in simulation, and policies do not require it in the real world.Our motion prior consists of a reference motion encoder and a low-level policy. Reference motion encoder maps varying reference motions into a condensed latent skill space, and low-level policy utilizes our imitation reward and reproduces the robot motion given a latent command. )) from which we draw a sample at each interval to guide the low-level policy. To maintain a temporal-consistent latent skill space, our training integrates an information bottleneck [51, 52] objective L AR , where the prior follows an auto-regressive model [53] . Specifically, given the sampled latent command for the previous time step z t-1 , we minimize the KL divergence between the current latent Gaussian distribution and a Gaussian prior parameterized byz t-1 , L AR (ŝ ref t , z t-1 ) = βKL N (µ t , σ 2 t ) ∥ N (αz t-1 , (1 -α 2 )I)where α = 0.95 is the scalar controlling the effect of correlation, β is the coefficient balancing regularization.Low-level policy training: Our low-level policy π low takes latent command z t and robot's proprioceptive state s t and outputs motor commands a t for the robot, where s t is encoded with a proprioception encoder E prop . We train low-level policy and reference motion encoder using PPO [54] in an end-to-end manner. We introduce learnable motion embeddings for the critic (V low in Figure 3 ) to distinguish reference motions. Episodes initiate with random frames of the dataset and terminate when the root pose tracking error is too large or the episode length is beyond the maximum length.Given the formulation of our motion prior, the robot learns diverse agile locomotion skills with our imitation reward and reward scheduling mechanics. Our reward offers consistent guidance, ensuring the robot captures both the functionality and style inherent to the reference motion.Learning Skill Functionality: To mirror the functionality of the reference motion, we translate the root pose discrepancy between agent trajectories and reference motion into a reward. The functionality reward r func includes tracking rewards for robot root position r pos func and orientation r ori func . Recognizing the distinct importance of vertical movement in agile tasks, the root position tracking is further split into rewards for vertical r pos-z func and horizontal movements r pos-xy func . r func (s t , ŝref t ) = w ori func * r ori func + w pos-xy func * r pos-xy func + w pos-z func * r pos-z funcThe formulation of our functionality rewards is provided as follows, similar to previous work [4] :r ori func (s t , ŝref t ) = exp -10 ∥q root t -q root t ∥ 2 , r pos-xy func (s t , ŝref t ) = exp -20 xroot-xy t -x root-xy t 2 , r pos-z func (s t , ŝref t ) = exp -80 ∥x root-z t -x root-z t ∥ 2where q, q and x, x are the root orientation and position from the robot and reference motion respectively. Unlike previous work [4] , we emphasize root height in our reward, crucial for mastering agile locomotion skills such as backflips and jumps.Learning Skill Style: Capturing the style of a reference motion, in addition to its functionality, expands the application of the locomotion skills by meeting criteria such as energy efficiency, and robot safety. Drawing inspiration from how humans learn [55, 56] -starting by emulating the broader style before focusing on intricate joint movements -our robot first mimics the broader locomotion style with an adversarial style reward and later refines its technique with a joint angle tracking reward.Adversarial Stylization Reward: We train discriminators D i , i = 1..N for all N reference motions separately to distinguish robot transitions from the transition of reference motion [22, 45] and use the output to provide high-level feedback to the agent. Our discriminator is trained with:argmin D i E d M i (s t ,s t+1 ) (Di(st, st+1) -1) 2 + E d π i (s t ,s t+1 ) (Di(st, st+1) + 1) 2where d M i (s t , s t+1 ) and d π i (s t , s t+1 ) denote the state transition pair distribution of the dataset, and the state transition pair distribution generated by the policy for ith reference motion respectively. For each reference motion, the likelihood from the discriminator is then converted to a reward with:r adv style (s t , s t+1 ) = 1 -1 4 * (1 -D(s t , s t+1 ))2 Initially, our adversarial stylization reward provides dense reward and enables the robot to learn a credible gait, but it can not provide more detailed instructions as the training proceeds, which leads to mode collapse and unstable training. Joint Angle Tracking Reward: On the contrary, joint angle tracking reward [57, 17] provides stable instruction for the robot to mimic the gait of reference motion, while the reward signal is small in scale during the initial training stage when the joint angle is away from joint target. Similar to our root pose tracking reward, our joint angle tracking reward has the following formulation:r joint style (st, ŝref t ) = exp   -5 j∈joints qj t -q j t 2   + exp   -20 f ∈feet êf t -e f t 2   + exp   -20 f ∈feet ĥf t -h f t 2  where q j t , qj t are the joint angle of robot and reference motion, e f t , êf t are the end-effector positions of robot and reference motion, h f t , ĥf t are the end-effector height of robot and reference motion. Stylization Reward Scheduling: To learn the style quickly and stably, we propose to use both adversarial stylization reward and joint angle tracking reward with a balanced scheduling mechanism. Considering the discriminator as a "coach", We utilize the mean adversarial reward as an indication of how the coach is satisfied with the current performance. When it's not satisfied with the current performance of the robot, it provides detailed instructions for the robot to learn. Specifically, our stylization reward follows: r style (s t , ŝref For intricate tasks like jumping over gaps, starting from scratch is challenging due to the need for agile locomotion skills and the intensive engineering to balance rewards and regularize motion. Using a low-level motion prior, robots can immediately use existing skills and focus on high-level strategies. For each distinct downstream task, we train a high-level policy π high (As shown in Figure 4 ) takes the high-level observation o high , and proprioception of the robot and outputs latent command for low-level motion prior to utilize the existing skills: a t = π low (π high(ohigh,st),Eprop(st) ).Additional implementation details about observation/action space, reference/proprioception encoder, low-level/high-level policy, and value network can be found in the Appendix F. 
experiments: 
 We evaluate our system in simulation and real-world, comparing with prior work for low-level skill learning and various high-level tasks. Our robot demonstrates life-like agility in the real world. Our system outperforms all baselines.Baselines: We benchmark our method against three representative baselines: Motion Imitation [4, 17, 20] baseline represents a thread of recent works whose imitation rewards are defined solely with errors between current robot states and the corresponding reference states. Generative Adversarial Imitation Learning (GAIL) baseline represents a thread of recent work [18] , whose imitation reward is solely provided by the discriminator trained to distinguish trajectories generated by the policy from the ground truth reference motions. WASABI baseline represents a modified version of WASABI [23] for our setting. Each method trains for 2 × 10 9 samples across 3 random seeds. Both our method and the Motion Imitation baseline adopt identical reward scales for all motion error-tracking rewards. Simulation Evaluation: In the simulation, we measure average imitation tracking errors, episode returns, and trajectory lengths across random seeds. As listed in Table 2 , the tracking error of root pose represents the ability of the robot to reproduce the locomotion skill, and the tracking error of joint angle and end effector position represents the ability of the robot to mimic the style of reference motion. Our method achieves a similar root pose tracking error as the motion imitation baseline with a much smaller joint angle tracking error. This shows that our method strikes a balance between functionality and style, superior to the motion imitation baseline that focuses mainly on functionality. Meanwhile, the GAIL baseline failed to learn the functionality of the reference motions leading to short episode length and the least episode return. We surmise that the GAIL baseline's inadequacy arises from the adversarial reward does not offer temporally consistent guidance throughout skill learning and the mode collapse issue inherent in adversarial training hinders the robot from mastering highly agile skills, such as backflipping. The poor performance of the Motion Imitation baseline may stem from the challenges of balancing different terms and selecting suitable hyperparameters when concurrently learning multiple agile locomotion skills.We provide the ablation study over the reward term and the scheduling mechanism as shown in Table 2 . We found that without Functionality reward, the learned controller could not robustly track the reference motion resulting in smaller Episode Return and, shorter Episode Length on the other hand, removing Style reward results in a significantly higher Joint Angle and End-Effector tracking error. Comparing VIM with and without stylization reward scheduling, we find the former exhibits enhanced style tracking performance, underscoring the value of stylization reward scheduling in refining robot gait tracking.Real World Evaluation: We evaluated learned agile locomotion skills in the real world using specific metrics tailored to different skills, as detailed in Table 3 . We repeated our experiment three times per skill per method per seed since our real-world experiment. For Jump While Running/Jump Forward/Jump Forward (Syn)/Backflip, we measured jumping height and distance. For Pace/Canter/Walk/Trot and Left Turn/Right Turn, we measured linear and angular velocity. Results show our method retains most of the reference motion functionality. The only significant deviation observed in Canter is due to differences between animal and robot capabilities, as quadrupeds use tendons to achieve higher running speeds, which our robot lacks. Despite similar root pose tracking errors in simulations, our method outperforms the Motion Imitation baseline in real-world metrics like jumping height, distance, and velocity tracking error, indicating that mirroring reference motion style improves sim2real transfer for natural gaits. The GAIL baseline struggled with real-world locomotion skills. Figure 5 visually compares real-world trajectories, showing our method's superiority in capturing both motion functionality and style. Due to poor simulation performance, the WASABI baseline was not evaluated in the real world.Latent Skill Space Visualization: We visualize the learned latent skill space in Figure 6 by visualizing the latent embedding corresponding to motion segments in our reference motion dataset via t-SNE [58] . We find that different skills are separated into different regions with clear boundaries. Our reference motion encoder also clusters the skills with similar semantic meaning together: embeddings from Left Turn / Right Turn sequence are close, which enables the smooth transition between different skills. embeddings from Jump While Running & Jump Forward & Jump Forward (Syn) sequence are clustered together. These observations suggest that our system learned a smooth and semantically meaningful latent skill space for solving high-level tasks.To evaluate how our method leverages learned agile locomotion skills for high-level tasks, we designed a set of tasks and tested our method against baselines in simulation and the real world.High-level Tasks & Observation: Our tasks include: Following Command: directing the robot to move with specific linear and angular velocities. Linear velocity commands range from 0 ∼ 2 m/s, and angular velocity commands range from -2 ∼ 2 rads/s. In our motion prior, the robot is trained to move and turn at the reference motion's speed. Hence, to follow a command precisely, the high-level policy needs to smoothly interpolate between different speeds. Jump Forward: directing the robot to jump while running. We have adapted a subset of jumping rewards from CAJun [59] to evaluate policy interpolation between jumping and running motions within a fixed timeframe. Following Command + Jump Forward: directing the robot to either jump forward or adjust to changing commanded speeds. To optimize episode return, the robot should not only use the agile locomotion skills from the reference motion dataset but also develop unobserved skills like executing sharp turns. Detailed high-level observations for different tasks are provided in the Appendix F.Baselines: Given the baseline's subpar performance in low-level motion prior training, we compare our system with three representative baselines without pre-trained low-level controllers: PPO [54] : Controllers trained exclusively on high-level task rewards. AMP [22] : Utilizes reference motion for styling reward in adversarial imitation learning and learns high-level tasks while mimicking reference motions. Hierarchical Reinforcement Learning (HRL) from Jain et al. [60] : Learns a high-level policy sending latent commands to a low-level controller, resembling works that decompose tasks into sub-problems [61, 62, 63, 64, 65, 66, 67] . For fair comparison, we removed the trajectory generator in [60] , used PPO for AMP and HRL, and used full reference motion for AMP and HRL with AMP.We trained all methods on each high-level task for 4 × 10 8 samples with 3 random seeds. Simulation results are detailed in Figure 7 , and real-world results are provided in Table 4 . Real-world Following Commands trajectory is also provided in Appendix I. For the Following Command task, all methods mastered basic locomotion, but ours excelled in efficiency and smooth transitions between diverse linear and angular velocities.In the Jump Forward and Following Command + Jump Forward tasks, which required advanced jumping abilities, baselines struggled. They either moved forward continuously, remained grounded when prompted to jump, or toppled to avoid energy consumption penalties. In contrast, our system seamlessly integrated jumping and running actions, achieving the highest episode return. Despite having a comprehensive reference motion dataset, baselines couldn't harness the skills effectively. This likely stems from the difficulty of deriving agile locomotion skills using only adversarial stylization rewards, similar to the GAIL baseline's poor performance in low-level training. 
limitations: 
 Our current system exhibits several limitations: 1) Safety is not assured with our current method.Introducing safety constraints during training could mitigate this issue; 2) The robot's limited capacity restricts its ability to fully replicate certain motion capture data, like cantering. Upgrading the hardware could address this limitation; 3) Currently, our system does not incorporate dynamics information. This could be improved by integrating adaptation techniques during deployment; 4) The system's low-level motion priors and high-level policies currently lack perceptual information. 
conclusion: 
 In this paper, we propose Versatile Instructable Motion prior (VIM) which learns agile locomotion skills from diverse reference motions with a single motion prior. Our simulation and real-world results show that our VIM captures both the functionality and style of locomotion skills from reference motions. Our VIM also provides a temporally consistent and compact latent skill space representing different locomotion skills for high-level tasks. With agile locomotion skills in our VIM, complex High-level tasks can be solved efficiently with minimum human effort.styling and dense tracking rewards, providing structured supervision for acquiring long-term motor skills, including agile locomotion abilities like backflipping and jumping forward. This sequence-level modeling is essential for the effective learning of complex locomotion skills.• Performance Evaluation: Since ASE learns different motor skills in an unsupervised manner, it's difficult to evaluate the performance of the learned low-level controller (There is no evaluation or benchmark for the low-level controller in ASE paper). While our method learns to imitate different locomotion skills in the dataset at the sequence level, we could directly benchmark the tracking error to evaluate the quality of the learned low-level controller. Benchmark over the low-level controller is also important if we want to build a backbone of low-level skills for diverse potential high-level tasks. 
 
 Appendix: In this section, we provide further discussion on the existing skill-learning framework:• Function Tracking: The resulting controller of a skill-learning framework can accurately track the movement of the robot's base.• Skill Tracking: The resulting controller can faithfully replicate the joint movement patterns of the robot.• Agility: The controller is capable of producing highly agile locomotion skills. Since there is no universally accepted definition of "agile", in our work, we consider a skill "agile" if it involves the robot leaving the ground, such as in a backflip or jump, or running/turning at high speed.• Control Skills to Learn: Given a fixed set of reference motions, the resulting controller can reliably reproduce specific skills. Unsupervised methods like ASE do not guarantee performance on any particular skill in the dataset.• Multiple Skills: The resulting controller is capable of performing a variety of different skills.• Diverse Sources: The resulting controller learns different skills from various sources.• Reusable: The resulting controller can be reused for tasks beyond reproducing the reference motion's skills.• No Privileged Information: The resulting controller does not require privileged information (such as the robot's velocity or position in the world frame) during deployment.• Real Deployment: The proposed framework is validated in real-world scenarios.Additional discussion on the performance of ASE/AMP-based methods in agile locomotion skills: ASE struggles to capture agile locomotion skills for the following reasons:• Difficulty in Learning Agile Skills: Agile locomotion skills, such as jumping and backflipping, are inherently more challenging to learn compared to other locomotion skills like walking or trotting. Due to the well-known mode-collapse issue in the generative adversarial learning paradigm, it is particularly difficult for generative adversarial methods (like ASE/AMP) to discover and learn these complex skills in an unsupervised manner.• Limitations of Transition-Level Learning: As discussed in Appendix C, ASE/AMP performs adversarial learning at the transition level, focusing on the current and previous states of the robot (as shown in Formula (3) in ASE [18] ). However, skills that require a longer sequence of actions, such as backflipping or jumping, are difficult to learn with this approach. For example, executing a backflip involves multiple stages: sitting down, lifting the front legs, pushing off with the rear legs, adjusting the pose in mid-air, and landing. Similarly, jumping and running require coordinated stages of movement. Transition-level supervision lacks the long-term guidance needed to learn these complex, agile skills. = 0.1875, w pos-z f unc = 1.5, w adv style = 1, w joint style = 0.5. Other Rewards: To smooth the robot trajectory, we also include energy penalty r energy , and action smooth reward r action . r energy = -1e -3 * i |τ i × qi | where τ i is the the motor torques applied to the ith joint, and the qi is the joint velocity for the ith joint.r action = -1e -2 * i |a t i -a t-1 i| where a t i and a t-1 i are the action from policy for ith joint at current timestep and the previous timestep.Simulation Setup: We utilize IsaacGym [68] to simulate 4096 robots in parallel and our simulation runs in 200 Hz. During motion prior training, for each robot, we uniformly sample a reference motion from the dataset for it to track.Our high level jumpping reward is adapted from CAJun [59] with the following terms.r jump = 2 * (2 -∥v robot -2∥)/2 + 5 * (Base Height -0.6) * max( f ∈feet ĉf -4, 0) + 3 * f ∈feet ∥1 + c f -ĉf ∥ 2 + 1 * f ∈feet ∥c f -ĉf ∥ * min(h f , 0.16)/0.16Here our desired foot contact ĉf at each step is a binary value generated by the task generator as in CAJun [59] with value 0 for no contact, and value 1 for contact, similar for the actual contact c f , and h f is the foot height over the ground.Our high level command following reward is defined as follows.r following cmd = 1.5 * exp ∥v command -v robot ∥ 2 /0.25 + 1.5 * exp ∥ω command -ω robot ∥ 2 /0.25 -2 ∥Base Height -Target Height∥Here the v command is the commanded target linear forward velocity, v robot is the current forward velocity of the robot. The ω command is the commanded target angular velocity, ω robot is the current angular velocity of the robot.We also include energy penalty and action smooth reward as shown in the other rewards in Appendix F.In addition to the low-level skill comparison in Figure 5 , we provide another low-level skill comparison in Figure 9 . Our controller learned to jump forward in the air with a natural gait, while baselines failed to leave the ground or failed to move forward To better understand of the performance of our high-level policy, we provide Following Command trajectory in Figure 10 . Though our low-level controller only learns to turn with specific angular velocity, our high-level could track different angular velocity in the real world. We provide more detailed observation space for our motion prior. Our Unitree A1 robot has 12 joints, corresponding to 12 Degrees of Freedom (DoF), and we use positional control for the 12 DOF (KP = 40 and KD = 1.0). Specifically, the proprioceptive state of the robot contains:Though adversarial training is generally unstable, we found it relatively stable during our training. We provide the training log of our discriminator and the average adversarial reward across epochs in Figure 11 . Specifically, We applied the following techniques to stabilize the adversarial training.• We clipped the gradient of the discriminator to have the maximum norm of 0.5• We applied gradient penalties during the training of the discriminator, following AMP [22] We think our motion imitation reward also helped stabilize the adversarial training since our joint tracking and end-effector position tracking reward provide fine-grained instruction for policy training. We didn't observe specific latent skill space collapse during our training, we think this is for the following reasons:• We provide motion embedding for the value function to distinguish different reference motions during training, as shown in Figure 3 in the manuscript. With motion embedding, the value function in our method could learn to distinguish different skills easily.• Our tracking reward terms provided dense instruction for the controller to generate different behaviors, which further regularized the latent skill space during training. To study the training behavior in more detail, we visualized the episode return for the joint angle tracking term with the average adversarial reward in Figure 12 . We found that in the first 1000 epoch, the episode return of joint tracking increased swiftly corresponding to the rapidly decreasing period of average adversarial reward. After 1000 epochs, the increasing rate of the episode return of joint tracking drops. We think this phenomenon corresponds to our claim in the manuscript that the model transits from learning the overall motion, where the episode return of joint tracking boost, towards learning the fine-grained behavior using the joint angle tracking reward, where the episode return of joint tracking grows slowly.  


Decision: The paper presents a novel algorithm for learning a single policy that can imitate a wide range of motor skills. While reviewers enjoy the results, there exist several concerns, such as (1) novelty against the baselines, (2) poor quality of the baseline approaches, and (3) missing technical details. We encourage the authors to discuss these issues in the rebuttal period.

During the rebuttal, the authors provided a lot of complementary experimental results against the baselines. It resolved a lot of concerns from the reviewers, and one updated his/her recommendation to WA. Based on these positive feedback, we recommend this paper to CoRL this year.

Reviews: review: title: title: Core idea is Adversarial imitation + joint and root tracking. But has some major issues that need to be addressed
paper_summary: paper_summary: This paper proposes a technique to train single policy for a collection of skills on a legged robot. The approach consists of two types of reward: 1. functionality and 2) style reward. The functionality reward tracks the root orientation and position(z). The style reward has two components that 1) tracks the joint angle and 2) captures the style using N discriminators (1 for each skills) to capture the distribution of state transitions (similar to AMP). The authors claim that their method works better than vanilla PPO, AMP and HRL and provide experiments in sim and real to justify their claims
main_review: main_review: The paper presents an approach to train a single policy to perform multiple skills on a quadruped robot. While the authors show some evidence of sim-to-real transfer and improvement over baselines, I have some concerns about the approach:

## Major concerns:

1) Transition between different skills: One of the advantages of learning a single policy for multiple tasks, is the ability to compose different skills by switching between them as required. It is unclear how the policy can learn this with the current framework. Is the policy exposed to different skill transitions during training? If yes, what happens to $z_t$ during such transitions? To showcase the ability to switch skills, the authors should present more experiments in addition to running-jumping. Also, the hardware results for just jumping is better than running to jumping (the robot does not land with its front feet). I feels like the policy is abruptly switching skills without learning anything about the optimal strategy to switch.

2) Position and joint angle tracking: Why do we want to track the orientation/xy position of the reference trajectory? Isn’t this unnecessarily increasing the complexity of the task? for example, if the reference has a trajectory where the robot jumps from (x=1,y=0) the learned policy can also only jump at (x=1,y=0). It is much more useful to have a policy that can jump at any position.

3) A fundamental problem with the way the tracking rewards seem to be structured is that they break the MDP assumption. Consider a robot is standing in the nominal pose (same state) at two different points in the reference trajectory $t_a$ and $t_b$. Lets say from $t_a$ point the robot moves its left leg forward and at $t_b$ it moves the right leg forward. According the the policy both these states are indistinguishable. But for the same action the policy will get a different reward based on what the current $t$ is. This violates the markovian property. Usually the tracking reward comes with a phase term to overcome this limitation.

4) Instead of using GAIL, using AMP with a single skill is a stronger baseline for the low-level skills. Also, in Table 4. does AMP fail to jump in (Jump Distance)? I believe that this task is simple enough for AMP to be able to solve and might be an implementation/hardware issue. In addition, there seems to be an asymmetry between left and right turning which the authors should explain. How many trials were done for the real world deployments?

## Minor remarks:
1. The paper has a quite a few typos and grammatical mistakes. I encourage the authors to proofread the paper thoroughly. I've listed some of the below:

line 50 - "We use three feedback"

line 85 - "contructing"

line 89 - "quadrupedsBBB"

line 204 - "Funtionality"

2. For HRL, the authors claim that using a trajectory generator is unfair for the purposes of comparison - But from my understanding the proposed method uses trajectories generated by optimization as reference. I'm not sure if this is justified.
3. In line 149-  "joint angle tracking reward [51, 17] provides sparse but stable instruction for the robot to mimic" Why is this reward sparse?
4. Lines 137-139 need citation or should be removed
questions: questions: Listed above.
limitations: limitations: 2
review_summary: review_summary: The paper seem to advocate for a mix of AMP like style reward and tracking rewards to train skills. But the current experiments and arguments presented in the paper do not make a convincing case.
title: title: review
paper_summary: paper_summary: This paper presents a system that can train a robot to imitate multiple reference motions as well as performing transitions between them.
main_review: main_review: Key components of the system:

1 A style reward that is a mix of AMP style reward and imitation-based reward, with a learning schedule to adjust the relative weighting between them.

2. A reference motion encoder that encodes the reference motion, and a high level policy to output the encoding to learn high level tasks.

Pro:
1. real-world demonstration of various quadrupedal skills.
2. A latent space that enables transitions between skills.

Cons:
1. I am surprised the baseline method cannot learn some of the skills demonstrated as many works have shown they can, e.g., 
(1)Adversarial motion priors make good substitutes for complex reward functions, IROS 2022. AMP reward is able to get the robot trotting and pacing.
(2) Lifelike Agility and Play on Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models, using imitation-based reward can also do jumping. 
I am especially concerned about the trotting being a failure in the baseline methods, as trotting is pretty easy.

2. There are some skills that are very interesting, like jumping while running and cantering. However, the generated motion is not as nice looking as the reference.

3. The description and experimentation on the high level policy part of the paper is limited.
questions: questions: 1. "To the best of our knowledge, this is the first work that allows a robot to concurrently learn diverse agile locomotion skills using a single learning-based controller in the real world.". I think "Lifelike Agility and Play on Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models" also learns diverse locomotion skills with a single policy, and has been on arxiv since August 2023.

2. Some concerns I listed in the review sections.
limitations: limitations: 3
review_summary: review_summary: I think this is a good attempt at generating diverse agile skills for quadruped robot. Although the potential of the system is fully explored, e.g, solving more intersting high level tasks, get nicer looking low level skills.
title: title: A reliable locomotion learning framework
paper_summary: paper_summary: This work addresses the challenge of learning a single reusable quadrupedal robot motion prior by imitating diverse reference motion trajectories. The proposed Versatile Instructable Motion (VIM) framework achieves this by learning to project reference trajectories into a latent space. By varying the latent vector, diverse motions can be generated through a learned low-level policy. The framework is trained using both a functionality reward and a style reward. Validation of the method was conducted in both simulation and hardware. The results demonstrate that the proposed method outperforms various baseline methods and successfully generates diverse motions on an A1 robot.
main_review: main_review: Overall, this work proposes a reliable method for learning a single policy that can exhibit multiple locomotion skills.

Strengths:
1. Extensive simulation experiments and quantitative comparisons validate the proposed method.
2. Solid hardware experiment results clearly show that the learned policy can be used to generate diverse motions on a hardware robot.

Weaknesses:
1. The method proposed in this work is not particularly novel; similar works such as [1][2] have been proposed. However, these works do not validate on real-world robots.
2. The clarity of the paper can be improved. For instance, in Table 1, some metrics are unclear. For example, what is meant by agility, and why can't ASE capture agile motion? I suggest that the authors illustrate these metrics and different methods in the supplementary materials.
3. The motivation is somewhat unclear. Why is it beneficial to learn a single controller to capture all motions instead of multiple sub-policies? In [3], different locomotion skills are separately learned, and the results demonstrate super agile movements.
4. There are missing references. [4] also learns a low-level controller and a latent space by imitating reference motions.


[1] Luo, et al. Universal Humanoid Motion Representations for Physics-Based Control.
[2] Luo, et al. Perpetual Humanoid Control for Real-time Simulated Avatars.
[3] Hoeller, et al. Anymal parkour: Learning agile navigation for quadrupedal robots.
[4] Li, et al. Planning in learned latent action spaces for generalizable legged locomotion.
questions: questions: 1.Weakness No. 3 is my main concern. The authors should focus on justifying the benefit of learning a single policy versus multiple sub-policies.
2. I am also interested in how the proposed method compares to baseline methods in learning a single reference trajectory.
limitations: limitations: 3
review_summary: review_summary: I am towards a weak acceptance due to the overall quality of the hardware valuations as well as the reliable of the proposed method.