GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots
Authors: Gilbert Feng
Hongbo Zhang
Zhongyu Li
Xue Bin Peng
Bhuvan Basireddy
Linzhu Yue
ZHITAO SONG
Lizhi Yang
Yunhui Liu
Koushil Sreenath
Sergey Levine
Venue: CoRL 2022 Poster (2022)
Keywords: Legged Locomotion
Reinforcement Learning
Transfer Learning

Abstract: Recent years have seen a surge in commercially-available and affordable quadrupedal robots, with many of these platforms being actively used in research and industry. As the availability of legged robots grows, so does the need for controllers that enable these robots to perform useful skills. However, most learning-based frameworks for controller development focus on training robot-specific controllers, a process that needs to be repeated for every new robot. In this work, we introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots. Our framework synthesizes general-purpose locomotion controllers that can be deployed on a large variety of quadrupedal robots with similar morphologies. We present a simple but effective morphology randomization method that procedurally generates a diverse set of simulated robots for training. We show that by training a controller on this large set of simulated robots, our models acquire more general control strategies that can be directly transferred to novel simulated and real-world robots with diverse morphologies, which were not observed during training.

GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots
Gilbert Feng" Hongho Zhang" Zhongyu Li', Xue Bin Peng', Bhuvan Basireddy", Linzhu Yue?, Zhitao Song?, Lizhi Yang', Yunhui Liu', Koushil Sreenath', Sergey Levine University of California, Berkeley The Chinese University of Hong Kong
Abstract: Recent years have seen surge in aateerrcilll-vvvllaiee and affordable quadrupedal robots, with many of these platforms being actively used in research and industry. As the availability of legged robots grows, so does the need for controllers that enable these robots to perform useful skills. However, most learning- based frameworks for controller development focus on training robot-specific controllers, process that needs to be repeated for every new robot. In this work, we introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots. Our framework synthesizes reneral-purooss locomotion controllers that can be deployed on a large variety of quadrupedal robots with similar morphologies. We present simple but effective morphology randomization method that procedurally generates a diverse set of simulated robots for training. We show that by training controller this large set of simulated robots, our models acquire more general control strategies that can be directly transferred to novel simulated and real-world robots with diverse morphologies, which were not observed during training. (Code and pretrained policies: https: //gi thub. com/ cea.oooooo.c.eoooooooooooooooo,o
HybridRobotica /GenLoco, Video: https: haapps//yyouuu.t wawpttu.ee/5Qus330 Nu4)
Keywords: Legged Locomotion, Reinforcement Learning, Transfer Learning
Introduction
Just as more general-purper models have gained prominence in supervised learning domains, with broadly applicable language models that can solve a variety of NLP tasks [1, 2] and language- conditioned visual recognition image generation models that can be applied to variety of problems and settings [3, 4]. so too we might expect that more powerful robotic learning systems might enable more broadly applicable robotic controllers. In particular, in settings where variety of robotic platforms share the same basic morphology (e.g., the set of commonly used quadrupedal bodies, the set of 7 DoF robotic arms, the set of quadcopters, etc.), we might expect that should be possible to train control policies that can be applied broadly to all robots within particular set. If this were possible, then practitioners who want make use of learned policies would not need start by training policies of their own for their own robot, but could instead simply download pre-trained policy for the general robot class from the web, and then deploy directly on their platform. In this paper, we take step toward this vision in the particular setting of quadrupedal locomotion.
Quadrupedal locomotion offers the potential for robotic agents traverse and operate in complex unstructured environments. However, designing effective locomotion controllers for quadrupedal robots is challenging, it typically requires detailed knowledge of the dynamics of particular system and careful controller design for each desired skill. Model-free reinforcement learning (RL) provides a paradigm that can automate much of the controller engineering process, where an agent learns locomotion skills automatically through trial-and-error. RL techniques have been effective for developing locomotion controllers for large variety of quadrupedal robots [5, 6, 7, 8, 10, 11, 12, 13]. While RL provides a general framework that can in principle be applied to any robot, the resulting controllers are most often specific to the particular robot that they were trained on. Therefore, these controllers will generally be ineffective when deployed on another robot, and new
equal contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

(a) Simulated
Simulated Mini Choctah
(c) Simulated Sirius
(e) Real Cheetah
(f) Real
Figure I: Testing of different simulated and real quadrupedal robots (A1, Mini Cheetah and Sirius) performing pacing gaits using a single locomotion controller. Our controllers can directly deployed from simulation to the real world and across robots with different morphologies (c.g.. body size, leg lengths, masses, etc.) and dynamics, without explicidly training on the specific robots used during testing.
controllers will need to be trained from scratch specifically for the new system. This requires either extensive data collection if using simulation to real-world transfer, detailed simulated models.
Recent years have seen the emergence of a growing catalog of commercially available quadrupedal robots, with many of these systems converging on similar body plans. This similarity in the morpholo- gies of quadrupedal robots may lead one to wonder: is possible to create a generalized locomotion controller that can be broadly applied to different quadrupedal robots? If so, such generalized controllers can greatly reduce the labor-intensive process of continually developing robot-specific controllers for new systems. In this work, we present framework for developing more general locomotion controllers, which can be applied a range of different quadrupedal robots. We focus our investigation on the setting where the test-time morphology is unknown, and thus the goal is to develop controllers that generalize to new robots not observed during training. To this end, we propose morphology randomization method that reduces the need for robot specific information during training by procedurally generating diverse set of morphologies. By randomizing the morphology and dynamics of the simulated robots, system is able train generalized controllers that can be deployed across a variety of different robots, as shown in Fig. 1.
The core contribution of this work is the development of a reinforcement learning framework that produces generalized locomotion (GenLoco) controllers for quadrupedal locomotion skills, which can be deployed on large variety of different robot morphologies. By training controllers on wide range of randomly generated robot morphologies, our system able learn policies that generalize to new robots not seen during training. We demonstrate the effectiveness of our model on many notable quadrupedal robots in simulation and real-world experiments. We show that the learned controllers can be deployed directly on number of commonly used quadrupedal robots, including the Unitree's A1, MIT's Mini Cheetah [14] and CUHK's s Sirius. We also demonstrate that, by introducing morphology randomization into the training process (varying robot parameters such as body size, leg length, and mass during training), our framework is able to more effectively transfer policies from simulation to the real world than policies trained on specific robot. In this paper, we focus on the randomized robot morphology while keeping the number of Degree of Freedoms (DoFs) and links constant. The open-souree source GenLoco policies could be used baselines test controllers for newly developed quadrupedal robots without the need to train a robot-specific policy from scratch.
2 Related Work
RL provides a general framework for learning robotic controllers for a large array of tasks. Instead of requiring tedious manual controller engineering, RL techniques can automatically synthesize controllers for a desired task by optimizing the controller against an objective function [15]. RL has been applied to develop wide range of motor skills for agents in simulation [16, 17, 18. 19]. and the real world [20, 21, 22, 23, 24, 5, 12). But due to challenges associated with applying RL algorithms on real-world systems [25], sim-to-real techniques are commonly used to adapt and
transfer controllers trained in simulation to a physical system [26, 27, 8, 7, 5, 28, 29, 13). Domain
randomization is one of the most commonly used sim-to-real transfer techniques, where the dynamics

---

Phase
GenLoco
Joint-Level
I.PF
Controllers
Nominal Standing Joint Positions
2- 16:6-
2- 15:z
The proposed generalized locomotion control framework for quadrupedal robots. GenLoco is designed to work on a large collection of robots with different morphology and dynamics. The input observations of the policy consist of phase variable é representing progression along motion, a history of the robot's 's raw sensor feedback, and history of past actions. The actions output by the controllers added to time-invariant nominal joint positions and passed through a low-pass filter before being applied to joint-level PD controllers to generate motor torques.
Figure 2:
of a simulator are randomized during training in order develop controllers that can adapt to differences between simulated and real-world dynamics [30, 31, 26, 8, 32, 33, 29, 34]. Real-world data can also be leveraged further improve real-world performance by adapting the dynamics of the simulator [35, 36, 5, 37, 38), or directly finetuning controller's ' behaviors [39, 7, 9, 13]. In this work, we also take a domain randomization approach to transfer controllers trained in simulation to robots operating in the real world. However, unlike many prior sim-to-real methods, which assume a known morphology for the real-world robot, our work explores the development of more general locomotion controllers deployable on a variety of different robots, including morphologies not known during training.
Developing ceneral-purposs controllers that can be deployed on robots with different embodiments can greatly mitigate the overhead of creating individual robot-specific controllers. Devin al. [40] utilized modular network architecture to transfer manipulation skills across a small set of manually-crafted simulated robotic arms. Chen et al. [41] proposed conditioning policies on learned encodings of robot's 's morphology, which allowed a controller to be deployed on number of different robotic arms. A similar morphology encoding approach has also been applied to train full-body motion controllers, which can be deployed on large variety of simulated humanoid agents [42]. Graph convolutional networks have been used to implicitly encode the morphological structure of simulated robots with varying numbers of degrees-of- -ffereesoo- [43, 44]. Gupta et al. [45] learned domain invariant feature spaces in order to transfer manipulation skills across simulated robots with different morphologies. While these transfer learning techniques have shown promising results in simulation, they have yet to be demonstrated on robots in the real world. In this work, we aim to develop general locomotion controllers that can be deployed morphologies. Although prior works suggest that recurrent or memory-based policies can improve transfer by performing "implicit system identification" [26, 34], we found that our approach was able to produce effective policies with fully feedforward neural networks. By taking in past observations and actions input, the model is able to implicitly encode task-relevant information about the robot's morphology, and successfully transfer locomotion skills across a diverse set of quadrupedal morphologies without requiring explicit representation of particular robot's body structure.
large variety of robots with different
3 Generalized Quadrupedal Locomotion Controllers
In this section, we present a framework for training a Generalized Locomotion Controller (GenLoco)) where a single controller can be deployed on large variety of quadrupedal robots with different morphologies. An overview of our system is shown in Fig. 2. The controllers are trained through model-free reinforcement learning. But unlike most prior RL frameworks for robotic locomotion, which assume fixed morphology for the robot during both training and testing, our work explores the setting where the embodiment of the robot can vary, and the particular embodiment at test time not known priori. Our models are trained by randomizing the morphology and other dynamics properties during training simulation, thereby encouraging the controller to learn adaptable strategies, which can be effectively applied on different robots. At every timestep, the controller receives history of sensory observations as input, which can be used to infer relk-nel information about the robot it is currently deployed on. The controller then outputs actions, which are added to time-invariant
3

---

Lakage
Robot Base
Hip Joint
Hip Link
Thigh Join
Jait
Thigh Link
SAI Chorat
Calf Link
Foot
Figure 3: Many common quadrupedal robots follow a common morphological template, consisting a robot base (6 DoFs) and four 3-DoF legs. This design is followed in rohots such as Unitree's 's A1, Gol, Aliengo, Laikago, MIT's Mini Cheetah, CUHK's s Sirius, and Boston Dynamies' Spot. The robots highlighted with solid circles are the ones used to validate our system in the real world.
Table I: Morphology generation parameters.
The nominal values are based on the Al robot parameters
(Appendix A).
Parameter
Min and Max Values
Parameter
Min and Max Values
Size factor a
0.2. 1.2
Calf length 4.
a x [0.11, 0.33) m
Base length by
a
0.134, 0.400 m
Calf radius
x [0.01, 0.03)
Base width b..
a
0.097, 0.291
Thigh length
0.75, 1. 1.25 x Calf length 4.
Base height by
X
00.057, 0.171 m
Thigh radius r.
0.75, 1.25 x Calf radius Te
Base density by
[400, 1200] kg/m²
Foot radius ry
1.5x Calf radius Te
Link masses
0..5, 1.5] nominal value
Joint PD gains
nominal value x
rohot mass
X 0.77 1.3)
nominal mass
nominal joint positions to specify target motor positions for each joint. The desired motor positions are passed through low-pass filter to mitigate undesirable high-frequency movements [7], before being applied to joint-level PD controllers to generate motor torques.
3.1 Morphology Generation
The key to developing generalized locomotion controllers lies in training the controller on a diverse collection of robots, which encourages the controller to learn strategies that can generalize to new robots not observed during training. At the start of each training episode, we procedurally generate a random robot morphology based on a predefined morphology template. As illustrated in Fig. 3, many commonly used quadrupedal robots follow a similar body structure, consisting of a robot base and four legs, each of which has 3 DoFs: hip, thigh, and knee joints. The robot's feet can be modeled as spheres, and the contacts between the feet and the ground can be approximated as point contacts. This design template has been followed in widely used quadrupedal robots, including Unitree's 's A1, Go1, Laikago and Aliengo, Boston Dynamics' Spot, MIT's Mini Cheetah, and CUHK's s Sirius. Therefore, system generates random morphological variations based on this template by randomizing the proportions of the various body parts and their respective dynamics properties. A sample of robots shown in Fig. 2. The randomized morphology
produced by our morphology generation procedure parameters and their respective ranges are recorded in Table 1. Below, we provide a more detailed account of the major parameters of variations in our morphology generation procedure.
Size factor. We introduce a size factor a, which uniformly scales the size of each body part (e.g. robot base and legs). Because a multiplicatively scales along every dimension, it is sensitive parameter, and we found randomly sampling values of a from the range [0.8, 1.2] to be sufficient in capturing the variation among most robots. This factor facilitates positive correlation between robot component sizes, reducing the likelihood of generating morphologies with distorted proportions (c.g.. a robot with an extremely large base but extremely small legs).
Robot base parameters. The parameters of the robot base include its size and density. We model the base geometrically as a rectangular box. The dimensions, such as length b, width b,s and height bn. of the base are randomized to cover a wide range of robot base sizes. The density of the base is also randomized between [400, 1200) kg/mi kg/m., /33 The ranges of values, detailed in Table 1, are selected to encapsulate the sizes and masses of popular industrial robots as listed in Appendix E.

---

Leg parameters. The robot's 's upper leg (thigh) and the lower leg (calf) are modeled as cylindrical solids, and the robot's feet are represented by spheres, as shown Fig. 3. Specifically, the sizes of the robot's thigh and foot are chosen based on the robot's calf link dimensions, as detailed in Table 1. The range of values for the thigh length I specified to be +:25% of the calf length l.- Similarly, the range of values for the thigh radius is defined as +25% of the calf radius Te- By correlating the properties of different segments of the legs, we can prevent the morphology generator from producing implausible designs, such as a robot with large calves but very small thighs and feet, which would hinder the robot's locomotion capabilities.
PD gains. Larger and heavier robots typically require stiffer joint-level PD gains in order to generate larger motor torques. Therefore, the gains used in the PD controllers are also scaled with respect to the mass of the robot. The nominal mass is set the mass of the A1 (12.458 kg). This randomization scheme helps to ensure that the PD controllers are sufficiently strong for larger and heavier morphologies.
4
Training
Our controllers are trained to perform various locomotion skills using reinforcement Saaning-based motion imitation framework based on Peng et al. [7]. The goal of these controllers to imitate a given reference motion q" {9h, qi...., } which specifies target poses q at each timestep I, The reward ry at timestep is computed according to:
rt wPr . + w'r + w , + "
with w. = 0.6, w" = 0.1, wh 0.15, by 0.15.
The pose reward rf = exp - 11ا1 12] encourages the robot to match its local joint rotations qr with those specified by the reference motion, where o' denotes the rotation of the j-th joint. The velocity reward rž follows a similar form, and encourages the robot to match the joint velocities of the reference motion. The base position reward r encourage the robot to track the motion of the base from the reference motion. A more detailed description of the reward function is available in Appendix B, and additional implementation details are described in Appendix C.
hp and base velocity reward r
State and action spaces. As shown in Fig. 2, actions from the controller a, specify target joint positions qd E R12, which are used by the joint-level PD controllers to determine desired motor torques. The controller operates at 30 Hz. To encourage smoother motions, actions are processed with low-pass filter before being applied to the robot. The observation S; of the controller includes three components: 1) 15-timestep history of robot raw sensor feedback q 15:t. and qr-15:t. 2) 15-timestep history of past actions -6-16:1-1+ and 3) a phase variable Φ E [0, 1] is the normalized time that indicates the robot's ' progress along the reference motion, where denotes the start of the motion and denotes the end [16]. The robot's ' sensor feedback at each timestep consists of the base orientation q hake R from IMU sensors, recorded
quotation, and the measured joint positions E 12, Note that the observations do not include quantities that require an explicit state estimator, such the linear velocity of the base [6, 7, 33, 11] or contact sensors [8]. The history of sensor readings and actions provides the GenLoco policy some information that can be used to perform state estimation and infer the robot's dynamics and morphology.
Dynamics randomization. In addition to randomizing the morphological structure of the robots, we also randomize the dynamics parameters of the simulation during training [31] improve the robustness of our policy and facilitate sim-to-real transfer. The randomized dynamics parameters include three categories: 1) link mass, link inertia, and ground friction to deal with modeling errors, 2) motor strength (torque limits), jovet-level PD gains and motor damping ratio to mitigate the uncertainties of motor dynamics, and 3) the latency between the policy and the joint-level controllers. The randomization ranges of each parameter are detailed in Table 3 in Appendix D.
Episode design. Each episode has 100 timesteps, lasting about seconds. Early termination is applied if the robot deviates too far from the reference base position and orientation [7]. At the start
5

---

SpotMicro Pacing
Pacing
Mini Cheetah Pacing
Aliengo Pacing
Laikago Pacing
Sirius Pacing
ANY mal-B Pacing
ANYmal-4 Pacing
Spot Pacing
SpotMicro Spinning
Spinning
Mini Cheetah Spinning
Aliengo Spinning
Laikago Spiniing
Sirius Spinning
ANYmal-B Spinning
ANYmal-C Spinning
Spot Spinning
Figure 4: GenLoco policies deployed on collection quadrupedal robots exist real life. separate models are trained to perform pacing gait and a spinning gait respectively. The GenLoco policies are trained in simulation using only procedurally generated robots, and robots used for testing are not used in the training process. The learned controllers can be directly deployed all of these robots, including the ANYmal-sentt robots which have distinct knee joint design, to perform agile maneuvers without additional training.
of each episode, new morphology is generated according to the procedure described in Sec. 3.1, and the dynamics parameters are also randomized according to Sec. 4.
GenLoco policies ve are trained using Proximal Policy Optimization (PPO) [46], and all simula- tions are performed in Pybullet [47]. Each policy was trained with 800 million samples, taking approximately weeks 16 CPU workers.
Simulation Validation
In this section, validate the effectiveness of models on controlling quadrupedal robots with different morphologies by direct transfer in simulation. We compare our GenLoco policies with the policies trained on specific robots, which use the same training settings introduced Sec. We train controllers for imitating two reference motions: forward moving pacing gait and an in-place spinning motion. The reference motions (using motion capture data from real dog [7]) are retargeted to the Al 's morphological features, and rescaled for each new morphology to account for differences in sizes and proportions. Thus, new reference motions are not required during test-time.
After being trained on procedurally generated morphologies, our controllers were tested on simulated models of commercially available robots using nominal standing poses and PD gains found in [48]. Robots used for testing include the A1, Aliengo, Go1, Mini Cheetah, Sirius, Laikago, Spot, SpotMicro,
ANYmal-B B. and ANYmal-C. The SpotMicro [49] is relatively tiny, with mass 4.8 kg and fully-standing height of 0.26 m. The Al and Mini Cheetah are small robots weighing about 10 kg and standing about 0.4 m height. The Sirius, Aliengo. Laikago, and Spot are heavier and larger, each having height 2 0.5 m. The morphological parameters of these robots generally lie within the training distributions. However, the ANYmal-B and ANYmal-C have a different knee joint design from the design template used for training. Detailed specifications of these robots are listed in Appendix E.
5.1 Zero-Shot Transfer to Novel Robots
As shown in Fig. 4 and the supplementary video, our controllers can be directly deployed on large variety of quadrupedal robots. None of the robots tested on were used during training. Notably, GenLoco policies successfully transfer to the ANYmal-series robots, which possess an inverted knee joint design different from the template used during training (Fig. 3), without additional tuning. This highlights the generalizability of our proposed GenLoco framework.
Our GenL Loco policies are some of the first locomotion controllers that can be deployed directly on different quadrupedal robots without further fine-tuning. By providing history of past observations and actions to the controller, we enable it to leverage the history to infer the morphology of the robot

---

Belurns
Retura
Size Factor a
Calf Longth & Thigh Length 4
Base Mass
Size Factor
Calf Length
Thigh Length
Base Mass
(a) Varying Morphology Parameters, Pacing
(b) Varying Morphology Parameters, Spinning
Notume
Returns
05
Cround Priction Motor Friction Motor Strength Latency
Ground Priction Motor Priction Motor Strength
Lateney
(c) Varying Dynamics Parameters, Pacing
(d) Varying Dynamics Parameters, Spinning
Figure 5: Benchmark of performance of GenLoco policies and policies trained specifically for Al robot to perform pacing and spinning skills on range of different robot morphologies and dynamics parameters in simulation. Green lines are the normalized return of the GenLoco policy (ours) while orange ones are those of the Al-specific policy. Returns are calculated by normalizing the cumulative reward (Appendix B) over the episode length. The red dashed lines indicate the training range while the black dashed lines denote the Al's morphological parameters. Note that for the varying morphology randomized morphology, and the dynamies randomization range is same for all of these policies. Overall, the GenLoco policies outperform the A Al-speiific policies over different morphology and dynamics parameters. Furthermore, GenLoco is able to generalize over larger range of morphologies and dynamics. Each testing episode lasts 100 timesteps and returns are averaged across 10 trials.
the A Al-speiiiic policy not trained with
it is currently deployed on, and by randomizing the morphology during training the model develops more general control strategies that effectively execute a desired skill different morphologies.
5.2 Out-of-Distribution Generalization
To further evaluate the ability of GenLoco to generalize learned locomotion skills different morphologies, we test the controllers extensively on a large range of simulated robot morphologies. We evaluate the controller's performance when varying four morphology parameters: 1) body size, which is scaled by size factor a, 2) calf length lc. 3) thigh length 4, and 4) body mass 6m The testing range set to be much larger than the range used during training detailed in Table We compare GenLoco to policies trained specifically for the AI, since the nominal values used in the morphology generation process (Table 1) are based on the original A robot. The comparisons with the pacing gait controllers and spinning gait controllers are shown in Fig. 5a and Fig. 5b, respectively, The GenLoco policies are overall able to better generalize to out-of of of-disstrnttion robots, exhibiting more robust behaviors and maintaining higher returns across larger variations in the morphological parameters than the A Al-speccfic policies. In the case the pacing skill, GenLoco's 's performance does not degrade until the robot size is more than times larger than the maximum training size. While the A Alsppecific policy shows some robustness to small variations in the morphology parameters, the policy fails when deployed other robots, such as the Sirius.
In order to understand the advantages of the morphology randomization during training, we test the robustness of the GenLoco policies and A Al-specific policies by varying the dynamics parameters. In addition to generalization to different morphologies, our model is also robust to large variations the dynamics of system. Fig. 5c compares the performance of GenLoco policies to Al-specific policies that were also trained using the same range of dynamics randomization detailed in Table 3. As demonstrated in Fig. 5c, 5d, for the same range of dynamics randomization during training, GenLoco policies consistently outperform the Al Alsspecific policies. In particular, GenL.co policies demonstrate notable robustness to changes in the dynamics parameters related actuated joints, such motor friction and strength. This improved performance on out-of-distribution settings is likely in part due to random morphologies introducing more diverse range of dynamics for the policy to train on.
6 Real-World Deployment
Finally, we evaluate the effectiveness of the learned controllers robots in the real world. We test the controllers on three robots, Unitree's AI, MIT's Mini Cheetah and CUHK's Sirius. As shown in Fig. 6 and the supplementary video, our GenLoco policies can be deployed the real robots in
7

---

A1
Mini Cheetah
Sirius
(a) Pacing gait using a single GenLoco
(b) Spinning gait using a single GenLoco
Figure 6: A single GenLoco policy can be deployed directly on different robots, which it was not trained on, in the real world, enabling them to perform agile maneuvers such as pacing and spinning.
Table 2:
Normalized return (mean 1 standard deviation) of policies deployed on real robots over test repetitions
Pacing
GenLoco on A1
A1-Specific
GenLoco on Cheetah
Cheetah-Specifi
Return
0.773 + 0.054
0.696 + 0.032
0.743 + 0.092
0 (Failed in 6/6 trials)
Spinning
GenLoco on A1
A1-Specific
GenLoco on Cheetah
Cheetah-Specific
Return
0.670 + 0.070
0.572 + 0.049
0.721 + 0.090
0.258 + 0.013
zero-shot manner, enabling the robots to perform agile maneuvers such as pacing and spinning. We further compare the real-world performance of Genl Loco to policies trained specifically for the Al and Mini Cheetah. We performed 6 trials for each policy using the same experimental setup, and the performance of these policies are available in Table 2. We note that the performance of GenLoco policies, which work both robots and were trained on neither, is overall better than the policies trained for specific robots, as shown in the supplementary video. Furthermore, the Mini Cheetah- specifie pacing policy failed in all 6 real robot deployment trials, despite working well in simulation as shown in Fig. 4. Notably, such performance degradations during sim-to-real transfer do not occur for the GenLoco policies under identical environment cireumstances. This suggests that solely randomizing the dynamics for a single robot may be insufficient due to differences in morphological characteristics between simulation and the real world. By training with randomized morphologies, the Genl Loco policy better adapts to kinematics variations and therefore maintains robustness during sim-to-real transfer. An additional note of interest that the timeline from completion of the Sirius hardware to our deployment of GenLoco control on Sirius was mere days. The successful zero-shot transfer observed in this experiment further highlights the utility of our model.
7 Limitations
As shown in Fig. 5a and 5b, the performance of GenLoco policies drops dramatically for larger robots. This may suggest that using more aggressive randomization and more expressive model architectures with recurrence, such as in [50], could improve the model. Moreover, our models cannot be deployed on robots with different number of Dols. More flexible architectures. such as graph neural networks [43, 44], may allow for adaptable models that can handle variable numbers of DoFs.
8 Conclusion
In this paper, we presented an RL-based framework for training generalized locomotion controllers that can be deployed on diverse set of quadrupedal robots. We show that simple history-based model, trained on procedurally generated robots in simulation, can be successfully transferred to a large variety of new robots, which were observed during training. The trained models also be deployed directly on real robots, without requiring any additional training on the physical systems. While our experiments have been focused on quadrupedal robots, our method is general and can also be applied to robots in other domains, such as robotic arms and quadrotors. However, the effectiveness of our models remains limited to robots that have the same number of DoFs and follow predefined morphological template. Despite these limitations, we hope our work will provide stepping stone towards more general-purpoot purpose controllers that can be widely and conveniently deployed on a diverse catalog of robots.
8

---

Acknowledgments
This work was supported in part by Hong Kong Centre for Logistics Robotics and in part by ARL DCIST CRA W911NF-17-2-0181. We thank Prof. Sangbae Kim, the MIT Biomimetic Robotics Lab, and NAVER LABS for lending the Mini Cheetah for experiments.
References
[1] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-41 16.. Association for Computational Linguistics, 2019.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 88668. PMLR, 2021.
[4] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learming, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, 18-24 Jul 2021.
[5] Hwangho, Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[6] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
[7] X. B. Peng, E. Coumans, T. Zhang, T.W.. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 07 2020.
[8] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems (RSS), 2021.
[9] L. Smith, J. C. Kew, X. B. Peng. S. Ha, J. Tan, and S. Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. 2022 International Conference on Robotics and Automation (ICRA), pages 1593-1599. IEEE, 2022.
[10] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. Robotics: Science and Systems, 2022.
[11] G. Ji, J. Mun, H. Kim, and J. Hwangbo. Concurrent training of control policy and state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7 (2):4630-4637, 2022.
[12] C. Yang, K. Yuan, Q. Zhu, W. Yu, and Z. Li. Multi-expert learning of adaptive legged locomotion. Science Robotics, 5(49)eebbb2174, 2020.
[13] Y. Ji, Z. Li, Y. Sun, X. B. Peng. S. Levine, G. Berseth, and K. Sreenath. Hierarchical rein- forcement learning for precise soccer shooting skills using quadrupedal robot. arXiv preprint arXiv:2208.0 01 /60, 2022.
[14) B. Katz, J. Di Carlo, and Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. In Int. Conf. on Robotics and Automation (ICRA), pages 6295-6300 2019.

---

[15] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11)):238-12)) 122411244,, 2013.
[16] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37(4):143:1- 143:14, July 2018. ISSN 0730-0301.
[17] L. Liu and J. Hodgins. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Trans. Graph., 37),,, July 2018. ISSN 0730-0301.
[18] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, Todorov, and S. Levine. Learn- ing Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[19] S. Lee, M. Park, K. Lee, and Lee. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG), 38(4): 2019.
[20] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. 2004, volume 3, pages 2619-2624, 2004.
[21] R. Tedrake, T. W. Zhang, and H. S. Seung. Stochastic policy gradient reinforcement learning on simple 3d biped. In Proceedings the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2004), volume 3, pages
Piscataway, NJ, USA, 2004.
[22] G. Endo, J. Morimoto, T. Matsubara, J. Nakanishi, and G. Cheng. Learning cpg sensory feedback with policy gradient for biped locomotion for a full-body humanoid. In Proceedings of the 20th National Conference on Artificial Intelligence Volume 3, AAAI'05, page 1267-1273. AAAI Press, 2005.
[23] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018
[24] T. Haarnoja, A. Zhou, S. Ha. J. Tan, G. Tucker, and S. Levine. Learning to walk via deep reinforcement learning. Robotics: Science and Systems, 2019.
[25] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations, 2020.
[26] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, 2018
[27] 7.. Xie, P. Clary, J. Dao, P. Morais, J. Hurst, and M. van de Panne. Learning locomotion skills for cassie: Iterative design and sim-to-real. In Proc. Conference on Robot Learning (CORL. 2019), 2019.
[28] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics, 2022.
[29] D. Rodriguez and S. Behnke. Deepwalk: Omnidirectional bipedal gait by deep reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 3033-3039, 2021.
[30] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning a universal policy with online system identification. In N. M. Amato, S. S. Srinivasa, N. Ayanian, and S. Kuin- dersma, editors, Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16, 2017, 2017.
[31] J. Tobin,, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 20/7 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23-30, 2017.
10

---

[32] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In 5th Annual Conference on Robot Learning. 2021.
[33] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2811 2811-2817, 2021.
[34] Y. Shao, Y. Jin, X. Liu, W. He, H. Wang, and W. Yang. Learning free gait transition for quadruped robots via phase-guide controller. IEEE Robotics and Automation Letters, 7(2): 1230-1237, 2021.
[35] J. Tan, Z. Xie, B. Boots, and C. K. Liu. Simulation-based design of dynamic controllers for humanoid balancing. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1729927736 Oct 2016.
[36] J. Hanna and P. Stone. Grounded action transformation for robot learning in simulation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI), February 2017.
[37] K. Lowrey, S. Kolev, J. Dao, A. Rajeswaran, and E. Todorov. Reinforcement learning for non non-prehensile manipulation: Transfer from simulation physical system. In 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), pages 35 42. IEEE, 2018.
[38] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Roboties and Automation (ICRA), pages 8973-8979. IEEE, 2019.
[39] W. Yu, V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3503-3510. IEFE, 2019.
[40] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference robotics and automation (ICRA), pages 2169 2179-2176, 2017.
[41] T. Chen, A. Murali, and A. Gupta. Hardware conditioned policies for multi-robot transfer learning, Advances in Neural Information Processing Systems, 31, 2018.
[42] Won and J. Lee. Learning body shape variation in physics-based characters. ACM Trans. Graph., 38(6), nov 2019.
[43] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. International Conference on Learning Representations, 2018
[44] W. Huang, Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In ICML, 2020.
[45] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In International Conference on Learning Representations, 2017.
[46] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint 1 17070063777 2017.
[47] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet..org 2016-2019.
[48] chvmp, "champ". attps: http://thhubbbcoa//c/////c/aaaaa Accessed: 2022-06-15 15.
[49] Spotmicroai.
[50] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint arXiv:2205.00 06175. 2022.


Reviews:
[{"review_id": "keZfp2dozW-", "reviewer": "Reviewer_3Unp", "summary": "This paper presents a reinforcement learning approach based on domain randomization for training unified adaptive locomotion controllers for generic quadruped robots with a fixed kinematic chain topology and a wide range of morphologies.\n\nGiven a target locomotion task, the approach trains a single policy in simulation by randomizing the quadruped’s links geometries and dynamics at each episode over a wide distribution encompassing those of many real-world quadruped designs.\n\nA curriculum that linearly extends the parameter distribution support as training progresses is shown to speed up convergence.\n\nExtensive experiments evaluate the learned controller on a wide range of quadrupeds in simulation and on two real-world ones on pacing and spinning gait locomotion tasks. It is shown to be more robust to varying parameters with respect to robot-specific policies and to be directly transferable to unseen real robots.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work has the significant merit of empirically exploring the capabilities of relatively simple domain randomization and model-free RL techniques to a previously unseen range and scale in robot learning of complex locomotion tasks.\nIt demonstrates that a single policy trained with domain randomization over a high-dimensional distribution of morphologic and dynamic parameters allows to control a wide range of robots and, for the first time, can be directly applied beyond a single real robot with good performance.\n\nFrom the technical point of view, the approach is relatively straightforward. It brings together several standard algorithms and methods for locomotion controller learning and domain randomization for sim to real transfer, Scaling them up well beyond previously published studies and demonstrating their effectiveness.\n \nExperiments are generally well-executed and support generalization and transferability claims, although several statements would benefit from more analysis and in-depth discussion. I report my comments for each experimental section:\n\n*5.1 Zero-Shot Transfer to Novel Robots*: The observation on successful deployment on ANYmal-B is purely descriptive and high-level. Too few details are provided on the setup and how joint references are adapted to work on such a different knee design with respect to the training distribution. This result is indeed interesting, but the reasons behind this successful transfer shall be more thoroughly discussed to be informative.\nAlso, at L213 authors claim that “we enable it to leverage the history to infer the morphology of the robot”. This appears too strong, since there are no detailed analyses of how the policy network actually employs the information included in the state and action historical data. It is very unlikely for it to infer the morphology of the robot explicitly. Still, one could say that the policy may learn to implicitly exploit part of the information contained in the history in a task-oriented way.\n\n*5.2 Out-of-Distribution Generalization*: This section tests the robustness and generalization to out-of-distribution morphology and dynamics parameters of the proposed method. The plots reported in Figure 5 successfully supports the claim of better generalization of the proposed policy trained with domain randomization with respect to a baseline policy trained on a specific morphology. This enhanced generalization is quite expected by a policy learned on a wide distribution of morphologies.\n\n*6. Real-world deployment experiments:* These experiments are relevant and successful in demonstrating the transferability of the learned adaptive controller to multiple different real robots. To my knowledge, this is the first individual locomotion policy successfully deployed on two different real-world quadrupeds, which represents a clear technical achievement.\nStill, the comparison with robot-specific policies and the claims of superior performance are not fully convincing for several reasons. First, the number of repetitions of the comparison experiments in Table 2 is not reported, which does not allow to conclude on this matter. Secondly, it is not possible to conclude on the fairness of the single-robot baseline policy training. It is surprising that a successful locomotion policy targeted to a single robot (and thus to a simpler learning task) cannot be trained and its suboptimal behaviors cannot be dealt with by using standard sim-to-real approaches, e.g. by dynamics randomization. The paper does not report enough details on how the baseline policies are trained ad on the possible causes for suboptimal behaviors (i.e., drifting and failure).\n\nThe quality of presentation is generally high. The paper is well-structured, well-written and pleasant to read. Figures are high-quality and informative.\n\nHowever, some parts of the method and results description appear to lack enough detail and depth. For example, some key elements like the policy network structure, initial state distribution (and reference state initialization?) are missing, and, as already mentioned above, some results and observed phenomena are only described without further analysis, e.g.: the reason why the policy may be compatible with a different knee design; the “morphology inference” enabled by historical data; reasons behind single-robot policies failures when deployed on real robots.\n\nMore comments on potential improvements:\n- L24 Consider mentioning \"Yang, Chuanyu, et al. \"Multi-expert learning of adaptive legged locomotion.\" Science Robotics 5.49 (2020): eabb2174.\", which represents a relevant line of work employing both mixture of expert models (inspired by PFNN and MANN) trained on reference data and RL and demonstrates real-world deployment on a quadruped.\n- L62 Consider including the following relevant work, which to my knowledge is the first successful humanoid (not just biped) walking policy learned in simulation and successfully transferred to a real-world humanoid: Rodriguez, Diego, and Sven Behnke. *\"DeepWalk: Omnidirectional bipedal gait by deep reinforcement learning.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.*\n- L85 “Although prior works suggest that recurrent or memory-based policies can improve transfer by performing “implicit system identiﬁcation,” …”: consider explicitly citing relevant prior work\n- L203 refers to ANYmal in Figure 3, where it is not actually shown\n- Minimum base length, width and height in Table 1 look way too small\n- L123 “This factor facilitates positive correlation between robot component sizes”: unclear statement\n- L144 “Our controllers are trained to perform various locomotion skills using a reinforcement learning-based motion imitation framework [3]”: The proposed approach does not seem to use exactly the same motion imitation framework proposed in [3], that learns a latent space representing morphological variability. Which parts are included and which ones are not?\n- What is the initial state distribution? Is reference state initialization employed?\n\nOthers:\n- L52 real-world $/rightarrow$ real world\n- L80 trained $/rightarrow$ learned\n- L88  to infer the morphological structure of a robot $/rightarrow$ to implicitly encode task-relevant information about the robot’s morphology (?)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper presents relevant results showing the successful scaling of existing domain randomization techniques for RL to unprecedented levels. Also, the first successful deployment of a single learned adaptive locomotion controller on two different quadruped robots in the real world represents a technological breakthrough.\n\nAuthors' responses and improvements to the submitted material successfully address my concerns.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "\nUpdate: Authors' responses and improvements to the submitted material successfully address my concerns.\n\n-------\n\nConsider addressing the weaknesses listed above in the strengths and weaknesses section, i.e., \n- Experiment repetitions\n- Single robot baseline fair comparison in real robot deployment experiment\n- Successful deployment with inverted knee design discussion\n- Better explain a few passages and concepts\n- Other points listed above\n\n"}}, {"review_id": "eovtzQuQkp0", "reviewer": "Reviewer_KDQ2", "summary": "The paper focuses on learning generalized locomotion policies that can be applied to a variety of quadrupedal robots of similar morphology (num DOFs and links are constant). The paper procedurally generates robots with randomized morphology and dynamics and then trains locomotion policies on these robots via a morphology curriculum.\n\nThe paper states that policy transfer across morphologies has been achieved before but only in sim.\n\nThe paper shows transfer to multiple robots in simulation including the AnyMal-B without fine tuning and shows that morphology randomization allows better OOD performance. The paper also shows transfer to two robots in the real world and shows that morphology randomization produces higher return.\n\nIn my opinion the primary contribution of the paper is showing that simple randomization strategies and policy networks can enable learning of policies that are applicable across multiple robots in simulation and the real world.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "----Strengths\n\nThe paper is well written, well illustrated and easy to follow.\n\nThe paper shows transfer to multiple robots in simulation including the AnyMal-B without fine tuning and shows that morphology randomization allows better out of distribution performance. The paper also shows transfer to two robots in the real world and shows that morphology randomization produces higher return.\n\nFigure 5 nicely shows out of distribution results but could be better illustrated and labeled (see suggestions later in review). \n\nLines 239-240 may be the most important sentences in the paper. Maybe there’s a general lesson to be learned here that even with strong kinematics models, dynamics randomization is not enough for robust performance in the real world, and kinematic randomization should also be performed. You may want to expand on this a bit more.\n\n----Weaknesses and questions\n\nThe paper’s methodology borrows very heavily from existing works, namely DeepMimic and Learning Agile Robotic Locomotion Skills, including identical or nearly identical actions, controllers, reward formulations, reference data and simulator.\n\nSome randomization values seem quite specific and the paper doesn’t provide much insight as to how one should in general determine these randomization ranges. One thing that would help is a table of parameter values for the real robots so that they can be compared to the randomization ranges used in the paper. [Edit: I see that some real world values are provided in Appendix E, but several are still missing.] It would also be good to have a more specific written justification for how the ranges were chosen.\n\nLink masses are not discussed in the explanation of parameters. Also, what is the “nominal value” and how was it chosen? [Edit: This seems to be given in Appendix A. This should be referenced in caption of Table 1 or in the description of table parameters.]\n\nWhat is the “nominal value” for the PD gains? How was it chosen?\n\nThe reward function weights seem arbitrarily determined or borrowed from previous literature. How should a researcher determine these weights for the first time? How sensitive is policy learning to variations in those weights?\n\nOnly two reference motions were used, which seems quite limited. More reference motions should be examined in order to strengthen the paper.\n\nAppendix D: Are the learning curves for one seed? If so, they should be shown for multiple seeds. Why does curriculum learning performance degrade? It looks like no-curriculum might even surpass curriculum if the graph is extended further. Longer term performance should be shown. The evidence for the benefit of curriculum learning does not seem convincing enough.\n\nWhy did transfer to AnyMal B work despite the very different morphology? What was the intuition?\n\nAs I understand, section 5.1 shows how training on procedurally generated robots transfers to real robot morphologies in simulation. I think the paper could be clearer that although the agent didn’t train specifically on those real robot morphologies, those morphologies were within the domain randomization distribution. If they weren’t, please state that clearly\n\nFigure 5: the out of distribution performance seems impressive. However I’m not exactly clear why GenLoco would perform better than an A1-specific policy at the point where the test time dynamics parameters exactly equaled the actual A1 dynamics parameters. Can you provide some insight here?\n\nFigure 5: What does return actually mean here on an absolute level? It would be good to know what this actually means in terms of for example mean joint position error, mean joint velocity error, and mean body pose error.\n\nI don’t believe RLOC (Gangapurwala et al) was cited in this paper and I believe it should be compared to as well.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In terms of methods the paper borrows very heavily from existing works, namely DeepMimic and Learning Agile Robotic Locomotion Skills, including identical or nearly identical actions, controllers, reward formulations, reference data and simulator. It might be the case that there is a new approach proposed in combining [morphology and dynamics randomization] and [using history of states/actions with feedforward networks]. However I am not confident about that and would like the authors to verify if that’s accurate. Also a nice array of robots were examined (including 2 in the real world) but only 2 reference motions were evaluated.\n\nIn terms of results there seems to be a new result in showing that morphology and dynamics randomization can achieve reasonable out of distribution performance in simulation and importantly can achieve and improve sim to real transfer. This might point to a lesson that to achieve robustness in the real world, morphological randomization can be useful even if there are strong priors on the robot kinematic model. The authors should consider expanding on this.\n\nTherefore I feel there’s maybe an incremental methodological advance but a useful and practical result. The effort to validate on multiple real quadrupeds is also commendable.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Please address the questions in the Weaknesses section earlier.\n\nHere are some additional minor comments----\n\nRobot’s feet are modeled as spheres, and contacts are approximated as point contacts. How do we know these are sufficient approximations?\n\nWhy 15 states/actions? How was that number chosen?\n\nRefs 1 and 19 are repeated\n\n“Code and obtained policies”: what will the code entail exactly? How will it be useful to others?\nWhat orientation representation is used? (I assume q indicates quaternion) What difference metric is used? (I assume this is the standard quaternion difference)\n\nLine 159: q_base is 16 time steps, and q is 15 time steps. Is this intentional?\n\nPlease explain how the phase variable is determined. Is it just normalized time?\n\nLine 164: “explicitly” → “explicit”\n\nAppendix C: motor friction is in units of Nm? So it’s friction torque and not a coefficient?\n\nAppendix C: what is “motor strength”? What does this correspond to in the real world and what are the units of this parameter?\n\nLines 175-176: episode length and horizon length are the same?\n\nAppendix D: Units are millions of samples, or hundreds of millions of samples?\n\nAt test time are the joint PD gains equal to (nominal value * robot mass / nominal mass)?\n\nFigure 5: The red lines in the top row are only applicable to the GenLoco policies, right, since A1 didn’t have morphology randomization? If so you might add another vertical line there to show what the specific A1 parameters were. It could also be really useful to add vertical lines or markers to show what the parameters were for the other robots on these plots. You have some remaining space in the paper and you could make these plots bigger.\n\nVideo: It would be good to visualize the reference trajectories alongside the real robots, otherwise it’s hard to know whether it’s doing the right thing.\n\nVideo: The drifting behavior isn’t too clear to me. Consider showing a better video of that.\n\nVideo: What are the white boxes that appear?"}}, {"review_id": "bRcbwbTk1Cy", "reviewer": "Reviewer_62wF", "summary": "This paper proposes a method to train generalized controllers for quadrupedal robots. The generalization in this paper means that a single controller can be deployed on various quadruped robots having similar morphologies such as A1, Mini Cheetah, or Laikago. The key idea is a parameterization of simulated quadruped robots, where the parameters include physical properties (e.g., shape, inertia, joint frictions, etc) and its ranges are designed to cover the entire spectrum of the target quadruped robots. A locomotion controller is then learned by deep reinforcement learning, where a simulated character is randomly instantiated every episode with domain randomization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The formulation is simple and straightforward, but it works pretty well not only in the simulation but also in the real world, which is impressive. \n\n- The controller does not require any state predictions or explicit parameters for physical properties of the robot, which makes hardware deployment easy.\n\n- The learned algorithm was tested for the two real robots and it worked without any special treatment.\n\nWeaknesses\n\n- A learned controller can only perform a single behavior. Different behavior requires a different controller.\n\n- The claim on the benefit of the proposed curriculum learning might be wrong (please see below for the details)\n\n\nSuggestions for Improvement\n\n- The controller learned with morphology randomization showed better performance in general over the specialized controller for the A1 robot (Figure5). The most surprising/impressive part here for me was that this benefit is even true when testing with the A1 robot in the simulation only. Did morphology randomization help the Deep RL algorithm to find a better solution by preventing it from converging to suboptimal solution in the early learning phase? If this claim can be verified, the paper could become more impactful. I really want to see more discussion on this in the revised version.\n\n- Details for my comment above “The claim on the benefit of the proposed curriculum learning might be wrong (please see below for the details)”: The performance (returns) gap between the learning with/without curriculum in Figure7 is the main evaluation result supporting the claim. First, I don’t think that the initial gap is not meaningful because the challenging morphologies are not presented for the learning with curriculum. Second, there is nothing in the gap when looking at the gap around 200M where the two learning algorithms see the same morphology random distribution (the explanation in the paper: “The morphology randomization curriculum linearly anneals the range of parameter values from 0 to their respective maximum range over the course of 200 million samples”). This actually means that the curriculum is not helpful. Finally, it was mentioned that the authors collected 800M tuples for their final results, and the green curve (the algorithm without curriculum) is still increasing. So, the figure should be shown upto 800M.\n\n- In the state representation, why is the time window different for the base joint (t-15:t) and the remaining joints (t-15:t-1)?\n\n- How was the history of q_base, q, and actions set when t=0?\n\n- “The reference motions are retargeted to match our robot’s morphology” in Line197 -> Do you perform retargeting every episode whenever a quadruped robot is randomly generated?\n\n- Figure5 : It might be good to have a vertical line that matches the A1 robot.\n\n- It is unclear how the ANYmal-B could be directly controlled by the proposed method. Isn’t the kinematics tree structure a bit different from what the parameterized controller can cover? I guess there were some technical tricks for setting the zero pose. If it exists, please provide more details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presented that a controller learned in the simulation with morphology and domain randomization can directly be deployed in the real world, it could even be helpful in cases where we only develop a controller for a specific robot. I think this factor is impactful enough and could be a stepping stone for future research. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "ROUyB1qVvX8", "reviewer": "Reviewer_GntH", "summary": "The paper proposed Generalized Locomotion Controller (GenLoco) which can be deployed on a wide variety of quadrupedal robots to produce locomotion behaviors.\nThe main contributions of the paper are following two points:\n1. The authors developed a morphology randomization method to learn a generalized policy of GenLoco.\n2. The authors showed the network architecture and learning framework to design GenLoco.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The authors successfully demonstrated that the locomotion behaviors of a large collection of simulated quadrupedal robots could be generated by GenLoco.\n* The authors successfully demonstrated that the GenLoco policy, trained on the simulated robots, could be transferred to real quadrupedal robots A1 and Mini Cheetah. Surprisingly, the control performances of the robot-non-specific policy (GenLoco) are superior to those of the robot-specific policy. The potential impact of the paper is high.\n\nWeakness:\n* Although the authors mentioned that GenLoco could reduce the labor-intensive process of developing robot-specific controllers for new systems, it is difficult for me to imagine when such advantages are beneficial. The practical situations, applications or scenarios should be discussed, and GenLoco should be evaluated in such or similar situations. For example, in what kind of situations must robots with similar but different morphologies work together?\n\nMinor comments:\n* A typo: Our Models in Line 98.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contribution of this paper is sufficient: strength in both originalities and potential impact of the approach. The experiments are performed in not only simulation but also real environments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* I am curious whether the randomization degrades the best control performances or not. I would like to see the comparisons between the GenLoco policies and non-randomization policies (A1-specific policies without the dynamics randomization).\n* I would like to know the practical situations, applications or scenarios where the advantage of mitigating the overhead of creating individual robot-specific controllers is beneficial. GenLoco should also be evaluated in such or similar situations.\n"}}]

Author rebuttal:

Decision: Accept (poster)