Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
Authors: Nikita Rudin
David Hoeller
Philipp Reist
Marco Hutter
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Legged Robots
Sim-to-Real

Abstract: In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach.

Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
Nikita Rudin ETH Zurich and NVIDIA rudinnGethz.co ch
David Hoeller ETH Zurich and NVIDIA dhoeller@ethz.cl
Philipp Reist
NVIDIA
preistenvidia.com com
Marco Hutter ETH Zurich mahutter@ethz.ccmm com
Abstract: In this work, we present and study training set-up that achieves fast policy generation real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy perfor- mance and training times. In addition, lum that well suited for training with thousands simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal walk challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup multiple orders of magnitude compared to previous work. Finally, transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion: https: /ggg//eeggrrbbbbiissoooo github. haap/llegee_yymmm
present novel game-inspired curricu-
Keywords: Reinforcement Learning, Legged Robots, Sim-to-real
Figure 1: Thousands of robots learning to walk simulation.
Introduction
Deep reinforcement learning (DRL) is proving to be powerful tool for robotics. Tasks such as legged locomotion [1], manipulation [2], and navigation [3], have been solved using these new tools, and research continues to keep adding more and more challenging tasks to the list. The amount data required to train policy increases with the task complexity. For this reason, most work focuses training in simulation before transferring to real robots. We have reached point where multiple days or even weeks are needed to fully train an agent with current simulators. For example, OpenAl's block reorientation task was trained for up to 14 days and their Rubik's cube solving policy took several months to train [4]. The problem is exacerbated by the fact that deep
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

reinforcement learning requires syyper-araamter tuning to obtain a suitable solution which requires sequentially rerunning time-consuming training. Reducing training times using massively parallel approaches such as presented here can therefore help improve the quality and time-to-deploymen of DRI. policies, as a training setup can be iterated on more often in the same time frame.
In this paper, we examine the effects of massive parallelism for on-policy DRL algorithms and present considerations in how the standard RL formulation and the most commonly used hyper- parameters should be adapted to learn efficiently in the highly parallel regime. Additionally, we present novel game-inspired curriculum which automatically adapts the task difficulty to the per- formance of the policy. The proposed curriculum architecture is straightforward to implement, does not require tuning. and is well suited for the massively parallel regime. Common robotic simulators such as Mujoco [5]. Bullet [6]. or Raisim [7] feature efficient multi-body dynamics implementations. However, they have been developed to run on CPUs with only a reduced amount of parallelism. In this work, we use NVIDIA's Isaac Gym simulation environment [8], which runs both the simulation and training on the GPU and is capable of simulating thousands of robots in parallel.
The massively parallel training regime has been explored before [4, 9] in the context of distributed systems with a network of thousands of CPUs each running a separate instance of the simulation. The parallelization was achieved by averaging the gradients between the different workers without reducing the number of samples provided by each agent. This results in large batch sizes of millions of samples for each policy update which improves the learning dynamics, but does not optimize the overall training time. In parallel, recent works have aimed to increase the simulation through- put and reduce training times of standard DRL benchmark tasks. A framework combining parallel simulation with multi-GPU training [10] was proposed to achieve fast training using hundreds of parallel agents. In the context of visual navigation, large batch simulation has been used to increase the training throughput [11]. Furthermore, GPU accelerated physics simulation has been shown to significantly improve the training time of the Humanoid running task [12]. A differentiable simu- lator running on Google's TPUs has also been shown to greatly accelerate the training of multiple tasks [13]. We build upon [10. 12] by pushing the parallelization further, optimizing the training algorithm, and applying the approach to a challenging wooldworld robotics task.
Perceptive and dynamic locomotion for legged robots in unstructured environments is demanding task that, until recently, had only been partially demonstrated with complex model-basee approaches [14, 15]. Learning- baarinng-based approaches are emerging as a promising alternative. For quadrupeds, DRL, has been used to train blind policies robust to highly uneven ground [16] (12 hours of training). Per- ceptive locomotion over challenging terrain has been achieved by combining learning with optimal control techniques [17, 18] (82 and 88 hours of training) and recently, a fully learned approach has shown great robustness in this setting [19] (120 hours of training). Similarly, bipedal robots have also been trained to walk blindly on stairs [20] (training time not reported). With our approach we single GPU, with the complexity of sim- to-real transfer to the hardware, which increases the performance and robustness requirements and provides clear validation of the overall approach. Training such behaviors in minutes opens up new exciting possibilities ranging from automatic tuning to customized training using scans of particular environments.
can train a perceptive policy in under 20 minutes
2 Massively Parallel Reinforcement Learning
Current (on-policy) reinforcement learning algorithms are divided into two parts: data collection and policy update. The policy update, which corresponds to back-propagation for neural networks, is casily performed in parallel on the GPU. Parallelizing data collection is not as straightforward. Each step consists of policy inference, simulation, reward, and observation calculation. Current popular pipelines have the simulation and reward/observation calculation computed on the CPU, making the GPU unsuitable for policy inference because of communication bottle-necks. Data transfer over PCIe is known to be the weakest link of GPU acceleration, and can be as much as 50 times slower than the GPU processing time alone [21]. Furthermore, with CPU data collection, a large amount of data must be sent to the GPU for each policy update, slowing down the overall process. Limited parallelization can be achieved by using multiple CPU cores and spawning many processes, each running the simulation for one agent. However, the number of agents is quickly limited by the num- ber of cores and other issues such as memory usage. We explore the potential of massive parallelism
2

---

with Isaac Gym's 's end-to-end data collection and policy updates on the GPU, significantly reducing data copying and improving simulation throughput.
2.1 Simulation Throughput
The main factor affecting the total simulation throughput is the number of robots simulated in par- allel. Modern GPUs can handle tens of thousands of parallel instructions. Similarly, IsaacGym's 's PhysX engine can process thousands of robots in single simulation and all other computations of our pipeline are vectorized to scale favorably with the number of robots. Using a single simulation with thousands of robots presents some new challenges. For example, single common terrain mesh must be used, and it cannot be casily changed at each reset. We circumvent this problem by creating the whole mesh with all terrain types and levels tiled side by side. We change the terrain level of the robots by physically moving them on the mesh. In supplementary material, we show the com- putational time of different parts of the pipeline, examine how these times scale with the number of robots, and provide other techniques to optimize the simulation throughput.
2.2 DRL Algorithm
We build upon custom implementation of the Proximal Policy Optimization (PPO) algorithm [22]. Our implementation is designed to perform every operation and store all the data on the GPU. In or- der to efficiently learn from thousands of robots in parallel, we perform some essential modifications to the algorithm and change some of the commonly used hyper-parameter values.
2.2.1 Hyper-Parameters Modification
In an on on-policy algorithm such as PPO, a fixed policy collects a selected amount of data before do- ing the next policy update. This batch size, B, is a crucial hyper-paramett parameter for successful learning. With too little data, the gradients will too noisy, and the algorithm will not learn effectively. With too much data, the samples become repetitive, and the algorithm cannot extract more information from them. These samples represent wasted simulation time and slow down the overall training. We have B RasepessRatepps where Rateps is the number of steps each robot takes per policy update and Ryohote the number of robots simulated in parallel. Since we increase Nyybots by a few orders of magnitude, we must choose a small Nateps to keep B reasonable and hence optimize training times, which is a setting that has not been extensively explored for on-policy reinforcement learning algorithms. It turns out that we can not choose Nyteps to be arbitrarily low. The algorithm requires trajectories with coherent temporal information to learn effectively. Even though, in theory, informa- tion of single steps could be used, we find that the algorithm fails to converge to the optimal solution below a certain threshold. This can be explained by the fact that we use Generalized Advantage Estimation (GAE) [23], which requires rewards from multiple time steps be effective. For task, we find that the algorithm struggles when we provide fewer than 25 consecutive steps, corre- sponding to 0.5 s of simulated time. is important distinguish stteps from the maximum episode length leading to a time-out and reset, which we define as 20 The environments are reset when they reach this maximum length and not after each iteration, meaning that a single episode can cover many policy updates. This limits the total number of robots training in parallel, and consequently, prohibits us from using the full computational capabilities of the GPU.
The mini-bate batch size represents the size of the chunks in which the batch size is split to perform back- propagation. We find that having mini-batch sizes much larger than what is usually considered best practice is beneficial for our massively parallel use case. We use mini-batches of tens of thousands of samples and observe that it stabilizes the learning process without increasing the total training time.
2.2.2 Reset Handling
During training. the robots must be reset whenever they fall, and also after some time to keep them exploring new trajectories and terrains. The PPO algorithm includes a critic predicting an infinite horizon sum of future discounted rewards. Resets break this infinite horizon assumption and can lead to inferior critic performance if not handled carefully. Resets based on failure or reaching goal are not a problem because the critic can predict them. However, a reset based on a time out can not be predicted (we do not provide episode time in the observations). The solution is to distinguish
3

---

(a)
(c)
(d)
Figure 2: Terrain types used for training and testing in simulation. (a) Randomly rough terrain with variations of 0.1 m. (b) Sloped terrain with an inclination of 25 deg. (c) Stairs with a width of 0.3 m and height of 0.2 (d) Randomized, discrete obstacles with heights of up +0.2 m.
the two termination modes and augment the reward with the expected infinite sum of discounted future rewards in a time-out case. In other words, bootstrap the target of the critic with its own prediction. This solution has been discussed in [24]. but interestingly. this distinction is not part of the widely used Gym environment interface [25] and ignored by popular implementations such as Stable-Bast Baselines [26]'. After investigating multiple implementations, we conclude that this important detail is often avoided by assuming that the environments either never time out only on the very last step of a batch collection. In our case, with few robot steps per batch, we can not make such an assumption since meaningful episode length covers the collection of many batches. We modify the standard Gym interface to detect time-outs and implement the bootstrapping solution. In supplementary material, we show the effect of this solution on the total reward as well as the critic loss.
3 Task Description
A quadruped robot must learn to walk across challenging terrain, including uneven surfaces, slopes, stairs, and obstacles, while following base-heading and linear-velocity commands. We conduct most of the simulation and real-world deployment experiments on the ANYbotics ANYmal C robot. However, in simulation, we demonstrate the broader applicability of the approach by additionally training policies for ANYmal B, ANYmal C with attached arm, and the Unitree A1 robots.
3.1 Game-Inspired Curriculum
The terrains are selected to be representative of real-world environments. We create five types of procedurally generated terrains presented Fig. 2: flat, sloped, randomly rough, discrete obstacles, and stairs. The terrains are tiled squares with 8m sides. The robots start at the center of the terrain and are given randomized heading and velocity commands (kept constant for the duration of an episode) pushing them walk across the terrain. Slopes and stairs organized in pyramids to allow traversability in all directions.
Previous works have shown the benefits of using an automated curriculum of task difficulty to learn complex locomotion policies [28, 29, 16). Similarly, we find that it is essential to first train the pol- icy on less challenging terrain before progressively increasing the complexity. We adopt solution inspired by [16], but replace the particle filter approach with new game-inspired automatic curricu- lum. All robots are assigned a terrain type and level that represents the difficulty of that terrain. For stairs and randomized obstacles, we gradually increase the step height from 5 cm to 20 cm. Sloped terrain inclination is increased from 0 deg to 25 deg. If robot manages to walk past the borders of its terrain, its level is increased, and at the next reset, it will start on more difficult terrain. However, if the end of an episode it moved by less than half of the distance required by its target velocity, its level reduced again. Robots solving the highest level are looped back to a randomly selected level to increase the diversity and avoid catastrophic forgetting. This approach has the advantage of training the robots at a level of difficulty tailored to their performance without requiring any external tuning. It adapts the difficulty level for each terrain type individually and provides with visual and quantitative feedback on the progress of the training. When the robots have reached the final level and are evenly spread across all terrains due looping back, we can conclude they have fully learned to solve the task.
'The Spinning up [27] implementation of PPO uses the same bootstrapping solution by keeping track of episode lengths within the algorithm, thus circumventing the limitation of the Gym interface.

---

Figure 3: 4000 robots progressing through the terrains with automatic curriculum, after 500 (top) and 1000 (bottom) policy updates. The robots start the training session on the first row (closest to the camera) and progressively reach harder terrains.
The proposed curriculum structure is well suited for the massively parallel regime. With thousands of robots we can directly use their current progress in the curriculum as the distribution of the policy's 's performance, and do not need learn it with a generator network [30]. Furthermore, our method doesn't 't require tuning and is straightforward to implement in parallel manner with near- zero processing cost. We remove the computational overhead of re-sampling and re-generating new terrains needed for the particle filter approach.
Fig. shows robots progressing through the terrains two different stages of the training process. On complex terrain types, the robots require more training iterations to reach the highest levels. The distribution of robots after 500 iterations shows that while the policy is able to and to go down stairs, climbing stairs and traversing obstacles requires more training iterations. However, after 1000 iterations, the robots have reached the most challenging level for all terrain types and are spread across the map. We train for total for 1500 iterations to let the policy converge its highest performance.
sloped terrains
3.2 Observations, Actions, and Rewards
The policy receives proprioceptive measurements of the robot as well as terrain information around the robot's base. The observations composed of: base linear and angular velocities, measurement of the gravity vector, joint positions and velocities, the previous actions selected by the policy, and finally, 108 measurements of the terrain sampled from a grid around the robot's base. Each measurement is the distance from the terrain surface to the robot's 's base height.
The total reward is weighted terms encourage the robot to follow the commanded velocities while avoiding undesired base ve- locities along other axes. In order to create smoother, torques, joint accelerations, joint target changes, and collisions. Contacts with the knees, shanks or between the feet and vertical surface are considered collisions, while contacts with the base considered crashes and lead to resets. Finally, we add an additional reward term encouraging the robot take longer steps, which results in policy with the same rewards for all terrains.
of nine terms, detailed in supplementary material. The main
natural motion, also penalize joint
visually appealing behavior. We train single
The actions are interpreted desired joint positions sent to the motors. There, PD controller produces motor torques. In contrast to other works [16, 20], neither the reward function nor the action space has any gait-dependent elements.
3.3 Sim-to-Real Additions
In order to make the trained policies amenable for sim-to-real transfer, we randomize the friction of the ground, add noise to the observations and randomly push the robots during the episode to
5

---

teach them more stable stance. Each robot has friction coefficient sampled uniformly in [0.5, 1.25]. The pushes happen every s0 The robots' base accelerated up =1 m/s in both x and y directions. The amount of noise is based on real data measured on the robot and is detailed in supplementary material.
The ANYmal robot uses series elastic actuators with fairly complex dynamics, which are hard to model simulation. For this reason and following the methodology of previous work [1], we use neural network to compute torques from joint position commands. However, we simplify the inputs of the model. Instead of concatenating past measurements at fixed time steps and sending all of that information to a standard feed-forwart network, we only provide the current measurements to an LSTM network. A potential drawback of this set-up is that the policy does not have the temporal information of the actuators as in previous work. We have experimented with various ways of providing that information through memory mechanisms for the policy but found that does not improve the final performance.
4 Results
4.1
Effects of Massive Parallelism
In this section, we study the effects of the number of parallel robots on the final performance of the policy. In order to use the total reward single representative metric, we have to remove the curriculum, otherwise a more performant policy sees its task difficulty increase and consequently a decrease in the total reward. As such, we simplify the task by reducing the maximum step size of stairs and obstacles and directly train robots on the full range of difficulties.
We begin by setting a baseline with Trabots 20000 and stteps 50, resulting in batch size of 1M samples. Using this very large batch size results in the best policy but at the cost of a relatively long training time.
We then conduct experiments in which we increase the number of robots while keeping the batch size constant. As a result, the number of steps each robot takes per policy update decreases. In this case, the training time decreases with higher number robots, but the policy performance drops if that number is too high. We start from 128 robots corresponding to the level of parallelization of previous CPU implementations and increase that number up 16384, which close to the maximum amount of robots we could simulate on rough terrain with Isaac Gym running on single workstation GPU.
In Fig. 4, we compare these results with the baseline, which allows us to select the most favorable trade-off between policy performance and training time. We see two interesting effects at play. First, when the number of robots is too high, the performance drops sharply. which can be explained by the time horizon of each robot becoming too small. As expected, with larger batch sizes, the overall reward is higher, and the time horizon effect is shifted, meaning that we can use more robots before seeing the drop. On the other hand, below certain threshold, we see a slow decrease in
17.5
35
15.0
15
Rinal rewerd
2h-
10
trainingg
10.0
1h
7.5
Batch sibe: 49152
30min
128
2048
20min
256
Batch size: 136508
25-
512
8192
Baseline uppar bound
10min
1024
16384
128 256 512 1024 2048 4095 819216384
128 256 512 1024 2048 4096 819215384
101
10a
Number robuts
Number
Training time [min)
Figure 4: (a) Average and standard deviation (over 5 runs) of the total reward of an episode after 1500 policy updates for different number of robots and 3 different batch sizes. The ideal case of a batch size of M samples with 20000 robots is shown in red. (b) Total training time for the same experiments. (c) Reward dependency on total training time. Colors represent the number robots, while shapes show the batch size (circles: 49152, crosses: 98304, triangles: 196608). Points in the upper left part of the graph (highlighted in green) represent the most desirable configuration.

---

100
100
Success (S6
00-
Success rate (16
40
obstacles Stairs lupt
40
20
20
Up
Stains (down)
Down
0.05
0.10
0.15
0.25
0.30
15
25
height [m]
Slope angle
(a)
(b)
Figure 5: Success rate of the tested policy on increasing terrain complexities. Robots start in the center of the terrain and are given forward velocity command of 0.75 m/s, and a side velocity command randomized within 1-0.1, 0.1] m/s. (a) Success rate for climbing stairs, descending stairs and traversing discrete obstacles. (b) Success rate for climbing and descending sloped terrains.
Figure 6: ANYmal C with a fixed arm, ANYmal B, Al and Cassie in simulation.
performance with fewer robots. We believe this is explained by the fact that the samples are very similar with many steps per robot because of the relatively small time steps between them. This means that for the same amount of samples, there is less diversity in the data. In other words, with low number of robots, we are further from the standard assumption that the samples are independent and identically distributed, which seems to have noticeable effect the training process. In terms of training time, we see a nearly linear scaling up to 4000 robots, after which simulation throughput gains slow down. As such, both final performance and training time, but there is an upper limit on this number after which an on-policy algorithm cannot learn effectively. Increasing the batch size to values much larger than what is typically used in similar works seems highly beneficial. Unfortunately. also scales the training time so it is trade-off that must be balanced. From the third plot using 2048 to 4096 robots with a batch size of z 100k or == 200k provides the best trade-off for this specific task.
conclude that increasing the number of robots is beneficial for
conclude that
4.2 Simulation
For our simulation and deployment experiments, we use a policy trained with 4096 robots and batch size of 98304, which we train for 1500 policy updates under 20 minutes?. We begin by measuring the performance of our trained policy in simulation. To that end, we perform robustness and traversability tests. For each terrain type, we command the robots traverse the representative difficulty of the terrain at high forward velocity and measure the success rate. A success is defined as managing to cross the terrain while avoiding any contacts on the robot's base. Fig. shows the results for the different terrains. For stairs, we see nearly 100 % success rate for steps up to m.2
m, which is the hardest stair difficulty we train on and close to the kinematic limits of our robot. Randomized obstacles seem to be more demanding, with the success rate decreasing steadily. We must note that in this case, the largest step is double the reported height since neighboring obstacles can have positive and negative heights. In the case of slopes, we can observe that after 25 deg the robots are not able to climb anymore but still learn to slide down with a moderate success rate.
Given our relatively simple rewards and action space, the policy is free to adopt any gait and behav- ior. Interestingly, always converges to a trotting gait, but there are often artifacts the behavior, such as dragging leg unreasonably high or low base heights. After tuning of the reward weights, we can obtain policy that respects our constraints and can be transferred the physical robot.
To verify the generalizability of the approach, we train policies for multiple robots with the same set-up. We use the ANYmal C robot with a fixed robotic arm, which adds about 20% % of additional
'Trained on: 19-199999 1190kk CPU, NVIDIA RTX A6000 GPU. VRAM requirements are in the supplementary material.

---

Figure 7: Locomotion policy, trained in under 20min, deployed on the physical robot.
weight, and the ANYmal B robot, which has comparable dimensions but modified kinematic and dynamic properties. In these two cases, we can retrain policy without any modifications to the rewards or algorithm hyper-parameters and obtain a very similar performance. Next, we use the Unitree Al robot, which has smaller dimensions, four times lower weight, and different leg figuration. In this case, we remove the actuator model of the ANYdrive motors, reduce PD gains and the torque penalties, and change the default joint configurations. We train a dynamic policy that learns to solve the same terrains even with the reduced size of the robot. Finally, we apply our approach to Agility Robotics' bipedal robot Cassie. We find that additional reward encouraging standing on single foot is necessary to achieve a walking gait. With this addition, we are able to train the robot the same terrains its quadrupedal counterparts. Fig. 6 shows the different robots.
4.3 Sim-to-real Transfer
On the physical robot, our policy is fixed. We compute the observations from the robot's sensors, feed them to the policy, and directly send the produced actions as target joint positions to the mo- tors. We do not apply any additional filtering constraint satisfaction checks. The terrain height measurements are queried from an elevation map that the robot is building from Lidar scans.
Unfortunately, this height map is far from perfect, which results in a decrease in robustness between simulation and reality. We observe that these issues mainly occur at high velocities and therefore reduce the maximum linear velocity commands to 0.6 m/s for policies deployed on the hardware. The robot can walk up and down stairs and handles obstacles in dynamic manner. We show samples of these experiments in Fig. 7 and in the supplementary video. To overcome issues with imperfect terrain mapping or state estimation drift, the authors of [19] implemented teacher-student set-up, which provided outstanding robustness even in adverse conditions. As part of future work, we plan to merge the two approaches.
Conclusion
In this work, we demonstrated that a complex real-world roboties task can be trained in minutes with an on-policy deep reinforcement learning algorithm. Using an end-to-end GPU pipeline with thou- sands of robots simulated in parallel, combined with our proposed curriculum structure, we showed that the training time can be reduced by multiple orders of magnitude compared to previous work. We discussed multiple modifications the learning algorithm and the standard required to use the massively parallel regime effectively. Using our fast training pipeline, we per- formed many training runs, simplified the set-up, and kept only essential components. We showed that the task can be solved using simple observation and action spaces as well as relatively straight- forward rewards without encouraging particular gaits or providing motion primitives.
The purpose of this work is not to obtain the absolute best-performing policy with the highest ro- bustness. For that use case, many other techniques can be incorporated into the pipeline. We aim to show that a policy can be trained in record time with our set-up while still being usable on the real hardware. We wish shift other researchers' perspective on the required training time for real-world application, and hope that our work can serve as a reference for future research. We ex- pect many other tasks to benefit from the massively parallel regime. By reducing the training time of these future robotic tasks, we can greatly accelerate the developments in this field.

---

Acknowledgments
We would like to thank Mayank Mittal, Joonho Lee, Takahiro Miki, and Peter Werner for their valuable suggestions and help with hardware experiments as well the Isaac Gym and PhysX teams for their continuous support.
References
[1] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[2] S. Gu, E. Holly, T. Lillicrap. and S. Levine. Deep reinforcement learning for robotic manip- ulation with asynchronous off- off-ppoicy updates. JEEE International Conference on Robotics and Automation (ICRA), May 2017.
[3] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-supervieed deep reinforcement learning with generalized computation graphs for robot navigation. In IEEE International Conference Robotics and Automation (ICRA), 2018.
[4] OpenAI, I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's 's cube with robot hand, 2019.
[5] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
[6] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http: http://pybullet.. org, 2016-2021.
[7] J. Hwangbo, J. Lee, and M. Hutter. Per-contact iteration method for solving contact dynamics. IEEE Robotics and Automation Letters, 3(2), 2018. URL www rwwsim.immm................m..m. com.
[8] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU based physics simu- lation for robot learning. Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
[9] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, M. A. Eslami, M. A. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments, CoRR, abs/ fos//7777.0226, 2017.
[10] A. Stooke and P. Abbeel. Accelerated methods for deep reinforcement learning. CoRR, abs/ htt/1/83.0011110 2018.
[11] B. Shacklett, E. Wijmans, A. Petrenko, M. Savva, D. Batra, Koltun, and K. Fatahalian. Large batch simulation for deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[12] J. Liang, Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning (CoRL), 2018
[13] C. D. Freeman, Frey, A. Raichuk, S. Girgin, Mordatch, and Bachem. Brax a differ- entiable physics engine for large scale rigid body simulation. In 35th Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
[14] A. Bouman, M. Ginting, N. Alatur, M. Palieri, D. D. Fan, T. Touma, Pailevanian, S.- Autonomous spot: Long-range
K. Kim, K. Otsu, J. Burdick, and A.-a. Agha-Mohammade autonomous exploration of extreme environments with legged locomotion. In JEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), 2020.
[15] C. Gehring, P. Fankhauser. L. Isler, R. Diethelm, S. Bachmann, M. Potz, L. Gerstenberg, and M. Hutter. Anymal in the field: Solving industrial inspection of an offshore hvde platform with a quadrupedal robot. In Field and Service Robotics, 2021.

---

[16] J. Lee, J. Hwangbo. L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science Robotics, 5(47), 2020.
[17] V. Tsounis, M. Alge, Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning, IEEE Robolics and Automation Letters, PP, 03 2020.
[18] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In IEEE Interna- tional Conference on Robotics and Automation (ICRA), 2021.
[19] T. Miki, J. Lee, L. Wellhausen, V. Koltun, and M. Hutter. Wild anymal: Robust zero-shot perceptive locomotion. Submitted to Science Rebotics, 2021.
[20] Siekmann, K. Green, J. Warila, A. Fern, and J. W. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. CoRR, foos/200508322 2021.
[21] C. Gregg and K. Hazelwood. Where is the data? why you cannot debate epu gpu per- formance without the answer. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2011.
[22] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, 0es//777.063344 2017.
[23] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Confer- ence on Learning Representations (ICLR), 2016.
[24] F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev. Time limits in reinforcement learning. CoRR, abs/1712.00378,, 2017.
[25] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[26] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu. Stable baselines. https:// /ttpt/b.tttt.tt///// www.ih......hhllloeoe........e.e
2018.
[27] Achiam. Spinning up in deep reinforcement learning, 2018. URL http://sppimmngg openai. www./o//aan/t/////////mmm
[28] R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (POET): end- lessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1991.01175 2019.
[29] Z. Xie, H. Y. Ling, N. H. Kim, and M. van de Panne. Allsteps: Curriculum-driven learn- ing of stepping stone skills. Proceedings of ACM SIGGRAPH Eurographics Symposium on Computer Animation, 2020.
[30] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, 2018
10


Reviews:
[{"review_id": "vKsAeob-H3b", "reviewer": "Reviewer_cSHN", "summary": "The paper demonstrates that it is possible to use a GPU-based physics simulator to run thousands of simulations in parallel, which can greatly accelerate the DRL training of quadruped locomotion. Batch size hyper-parameters are tuned to adapt to this setting, and a curriculum of \"different levels\" with gradually increasing terrain difficulty is designed. The learned policy can also be deployed on a real robot, showing its robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's main strength is the quality of results -- as the authors claimed, it was not previously possible to train quadruped locomotion on a single workstation within minutes. The presentation of the material is also pretty clear.\n\nThere are two apparent weaknesses for papers like this. 1. lack of technical novelty and 2. lack of open-source implementation. The reviewer would especially like to emphasize the importance of 2 -- CoRL has long been advocating for open-sourcing; if the authors do want to publish their findings at CoRL, they should seriously consider packaging and releasing their implementation to help real advance of the community.\n\nRelated work (GPU for massively parallel DRL, with open-source code: https://graphics.stanford.edu/projects/bps3D/)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The results of the paper look impressive. The main reason for \"weak\" accept is that without open-sourced code, it is difficult to validate how fast the implementation really is.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "As discussed, code release will make this paper more consistent with the spirit of CoRL.\n\nQuestion:\nIs it possible to reset simulation to previously visited states, rather than always close to the starting pose? Seems like if so we can worry less about number of consecutive steps being too small?"}}, {"review_id": "i2fAPWoPZX", "reviewer": "Reviewer_2XwF", "summary": "This paper presents an experimental setup for learning locomotion controllers for  legged robots, with an emphasis on robotic quadrupeds. The system consists of a simulation pipeline on which multiple robot simulations can be run in parallel, along with a policy optimization algorithm that combines data from all simulations. The proposed simulation environment is shared by multiple instances of simulated robots. The environment is divided into tiles with different terrains, which pose challenges of varying difficulties to a walking robot. This setup is used to design a curriculum where robots are assigned a tile, a desired heading and desired velocity, and are promoted/demoted to harder/easier environments depending on whether they exit the tile at the desired heading. The paper shows how this setup can be used to train walking controllers with online policy search (PPO) on a single GPU (with 48gb of VRAM)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presentation and the engineering work done to produce the results in this paper are very itneresting. The proposed approach of large scale data parallelism on a single GPU is definitely relevant for robotics research.\n\nIt is not clear whether the technique for reset handling described in section 2.2.2 is doing anything different from existing implementations of PPO. Reset handling as described in this section is already present in other implementations of PPO (e.g. in openai baselines and the spinningup repositories), which use the standard gym interface.\n\nSince the paper makes an emphasis on training on a single GPU, it would be very useful to have an evaluation in terms of available GPU RAM given that not many practitioners have access to the GPU used in this work. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a neat demonstration of how existing algorithms for policy search can benefit from scaling up data collection in simulation. This paper describes the engineering decision taken to enable training on a single GPU. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Some references to related work on curriculum learning for locomotion [1] and [2], and on adapting PPO to parallel simulation [3].\n\n[1] Paired Open-Ended Trailblazer (POET):\n[2] ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills\n[3] Accelerated Methods for Deep Reinforcement Learning"}}, {"review_id": "GHMVHRUdDL", "reviewer": "Reviewer_d4o6", "summary": "The paper explores learning locomotion policies for quadruped robots using massive parallel simulation. The main technical contributions of the paper is an adapted setting of the PPO algorithm for massive parallel simulations, as well as a curriculum learning scheme for training the policies to tackle complex environments. They also demonstrated that the resulting control policy can be successfully deployed to the real robot.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe main strength of the paper is that it demonstrates the possibility of leveraging large-scale simulations to learn legged locomotion for a real robot in a very short amount of time. The modifications to the standard PPO setup in response to the issues arising from the massive parallelization is reasonable and supported by their experiments. The proposed curriculum scheme also makes sense.\n\n\nWeaknesses:\n1. Some details in the paper are not clear. For example, the paper mentioned that the episode length for each simulation is short (<5s) due to the large number of robots being simulated. At the end of the episode, is the robot reset to the initial state? The paper also mentioned that a random push force is applied every 10s to the robot. How is this 10s computed since it will span multiple episodes?\n2. In [10], the GPU-based simulation was also used to learn several robotic tasks in a short amount of time, similar to the presented work. It’s not very clear what’s the main difference between the two works in terms of large-scale training. For example, if one takes the learning algorithm from [10], applies the curriculum learning scheme and reward function in this work, what would the result be like?\n3. The proposed method converged to the same locomotion gait (trotting) for all three robots tested. This is good for real robot transfer given the stability of trotting gaits, meanwhile the lack of diverse behaviors in the robots is a bit unsatisfying. It would be nice if more locomotion forms can be tested such as bipedal walking.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an interesting direction in learning robot locomotion controllers with massively parallelized simulations. The result is solid in general with real robot validations and can inspire further research in this direction. The several technical details introduced in the paper to help improve training performance is also reasonable and effective. However, there are a few issues with the paper such as missing details, limited discussion with relevant work, and limited diversity in the learned behaviors. It would be a good contribution to CoRL if those can be addressed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "As mentioned in the weakness part, I think there are three issues I’d like to see more discussions/experiments:\n\n1. How is the perturbation applied to the robot during training? Is the robot reset if the episode ends due to time-out?\n2. Add discussion on how the work differs from [10] and ideally experiments to show that the delta in the learning algorithm makes a significant difference.\n3. It would be great to have more diverse behaviors emerge from this nice framework, for example biped locomotion or manipulation.\n"}}, {"review_id": "-MlDWNjMHw", "reviewer": "Reviewer_Y3ip", "summary": "Main Contributions\n================\nA new training setup for model-free reinforcement learning that allows training policies for quadruped robots very efficiently.\nAn ablation study that investigates the impact of batch size and number of robots simulated in parallel on final performance.\n\nSummary\n========\nThe paper demonstrates a learning strategy for quadruped locomotion that is efficiently implemented entirely on the GPU and allows massive parallelization during training time. \nCombined with a game-inspired curriculum learning, the policies successfully learn to navigate complex terrain and also transfer to a real robot. \nIn the experimental part, the paper presents an ablation study of batch size and number of robots on the final performance and training time required. \n\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n- Nice video!\n- The paper is well written. \n- This training methodology would be very useful if made open-source, as it might be generalized to other robotic platforms as well. \n\nWeaknesses\n===========\n- The paper contains very limited scientific novelty. Previous works [1, 2] have already demonstrated the feasibility of training a navigation policy for a quadruped robot in simulation and deployment in the real world. \n- The game-inspired curriculum is the main novelty of this work, but is not compared to the previous approach used for training (particle filter approach proposed in [1]).\n\n\n[1] Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. and Hutter, M., 2020. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47).\n\n[2] Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V. and Hutter, M., 2019. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The usability of this work is highly dependent on the fact if the code will be available open-source. Due to the very limited scientific novelty (i.e. the game-inspired curriculum), the main purpose of this paper would be to publicize a new highly parallelizable training framework. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The paper would be stronger if the game-inspired curriculum would be compared to the previous, particle-filter based approach of curriculum learning. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the acceleration issue on the DRL training for quadruped locomotion, and presents a training set-up by using massive parallelism on a single GPU workstation, and a novel game-inspired curriculum well suited for parallel training.  

I agree with reviewers that this paper is well organized and the experimental results is quite promising, while the scientific novelty seems to be limited, reviewers also concerns on the lack of open-source details.  

Reviewers also pointed out several other concerns or deficiencies about the paper. I agree with these comments which might be valuable in improving the quality of the paper, mainly including:

1. Comparison or discussion to some related works, both on quadruped robot locomotion and the game-inspired curriculum learning.
2. More expeirmental details to make clear some points reviewers raised, and more locomotion forms like bipedal walking tested if possible to make the paper more solid.
3. Other minor points as reviewers mentioned.

More added: 
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers.
Although there are still concerns on the limited scientific novelty, the scores of this paper was improved after rebuttle. 
I thus recommend this paper to be accepted.

Decision: Accept (Poster)