Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion
Authors: Michael Bloesch
Jan Humplik
Viorica Patraucean
Roland Hafner
Tuomas Haarnoja
Arunkumar Byravan
Noah Yamamoto Siegel
Saran Tunyasuvunakool
Federico Casarini
Nathan Batchelor
Francesco Romano
Stefano Saliceti
Martin Riedmiller
S. M. Ali Eslami
Nicolas Heess
Venue: CoRL2021 Poster (2021)
Keywords: Legged Locomotion
Reinforcement Learning
Vision

Abstract: Algorithms for self-learning systems have made considerable progress in recent years, yet safety concerns and the need for additional instrumentation have so far largely limited learning experiments with real robots to well controlled lab settings. In this paper, we demonstrate how a small bipedal robot can autonomously learn to walk with minimal human intervention and with minimal instrumentation of the environment. We employ data-efficient off-policy deep reinforcement learning to learn to walk end-to-end, directly on hardware, using rewards that are computed exclusively from proprioceptive sensing. To allow the robot to autonomously adapt its behaviour to its environment, we additionally provide the agent with raw RGB camera images as input. By deploying two robots in different geographic locations while sharing data in a distributed learning setup, we achieve higher throughput and greater diversity of the training data.
Our learning experiments constitute a step towards the long-term vision of learning "in the wild" for legged robots, and, to our knowledge, represent the first demonstration of learning a deep neural network controller for bipedal locomotion directly on hardware.

Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion
Michael Bloesch*, Jan Humplik*, Viorica Patraucean*, Roland Hafner*, Tuomas Haarnoja, Arunkumar Byravan, Noah Siegel, Saran Tunyasuvunakool, Federico Casarini, Nathan Batchelor, Francesco Romano, Stefano Saliceti, Martin Riedmiller, Ali Eslami, Nicolas Heess
DeepMind
[bloesch, bbloesch,jhumllik, viorica rhafner, tuomash, abyravan, bbiyravansseeellln siegeln, stunya, ff fasariini batchelor, faahhelorrfraroooo fraromano, ssalicetis ri 11er, aes amii, heess @deepmind. com *These authors contributed equally
Algorithms for self-learning systems have made considerable progress in recent years, yet safety concerns and the need for additional instrumentation have so far largely limited learning experiments with real robots to well con- trolled lab settings. In this paper, we demonstrate how small bipedal robot autonomously learn to walk with minimal human intervention and with minimal instrumentation of the environment. We employ data-efficient off off-policy deep reinforcement learning to learn to walk end-to-end, directly on hardware, using rewards that are computed exclusively from proprioceptive sensing. To allow the robot to autonomously adapt its behaviour to its environment, we additionally pro- vide the agent with raw RGB camera images as input. By deploying two robots different geographic locations while sharing data in a distributed learning setup, we achieve higher throughput and greater diversity of the training data. Our learn- ing experiments constitute step towards the long-term vision of learning "in the wild" for legged robots, and, to our knowledge, represent the first demonstration of learning a deep neural network controller for bipedal locomotion directly on hardware.
Abstract:
Keywords: Legged Locomotion, Reinforcement Learning, Vision
1 Introduction
Enabling robots to learn and adapt completely autonomously only through interaction with their environment is one of the long-standing goals of Al and robotics. The idea that robots might one day learn and cohabit outside controlled laboratory environments, taking full advantage of the richness of the real world, is appealing especially for mobile robots. However, even though algorithms for self-learning systems have made considcrable progress in recent years in terms data efficiency and robustness (e.g. [1, 2I), concerns around safety and the need for instrumentation (for instance, for reward computation or resets) have so far largely limited experiments with real robots to controlled lab settings [3, 4] or sim-to-real transfer [5, 6, 7].
While transfer of behaviours from simulation to real can sidestep the challenges around data effi- ciency and safety, it requires significant engineering effort in the design of high fidelity simulations that represent not only the correct dynamics of the robot but also offer increasingly complex and rich scenarios. More importantly, learning only in simulation exposes the agent to the sim-to-real gap, leaving questions around adaptation to unfamiliar situations after deployment.
Despite first successful applications (e.g. [8]) learning "in the wild" directly on hardware is often thought of as an infeasible method for real legged robots. In this paper, we show that using state- of-the-art model-free deep reinforcement learning (RL) and intrinsically defined rewards, we can learn walk and avoid walls autonomously in environments with minimal instrumentation. In particular, we show how small humanoid robots can learn to walk and interact with their environment learning scenario outside of the laboratory, relying only on on-board sensors and limited prior knowledge of the hardware.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

1=0.0s 0s
t=1. 1,0s
1:2.0s
1=3.0s
1=4.0s
No training
Falls down
After 3h
Does fall
After Walks
After 32h and sieers away from walls
Two robots are trained simultancously in different locations depicted on the right. They are sharing the same training data. On the left we show one of the robots and the approximate experiment time it takes for the policy to reach important milestones: maintaining upright position, walking, and eventually walking and avoiding walls. We define experiment time as the total time during which least on the robots was on.
Figure 1:
The robot uses proprioceptive sensing consisting of inertial mcasurement unit (IMU) and joint position encoders together with egocentric RGB vision to perceive its own body and the environ- ment. Rewards and termination signals derived from the proprioceptive sensors only. We employ a data-efficient off-policy algorithm [1] that allows for distributed training with a variable number of actors. The learning setup does rely additional external hardware for state estimation resets. This allows us to train multiple robots simultaneously, outside of the controlled laboratory environment, different geographic locations, exposing the robot range of environments with varying shape. appearance, and lighting conditions; see Figure (right).
Our results show that such an intrinsic learning signal designed promote forward movement (com- bined with the sensory capabilities of the robot and the partly structured environment) sufficient learn bipedal gaits and simple wall avoiding behavior requiring only minimal amount of prior knowledge, see Figure/ 1 (left) and supplementary video!. The distributed training with multiple robots improves data throughput and thus the complexity of the behavior that can be learned. Over- all, the results provide glimpse of the possibility of real-world learning for legged robots.
Our key contributions are: 1. First demonstration of learning deep non-parametric policy for unconstrained bipedal locomotion on hardware; 2. data-efficient autonomous learning setup for legged robots that can be easily deployed in different environments and multiple robots simul- tancously; Learned vision guided legged locomotion with simple obstacle avoidance capabilities.
2 Related Work
Conventional approaches to legged locomotion typically rely on models and optimization [9, 10] and concerned with concepts gait stability, such as zero moment point [1 1] and limit cycles [12]. While optimization based methods can yield highly effective solutions, they are laborious to imple- ment, require rigorous domain understanding, and can struggle to generalize diverse real-world environments and high-dimensn dinensiiinnion input modalities, such as vision.
Learning and data-driven approaches hold the promise cireumvent these limitation by avoiding the need for accurate modelling. Deep reinforcement learning for locomotion has been studied extensively in simulation [13, 14), and recent papers have shown progress in sim-to-real transfer
'https://sites.eee hattp:r//steeegggol.o.c///tt////

---

for both quadrupeds [15, 7. 16] and bipeds [17, 6, 5]. Compared to these works, we largely lift the assumption of the availability of simulation model.
Closest to our approach is the line of work where learning is directly applied on hardware. Unlike ours, these methods consider either constrained, hand-designed gait parameterization [18, 19, 20, 21J or apply unconstrained deep reinforcement learning to an inherently stable [22] or a quadrupedal robot [3, 4J. None of these works consider an unconstrained humanoid robot or vision observations, and, unlike our approach, they require external instrumentation such as motion capture or other means to quantify the obtained gait.
3 Methods
3.1
Generalizable Instrumentation and Learning Autonomy
We aim to create learning setup for real robot to learn to walk in different uncontrolled environ- ments (e.g. households). Consequently, we cannot rely on external sensing such as motion capture systems and we cannot reset mechanisms such gantry. The setup also has to deal with natural, possibly cluttered and irregular environments with variable natural and artificial lighting.
In this paper, we focus on simpler instantiation of this setting in which the robot is tasked to maintain an upright position, and to move forward fast as possible in an area surrounded by walls. This forces the robot to use its legs for locomotion while avoiding the walls. Access to camera observations enables the robot to look ahead and adapt its behavior its surroundings and thereby to deal with a variety of environments. For generality and robustness reasons, we do not employ any (possibly brittle) image pre- pre-processing steps, c.g. for wall detection. Instead, we learn the mapping from the camera pixels to the control signal end-to-end.
The design of our rewards and episode termination conditions is based on [8]. The reward is gen- erated when the feet exhibit a positive forward velocity in the robot's ' body coordinate frame. To assess this, the velocity of the robot body is computed by assuming that the lowest foot is in contact with the ground and the contact point is stationary (no-slip condition). While these assumptions may not always hold, Hafner al. [8] have shown experimentally that this reward may still be sufficient to learn forward motion when combined with a reward for keeping the torso upright. This upright- ness is in turn estimated from a gravity vector obtained via the IMU's internal orientation estimate. Given that the velocity reward encourages forward movement in the robo-centric coordinate frame, there is a risk of promoting walking in small circles. To avoid this, we penalize large changes in the heading direction, again based the IMU's orientation estimate. The three reward terms (ve- locity, uprightness, and no turning) are linearly combined to form the final reward. Episodes are terminated whenever the robot's body inclination exceeds certain threshold (see Section 4 in the supplementary material for more details about the reward).
Finally, we make use of simple reset mechanism to allow autonomous learning: At the end of each episode, the robot has fallen (as inferred from the IMU orientation estimate), we use a robust pre-programmed feed-forward controller to stand up. On rare occasions multiple attempts may be required. The robot does not need to be moved and simply continues from the location where ended the previous episode. See Section 2 in the supplementary material for a detailed overview.
3.2 Robot Platform
We focus on the off-the-shelf OP3 bipedal robot platform [23], or its simulated analog modelled in MuloCo [24]. This platform is built 20 Dynamixel servo motors XXM430-3350RR that we use in position control mode. All controllers use 20Hz control frequency. In real robot experiments, all controllers run directly on the onboard Intel NUC i3 computer using ROS [25] to send commands to the actuators. While the low weight and low cost of the robot are advantageous for real robot learning experiments, the low actuator and sensory specifications increase the difficulty of the learning task.
3.3 Distributed Data Efficient Reinforcement Learning
The input to our policy consists of proprioceptive observations listed in Table 1, and the two most recent RGB camera observations down-sampled to 64x64 resolution. The output of the policy is a distribution over 20 dimensional actions corresponding to joint position commands.
3

---

Observation
Dimension
history of joint positions
x 20
history of IMU accelerations
5 X 3
history of IMU angular velocities
5 x 3
history of egocentric gravity directions
5 x 3
history of past actuator commands history of linear velocity estimates (see [8])
5 x 20
5 x
heading change over 500ms
2
Table 1: List of proprioceptive observations received by our policies.
Our policies are trained using an asynchronous distributed version of the actor-critic MPO algo- rithm [1, 26] which uses distributional critic [27] (DMPO), and which has shown be effective in addressing high-dimensional tasks with continuous action space. The algorithm is implemented in JAX [28] using the code² released as part of [29] as a reference. All hyperparameters are listed in Section 5 in the supplementary material.
The robots' onboard computers actor processes which synchronize the policy's 's neural network weights with remote learner process at the beginning of each episode. After an episode of execut- ing the policy the robot, the actor process sends all data experienced in that episode to remote replay buffer implemented using the Reverb library [30]. The learner process continuously reads data from the replay buffer, and updates the network weights via gradient descent the DMPO actor and critic losses. In order to prevent overfitting, the learner process is blocked when the total number of gradient steps exceeds some threshold times the number of control steps executed by all robots. In our experiments, this threshold is 1/4. The remote learner process runs in the cloud. Communication with the robots is implemented using the Launchpad API [31].
One advantage of this setup is that we can scale from one robot to multiple robots in practical and application-relevant way. But it also shifts the computational load away from the robot and the location of the experiment. This in principle allows to learn with robots having weaker onboard computing power, basically requiring only internet connection allow learning experiments.
x4 64 x 64 x RGB images
History of propriaceptive observations 5 52 2
256
250
256
Resnet block (N=8)
Resnet block (16)
Policy
stop_grad
256
Action 1 20
conv 3x3 max pool 3
dense (FC)
Critic
layer norm tanh
concat
ReLU
sum
N # conv features
Figure 2: Network architecture. A 3-block Resnet produces image representations, which are con- catenated with proprioceptive observations, and fed to separate policy and (action-conditionee) critic networks; see text for more details on tensor shapes and losses.
3.4 Network Architecture
The network architecture is depicted in Figure 2 and consists of an image encoder shared between the policy and critic networks. It processes the latest two RGB images received from the robot's s camera. The outputs of the encoder and processed by separate policy and critic networks. The shared encoder is trained using gradients from critic losses only. The output of the critic is 51-dimensional corresponding to the logits pa-
concatenated with proprioceptive observations (see Table 1)

---

rameterizing the distribution of future returns [27]. The policy outputs the mean and the standard deviation of diagonal Gaussian distribution from which we sample the actions for the actuators.
4 Experiments
For safety reasons and to ensure autonomous learning, the workspace in our experiments consists of rectangular court delimited by foam walls. This prevents the robot from walking away, and, in combination with stand-up resets, represents a very effective way to minimize human intervention. In some sense, this is similar to the play pens used for babies when they start learning to walk, Except for this, all other environmental conditions are uncontrolled, e.g. we train in different daylight conditions, with various furniture items present around the workspaces, and different court sizes (see Figure 1). To avoid entanglement with the power cable, real robots are powered by a battery pack which needs to be changed roughly every 30 min.
At the beginning of training. the policy explores randomly which causes the robots to fall very often. To reduce the risk of damaging the hardware, we equip the robots with a 3D-printed body armour covering the front and back of the torso. In addition, we limit the range of the joints to prevent self-collision that might damage individual actuators or other hardware parts. We also apply an exponential filter on the actions sampled from the policy (see Section 6 in the supplementary material). History of these filtered actions is included in the policy's 's inputs (see Table 1) in order to mitigate partial observability due to delays.
4.1 Simulation Experiments
A simulation environment is used to make preliminary tests ahead of the real world experiments. Aside from using it to tune network architecture and hyperparameters, we want to know if the sample complexity is within a reasonable range and whether vision can be used as viable alternative to instrumented state estimation.
We roughly replicate the real world setup a 4 x 4 m court surrounded by walls - in simulation (see Figure 3). We randomize the textures for the floor and walls in every episode to roughly mimic the diversity of visual inputs in the real world setup where the learner receives trajectories from actors running in different locations and under various lighting conditions. The position and orientation of the robot are randomised at the beginning of every episode. We do not make additional efforts to make this simulated environment more realistic as we do not aim for sim-to-real transfer (see Section 1). Given the lack of rich textures, realistic lighting conditions, and motion blur it would be unlikely for a policy to transfer (see Figure 6 in supplementary material).
Figure 4 depicts the training curves of three different agents: 1) vision agent using raw images, 2) a pose agent having access to ground-truth robot pose (position and orientation), and 3) a blind agent without additional non-proprioceptive measurements. All are trained in an open square court but tested either with or without an additional disturbing wall (see Figure 3). Starting with the blind agent, we can observe that the attained performance is only marginally affected by the presence of an additional wall because it learns a behavior that is robust to changes of wall configuration.
With access to ground- ground-truth pose (pose agent), the robot learns very quickly to walk within the court. But when adding a wall at test time the performance strongly degrades as the robot collides with the wall and is unable to recover. This is improved by the proposed vision agent. While performing almost on par with the pose agent in the simple square court, it is also able to naturally generalise when the wall configuration is altered. This shows that there is a significant benefit when including vision in real world settings were space is usually limited and more cluttered. The difference in behavior can also be appreciated when visualising the policy (see Figure 3): The policy with vision avoids the wall (which was not present during training) and turns into a different direction (see Section 7 in the supplementary material for further details). Overall, the vision agent achieves good performance after about 1M environment steps. On a real robot, and taking into account repair and maintenance, this could be collected within 5 days.
Given the limited space of our environments all experiments have the tendency to develop circular motion patterns (see Figure 3) despite the "no turning" component of our reward. This is expected- we only use the "no turning" reward to discourage walking in very small circles which would prevent the robot from exploring the workspace and interacting with the walls.
5

---

Figure 3: Overview of the simulation setup. Our simulated model of the OP3 robot is in 4 x 4m square court with random textures. Top: training setup. Bottom: test setup with additional wall to observe the reaction of the robot. We also visualise trajectories of the hest performing vision agent which is also the one we train on the real robot. One can observe the circular motion that is learned and how the robot avoids walking into the walls. The white square highlights the start position.
Extra wall: False
Extra wall: True
600
Episode reward
450
Blind
300
Pose
Vision
150
0.25
0.5
0.75
1.0 0
0.25 0.5
0.75
1.0
Environment steps (millions)
Figure 4: Simulation results comparing agents trained with vision and proprioceptive sensing (green), with groundtruth pose and proprioceptive sensing (blue), and with proprioceptive sensing only (red). The right plot shows the performance of the same agents in court with an additional wall which not present during training. The shaded corresponds 95% confidence interval across seeds.
4.2
Real robot experiments
Our robot experiments focus sample complexity relative to simulation experiments, on assessing whether our trained policies make use of vision, and on evaluating the degree to which they can generalize to different robots and environments.
Figure (left) shows the learning curve for the distributed two-robot experiment. We plot the episode results experienced by different robots in different colors to demonstrate that the robots had to be occasionally stopped for repairs (while the other robot would keep collecting data). Interesting behaviors start to emerge after about 25k environment steps of training which corresponds to 2-3 hours of experiment time. This includes battery changes, time spent on resets and logging data in between episodes. It takes much longer to collect 25k environment steps at the beginning of the experiment than later stages because the robot falls a lot and experiment time dominated by the "stand up" reset behaviors (see Figure 5 in the supplementary material). At this point the robot has learned to balance and to avoid falling which causes the episode lengths and the returns from the "upright" component of the reward to increase. From this point the robot starts taking

---

first steps, steadily improving its walking speed and stability. It develops a bias for turning right and converges to circular walking behavior inside the enclosed workspace. See supplementary video for recordings from the experiment.
600
500
450
Ealsode reward
Eploode reward
300
Robot 2
Robot ?
Single
200
150
100
0
0.25
0. 0.75
1.0
10 12
1.4
Environment steps (millions)
Per robot uptime (hours)
Figure 5: Left: Training curve for a distributed experiment with two robots called Robot / and Robot 2. The episode rewards achieved by each robot plotted in different colours. We do not smooth the curves to preserve information. The intervals where only one colour appears correspond downtime of one robot. We also show learning curve from an analogous simulation experiment. Right: Episode reward plotted against per robot uptime, comparing a 2-robot experiment and an independent single robot experiment.
Figure 5 (left) also shows a learning curve from an analogous simulation experiment. Once the robot learns to avoid falling, the rate of learning progress about half that of the simulation. We believe this gap is caused by several factors which make the real environment more complex than the simulation: inaccurate model of delays in simulation; changing robot dynamics due wear and tear of the joints causing them to become loose and eventually completely broken; changing robot dynamics due battery voltage fluctuations: and non-stationary lighting conditions.
We also ran a shorter, single robot experiment and observed a similar sample complexity as that of the two-robot experiment (see Figure 4 in supplementary material). This is not obvious priori because larger diversity of inputs (as is the case in the two-robot experiment) typically renders reinforcement learning more difficult. The single robot experiment also serves "seed" and provides evidence for the reproducibility of our setup. To demonstrate the increased throughput of the two-robot experiment, Figure (right) shows the dependence of individual robots' performances on their uptime, and compares it to the same curve for the robot in the single robot experiment. We that individual robots in the two-robot experiment, where data is shared, improve faster, pointing to a promising direction of scaling up real experiments.
additional
Visual wall avoidance To demonstrate that the learned policy uses camera observations avoid walls, Figure 6 depicts the reaction of the robot when encountering the wall. Because of the policy's 's right turning bias, we initialize the robot so that wall on its right (red square marks the robot's initial position). If no wall is present, then the robot turns in a large circle (top row). In comparison, when a wall present, the robot takes a sharper turn to only just brush the surface of the wall (bottom row).
Further robustness tests We evaluated the final policy obtained the two-robot experiment in two additional settings: 1. on third robot which was not used during training, and 2. the same robot which was used in the experiment but using a black floor instead of a blue one. Table 2 shows the results of this evaluation. The policy works well on Robot achieving only slightly lower per step reward than the robot used during training (Robot 1). However, it is less stable on Robot causing the robot to fall more frequently. While this difference can be due to transfer, it is likely that the state of Robot 3 was unfavorable as mentioned before, the dynamics of the robots vary across time and things like loose joints screws impact what robot can do.
The policy performs considerably worse on black floor demonstrating that the visual module adapted to the blue floors encountered during training (also see supplementary video). However, as demonstrated in the simulation experiments, this could improved by exposing the agent to black floors during training as well.
7

---

Figure 6: Qualitative comparison of robot trajectories when encountering a wall. The robot starts in the red square facing right. It then starts walking while slightly turning towards its right. The robot makes a sharper
when there is wall (beige foam block, bottom row) than when remove the wall (the white dots show where it would be). The first point of contact with the (potentially virtual) wall is depicted by red dot. In the presence of an actual wall contact can either be avoided or is shifted towards the left as to only just brush the wall.
Evaluation context
Per step reward
Episode length
Robot
0.408 + 0.002
857 = 151
Robot 3
0.389 + 0.003
520 + 160
Black floor
0.318 + 0.005
136 + 36
Table 2: Evaluation of the learned policy under various contexts.
5
Conclusions
In this paper we showed that we can learn walking gait for a bipedal, off-the-shelf robot, directly on hardware, outside well-controlled lab infrastructure. Using state-of-the art RL method (DMPO) renders learning with minimal prior knowledge feasible in terms data efficiency.
Scalable reinforcement learning in the real world is still in its infancy, and to the best of our knowl- edge, this one of the largest scale experiments for model-free learning for legged robots. Limiting ourselves to on-board sensors only and collecting data from multiple robots at various geographic locations allows us to scale up the diversity of environments which the robots encounter. While in this paper used only two locations with different lighting conditions, arena sizes, and back- ground visuals, in future this can include many more locations with varying terrains and obstacles. This should be contrasted to lab settings with immobile equipment for external state estimation. We further also intend to remove the requirement for a reset controller by learning to stand-up.
Specifically, we plan to explore learning more dynamic behaviours, as well as tasks specified by rewards derived from camera images such as searching for an object in a cluttered environment. Such tasks require memory, complex visual processing, and longer horizon decision making, hence will necessary to train for much longer than 1M steps. We believe that the presented approach is well suited for learning at such scale. We also plan follow "curriculum" and progressively move, or completely remove, the protective walls.
Social Impact Statement Embodied agents using perception and locomotion have the potential to enable applications with significant positive social impact. E.g. they can be deployed in search-and- rescue operations (required in force majeure events like earthquakes, landslides, nuclear disasters), infrastructure inspections that might pose risks for humans (c.g. assessing a bridge or tunnel is in a healthy condition). and many others. However, there is also the risk that AI-driven technology in general can be used for harmful purposes, e.g. in military applications. We strongly oppose such use cases. We believe that our particular application presents low risk of being used in such situations as it has limited potential for causing large-scale damage, but we actively support discussions around safety and fairness in Robotics.
8

---

References
[1] A. Abdolmalcki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posterioni policy optimisation, 2018.
[2] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[3] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep inforcement learning. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgauu, Germany, June 2019. doi:10. 100.1115660/RRS2201..1110
[4] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. CoRR, abs/2002.08550, 2020. URL attpss: /ates://arrxa //axxivvv. www.i.org.......o.o........ /a/aa////20000 000.//022220000.0.2 08550.
[5] Siekmann, K. Green, J. Warila, A. Ferm, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint
2021.
[6] Z. Li, X. Cheng. X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. arXiv preprint arXiv:2/03: /4295, 2021.
[7] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science Robotics, 5(47)xeabc5986, Oct 2020. ISSN 2470-9476. 1oi: / abc5986.
URL http://dx: doi orriiiorg/1111 10. /116/scirobotics.
[8] R. Hafner, T. Hertweck, P. Klöppner, M. Bloesch, M. Neunert, M. Wulfmeier, S. Tunya- suvunakool, N. Heess, and M. A. Riedmiller. Towards general and autonomous learning of core skills: A case study in locomotion. CoRR, abs/2008. 12228, 2020. URL https:
// //arxiv.. org/ abaaaabs......////////////////// 12228.
[9] M. Hutter, H. Sommer, C. Gehring, M. Hoepflinger, M. Bloesch, and R. Siegwart. Quadrupedal locomotion using hicrarchical operational space control. The International Journal of Robotics Research, 1087-1004-1122 2014.
[10] S. Feng. E. Whitman, X. Xinjilefu, and C. G. Atkeson. Optimization based full body control for the atlas robot. In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 120- 127, 2014. 1oi: 109/H/MANOIIS 201...00...21....111..
[11] M. Vukobratović and Borovac. Zero-moment point- thirty five years of its life. Interna- tional journal of humanoid robotics, (01):157-177) 2004.
[12] T. McGeer et al. Passive dynamic walking. I. J. Robotic Res., 9(2):22-8) 82, 1990.
[13] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[14] N. Heess, D. TB, S. Sriram, Lemmon, Merel, G. Wayne, Y. Tassa. T. Erez, Wang, S. Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: (00::/777.0000 02286 2017.
[15] Hwangbo, J. Lee, A. Dosovitskiy, Bellicoso, V. Tsounis, V. Koltun, and Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[16] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784 2020.
[17] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. arXiv preprint arXiv: 1903.01900 2019.
[18] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 3, pages 2619-2624. IEBE, 2004.

---

[19] R. Tedrake, T. W. Zhang, and H. S. Seung. Learning to walk in 20 minutes. In Proceedings of the Fourteenth Yale Workshop on Adaptive and Learning Systems, volume 95585, pages 1939-1412. Beijing, 2005.
[20] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5-23, 2016.
[21] A. Rai, R. Antonova, S. Song, W. Martin, H. Geyer, and C. Atkeson. Bayesian optimiza- tion using domain knowledge the atrias biped. In 2018 JEEE International Conference Robotics and Automation (ICRA), pages 1771-1778. IEEE, 2018.
[22] S. Ha, J. Kim, and K. Yamane. Automated deep reinforcement learning environment for hard- ware of modular legged robot. In 2018 15th International Conference on Ubiquitous Robots (UR), pages 348-354. IEEE, 2018.
[23] Robotis op3.
introduction/
[24] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033, 2012. 200:10..100//R00.2022...33
[25] Stanford Artificial Intelligence Laboratory et al. Robotic operating system. URL https:
// //www.row www.ros..o org...
[26] A. Abdolmaleki, J. T. Springenberg. Degrave, S. Bohez, Y. Tassa, Belov, N. Heess, and M. Riedmiller. Relative entropy regularized policy iteration, 2018.
[27] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[28] Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Sunderman-Milng and Q. Zhang. JAX: composable transfor- mations of Python+NumPy programs, 2018. URL http: //tth/b.ttt/../// com/ google/jax..
[29] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Ab- dolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00 00979, 2020. URL https: //axxiv. wwexiii.or//aaeee orgg/ba/ 2006. 009799999............00.
[30] A. Cassirer, G. Barth-Maron, F. Brevdo, S. Ramos, T. Boyd, T. Sottiaux, and M. Kroiss. Reverb: framework for experience replay, 2021.
[31] Yang, G. Barth-Maron, P. Stanczyk, M. Hoffmann, S. Liu, M. Kroiss, A. Pope, and A. Rrustemi. Launchpad: A programming model for distributed machine learning research, 2021.
10


Reviews:
[{"review_id": "yOmcbdqUOyy", "reviewer": "Reviewer_Mhfu", "summary": "The paper describes the framework to produce walking gaits from on board proprioceptive sensors of an off-the-shelf small humanoid robot, by using Distributional Maximum a posteriori Policy Optimization (DMPO) and processing of visual data directly to policy generation. \n\nThe paper highlights that the reinforcement learning training is executed on the robot in real time with visual input data processed through frames with 3-block Resnet networks for the policy and critic networks. \n\nThe experiments show that the robot learns to walk while avoiding collision with the walls, and can benefit from sharing the learning experience with another robot. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe inclusion of proprioceptive sensing provides another venue for the learning of gaits and navigation for humanoid robots.\nThe policy training was realized in real time, so the robot had to be provided with a standing up strategy that allowed it to keep training.\nIt is stated how the vision system improves on the ground truth pose when walls are added, providing evidence of the importance of the vision system.\n\nWeaknesses:\nThe paper lacks a theoretical formulation to support the generality of their learning method. Could the experiments be scaled for more complex bipedal robots?\nThere is little detail about the weights of the reward function or the reinforcement learning action space. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper requires to be written with a proper formulation of the reinforcement learning algorithm that can be generalized to more complex or different bipedal systems. Also, there are many missing details on the construction and training of the neural networks and their action on the robot. \n", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "More mathematical background is required for the formulation and justification of the proposed reinforcement learning method. The idea of using a vision system for the walking gait is relevant, as well as a discussion about the scalability of the method.\n"}}, {"review_id": "swrnwKTvnp2", "reviewer": "Reviewer_tDwM", "summary": "This paper presents a system to train bipedal robot policies directly on hardware, with only on robot sensors, including an RGB camera. Result demonstrates that it is possible to train bipedal policies from scratch on hardware, including learning behaviors such as wall avoidance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n1. Learning bipedal locomotion policies on hardware, with RGB camera as sensors.\n\nCon:\n\n1. A manually defined reset controller is still needed. To truly \"learn from scratch\", it feels like this reset controller should also be learned. And it is not clear how this reset controller should be constructed for larger legged robot. For example, generating a stand up controller for a large quadruped is not trivial (e.h, Learning agile and dynamic motor skills for legged robots, by Hwangbo et al), not to mention a large biped.\n\n2. It feels like using simulation to tune some of the hyper-parameters is still necessary. (Please correct me if I am wrong).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this paper presents an approach to learn a policy directly on real bipedal robots (and maybe the first one to do so), one could raise the question whether \"learning from scratch\" is actually true.\n\nFirst, a reset mechanism (in this case, a manually constructed controller)  is still needed. And given the complexity of the policy structure, it seems like a simulation is still needed to tune hyperparameters before deploying it on the robot, so the claim that \"largely lift the assumption of the availability of a simulation model\" is not completely true. This raises the question, since we already need a simulation anyway, why not warm start from a policy trained in simulation instead of wasting additional efforts by training from scratch?\n\nThat being said, it is still interesting to see a policy can be learned on hardware directly within reasonable amount of time. Allow training on hardware is definitely valuable (wether it is training from scratch or warm starting), and this paper presents a system to do that.  And using a simple RGB camera is enough for learning wall avoidance is also a nice contribution.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. It will be interesting to know how much hyperparameter tuning is necessary in simulation before deploying the system on the real robot. e.g, the reward weights, the history length of the observation. It is not clear to me that I can arbitrary choose these parameters (within reasonble ranges) and the system will still work well on the real robot (or simulation). The supplementary material provides ablation on the resnet module, which seems to indicate tuning in simulation is a necessary first step.\n"}}, {"review_id": "klnZQ9XzGQF", "reviewer": "Reviewer_n7gp", "summary": "The paper presents a case study of learning locomotion skills on a small table-top size robot, Darwin OP3. The key achievements include 1) deploying DMPO, a data-efficient off-policy deep RL algorithm, on real robots, 2) training a policy from two robots in different geographic locations, and 3) training a visual policy out of real-world experience. As far as I know, these have not been achieved for real-world robot learning. The authors have demonstrated that the proposed on-robot learning system can train an effective locomotion policy on a small humanoid robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper demonstrates a practical case study of learning bipedal locomotion from real-world experience.\n+ The paper achieves some state-of-the-art results, such as learning bipedal locomotion, multi-robot learning, or learning visual policies from real-world experience.\n- The techniques in the paper are not too novel (because this is a case study).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a case study paper that investigates the scenario of deploying deep RL algorithms to on-robot learning environments. In summary, the presented experiments are quite interesting and can serve as a good data point for other researchers. However, we have to admit that the paper does not contain much technical novelty.\n\nThe most impressive result is visual navigation. Although it is hard to say that the robot learns a precise visual navigation skill: rather, the robot seems to estimate the distance to the goal from RGB images and tries to avoid the collision. But I can imagine that visual inputs would also suffer from the sim-to-real gap. It will be great if the authors can highlight the sim-to-real gap a little bit more by comparing simulation-generated images and real observations. It might be pretty obvious, but better to be explicit about it.\n\nAnother interesting contribution is to learn a locomotion skill based on proprioceptive observation. I am a little bit surprised (in both good and bad ways) that it works for real bipedal locomotion. It might be a little bit better if the authors can discuss this issue more. Why can we assume non-slip conditions? I guess locomotion of many small bipedal robots accompanies foot slips. Would it be possible to measure the accuracy of this reward function design by comparing it against the groundtruth mocap data?\n\nI also want to see more discussion about the resetting. Human interventions are also needed when the robot is out of bound (happens very often when the policy is matured) or stuck at the corner. Did you also conduct manual resets in this case? I would suggest the authors describe their approaches. \n\nSimulation results are not presented in the supplemental video. It might be good to include a couple of simulation results as well.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I made most of the suggestions above: please refer to the comments.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is borderline. Following the authors' responses, two of the three reviewers are positive. The AC is also positive, in part due to the focus on experiments with real hardware, which is encouraged at CoRL. The AC supports the majority recommendation.
Decision: Accept (Poster)