Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training
Authors: Wenhao Cui
Shengtao Li
Huaxing Huang
Bangyu Qin
Tianchu Zhang
hanjinchao
Liang Zheng
Ziyang Tang
Chenxu Hu
NING Yan
Jiahao Chen
Zheyuan Jiang
Venue: CoRL 2024 (2024)
Keywords: humanoid robots
locomotion
reinforcement learning
curriculum
sim-to-real
TL;DR: A training framework that leverages a  2 phase reinforcement learning and command curriculum learning, also refining DreamWaQ for mitigating joint oscillations, culminating in the successful sim-to-real transfer.

Abstract: Humanoid robots are a key focus in robotics, with their capacity to navigate tough terrains being essential for many uses. While strides have been made, creating adaptable locomotion for complex environments is still tough. Recent progress in learning-based systems offers hope for robust legged locomotion, but challenges persist, such as tracking accuracy at high speeds and on uneven ground, and joint oscillations in actual robots.
    This paper proposes a novel training framework to address these challenges by employing a two-phase training paradigm with reinforcement learning. The proposed framework is further enhanced through the integration of command curriculum learning, refining the precision and adaptability of our approach. Additionally, we adapt DreamWaQ to our humanoid locomotion system and improve it to mitigate joint oscillations. Finally, we achieve the sim-to-real transfer of our method. A series of empirical results demonstrate the superior performance of our proposed method compared to state-of-the-art methods.

Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training
Wenhao Cui ", Shhnnttao Li Huaxing Huang Bangyu Qin, Tianchu Zhang, Jinchao Han, Liang Zheng, Ziyang Tang, Chenxu Hu, Ning Yan, Jiahao Chen, Zheyuan Jiang 4| Noetix Robotics, Beijing
Figure 1: Our method enables the humanoid robot perform stably across various simulated envi- ronments. We successfully transfer the policy to the physical robot in various real-world environ- ments.
Abstract: Humanoid robots are key focus robotics, with their capacity navigate tough terrains being essential for many uses. While strides have been made, creating adaptable locomotion for complex environments is still tough. Re- cent progress in learning-based systems offers hope for robust legged locomotion, but challenges persist, such as tracking accuracy at high speeds and on uneven ground, and joint oscillations in actual robots. This paper proposes a novel train- ing framework to address these challenges by employing a two-phase training paradigm with reinforcement learning. The proposed framework further en- hanced through the integration command curriculum learning, refining the pre- cision and adaptability of our approach. Additionally, our humanoid locomotion system and improve it to mitigate joint oscillations. Fi- nally, we achieve the sim-to-real transfer of our method. A series of empirical results demonstrate the superior performance of our proposed method compared to state-of-the-an methods.
adapt DreamWaQ
Keywords: humanoid robots, locomotion, reinforcement learning, curriculum learning, sim-to-real
'Denotes equal contribution. Videos are available on locomotion/two petpotioontwo-phhee-ttee phase-training 'Correspondence merlin. nei.i...janggdooriirrbbooeo. jlanghoottixrroot.....coo
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
The humanoid robot has gained significant interest due to its human-like capabilities in performing various tasks [1, 2, 3]. Recent advances in reinforcement learning-basee legged locomotion sys- tems [4, 5, 6, 7] have shown great potential in dealing with complex, challenging environments. In this paper, we aim to address the challenges of achieving robust walking for humanoid robots on difficult terrains. We propose a two-phase sim-to-real (Sim2Real) learning framework, integrated with novel learning-based state estimation and curriculum learning, for humanoid locomotion.
The proposed framework divides the training process into two phases. In the first phase, the robots are trained on less challenging terrains with hand-crafted reference motion based on their structural and kinematic characteristics. Several reward functions are introduced encourage the robots to imitate these motions quickly. sinuous wave reference motion creates alternating leg movements for stable walking, encouraging quick imitation. However, these rewards may hinder exploration in phase two, which focuses on challenging terrains where adaptive gaits are necessary. Therefore, in phase two, by eliminating the reliance on predefined reference motions, our approach enables the robot to navigate more complex terrains with enhanced agility and efficiency.
Besides the two-phase training paradigm, we incorporate curriculum learning strategy and learningbbased state estimation method into framework. We adopt curriculum learning locity command tracking, proposed by Margolis et al. [8]. This progressive expansion in velocity enables the robot to develop robust and wide-ranging command tracking capabilities, essential for the overall training process. Our experimental validation shows that our robot can achieve a maxi- mum speed of m/s on diverse terrains. Furthermore, inspired by the DreamWar method [9], we design an estimator network to estimate the robot's 's base velocity. Compared the original method, we utilize multi-)rame((0) observations rather than single-frame inputs to the actor policy.
We design qualitative and quantitative experiments with various baselines to evaluate the proposed approach, as shown in Fig. 1. Empirical results indicate that our method outperforms the baselines in terms of command tracking, task success rate, and robustness. The proposed approach demonstrates excellent robustness, even when transferred to novel terrains that are not initially present in the simulated environments. Our contributions can be summarized as follows:
We propose a simple yet effective two-phase training paradigm that enables us to learn robust, relatively high-speed locomotion controllers to traverse various challenging terrains on a real humanoid robot.
We improve the DreamWaQ method by changing the input for the policy network from single-frame to multi-frane (ulti-)raam((O) information. This straightforward yet practical approach enhances walking stability and significantly benefits the Sim2Real transfer.
We adopt a velocity command curriculum to our training framework, enabling the robot to work wide range of velocity commands over challenging terrains.
2 Related Work
Reinforcement Learning for Legged Locomotion. Deep reinforcement learning has shown great potential in legged locomotion control and becoming the mainstream control algorithm for legged robots. Applying the teacher-student training paradigm in Sim2Real learning of legged locomotion is popular among research works. Lee et al. [10] applied teacher-student training to the quadruped robot ANYmal, resulting in robust controller capable of traversing challenging terrains. Kumar et al. [11] trained locomotion policies with rapid motor adaptation, enabling them to adapt to environ- mental changes quickly. The teacher-student training paradigm applics a neural network to perform system identification. contrast, another stream of research uses a separate neural network to per- form state estimation, which is relevant this work. Ji al. [12] trained a state estimator policy to estimate robot base velocity and feet contact. Dream WaQ [9] further extends learning based state estimation into estimating hidden states. Learning with reference motion promote the training

---

Actor
A
PD
Temporal partial Observation (of))
Controller 1000
Encoder
Decoder
Hz
z
o++1
Torques
Privileged observations (s!")
Critic
Joint state
Value
Loss
Reward
Gradient descent
The temporal partial observation of is encoded by an encoder which outputs the velocity estimation v and the latent z to be received by actor-network and decoder. The decoder predicts the observation at the next time step, denoted by o combining with o++1 obtained from the simulation calculate the loss. The details of loss function designing are in Sec 3.3. Action from the actor network presents the joints' target position; then the PD Controller calculates the torques. This framework applies both phases.
Figure 2: Overview of our method:
process and make the resulting gait more natural. Peng et al. [13] introduced the use of Deep Mimic [14] to learn robotic locomotion skills by imitating animals. Escontrela et al. [15] further utilized Adversarial Motion Priors (AMP) to train control policies for quadrupedal robot, highlighting that AMP can effectively substitute complex reward functions. Yang al. Bohez et al. [16] trained low-level locomotion controller for quadruped robot by imitating actual animal data for various tasks. Curriculum learning. which can help with exploration, is essential in legged robotics. Mar- golis et al. [17] applied curriculum learning to train a locomotion controller for the Mini Cheetah robot, enabling it to achieve speeds of up to 3.9m/s, surpassing traditional controllers' speeds by a large margin. Other notable works include directly learning locomotion skills in the real world [18, 19], learning from both proprioceptive observation and exterioceptive observation [20, 21, 22], applying unsupervised skill discovery to discover locomotion skills [23].
Reinforcement Learning for Humanoid Locomotion. Humanoid locomotion is one of the most challenging tasks in legged locomotion and has been getting increased attention in recent years. Due to the embodiment difference and the inherent instability of bipedal robots, algorithms that work well on quadruped robots may need improvements to function fully on a bipedal robot. Kumar et state estimation
al. [7] extended the A-RMA algorithm the bipedal robot Cassie. Learning-bart further enhanced and studied in [24]. Imitating human motion is also of significant attention in bipedal locomotion. Cheng al. [25] relaxes imitation constraints legs and focuses imitating the upper body movements of humanoid robot. Zhang et al. [26] introduced AMP into humanoid locomotion. Other works include designing appropriate reward functions [27], using transformers to represent locomotion policy [28], Sim2Real learning of humanoid control policy (29).
3 Method
The overall architecture of our method can be depicted as illustrated in Fig. 2. It provides a schematic representation of the network's 's composition, detailing the interconnected layers and components that constitute the system.
3.1 Network Input
Estimator Network: Similar to the works from Kumar et al. [11] and Margolis et al. [8], an es- timator network using the history observation of [o, 0:-1 timating the latent variable. In contrast Nahrendra et al. [9], we utilize of as part of the in- put for the policy network, rather than inputting single frame of observation o,, demonstrating considerably superior performance. The single observation is a 40-dimensional vector encom-
0z-H]² is necessary for es-
3

---

passing a rich set of sensory inputs and robot dynamics. It is an n x 1 vector defined as follows: o, [Pt C₁ Yt qt d a--1]", Where P. (2D) is the current phase indicated regular mo- tion which is generated by sinusoidal signal, C₁ (3D) is velocity commands used to give the current desired speed direction of the body, y: (5D) is the body angle (only roll and pitch used) and angular velocity, (10D) is the amount by which the joint angles changes relative to the default joint angles, qe (10D) is the joint angular velocity, and a,-1 (10D) is the previous action.
Policy Network: The latent variable obtained from the output of the estimator network are z, and body velocity Vi, and then, they coated together with the temporal proprioceptive observation of as the input of the policy network. Finally, the complete policy network, parameterized by s, can be expressed as ( o."" Z,, Ve).
Value Network: The value network receives more information from the world's state, which can be obtained directly from the simulation environment. The privileged observation S, defined as: 8; = (o; Ve h f,] where h, and f means the topographical scans. Specifically, h, denotes the terrain map of the robot's surroundings, while f represents localized terrain map for the robot's left and right feet.
3.2 Reward
The design of the reward function is pivotal in shaping the agent's policy, it directly influences the actions selected and the resulting motion patterns exhibited by the humanoid robot. In humanoid locomotion, the reward signal reflects the efficiency and effectiveness of movement and embodies the safety and adaptability required for navigating unpredictable terrains and situations.
In this work, we develop reward system that propels the humanoid towards achieving agile, sta- ble, and energy-efficient locomotion across diverse environments and tasks. The rewards delineated in [9, 29] facilitate the acquisition of basic ambulation; however, the stability of the robot's move- ments leaves room for improvement. The reward architecture comprises several components:
r
Tgait + Tcommmond + Troot Tenergy + p Teeerrreeeeee ferenccee
where gaatt regulates gait patterns, Yrommand promotes speed commands tracking, Tront ensures the upright posture and correct orientation of the robot's base link, Tenergy penalize excessive energy consumption, Preference encourages adherence reference motion, p {0, 1} equals one only in training phase one. The detailed reward function and weights are in Table 4.
3.3 Loss Function
For estimator, similar to Nahrendra et al. [9], in our approach, we require encoder-decoder framework where the decoder generates 0/+1 to compute the encoder's loss explicitly. The specific method involves calculating the mean-squared-error (MSE) loss between the estimated body veloc- ity ve and the ground truth v, from the simulator. Additionally, we employ the standard 3-VAE loss to compute the loss for decoder result 0/41 and the ground truth 0 and sum these two losses. To further increase the sim-to-real robustness of the learned policy, we also utilize adaptive bootstrap- ping (AdaBoot), technique similar to that used in Nahrendra et al. [9], to stabilize the estimator's updates.
Lest = LMSE + LVNE
TISE MSE(V,, v(), TAAE = MSE(0041,0'4) + (KIL(g(a)l
3.4 Learning System
The training is divided into two phases. In the first phase, the reference motion incorporated as a reward to encourage the robot to learn to move quickly according to a standard sinusoidal gait. In the second phase, the reward for the reference motion removed to allow the robot to learn better strategies to adapt to the terrain. The terrain includes planes, steps, slopes, and stairs. During the first phase, the difficulty of the terrain is relatively low, with more simple
Two-phase training:

---

terrains and lower stairs height. However, in the second phase, we increase the difficulty of the terrain, raising the proportion of high-difficulty terrains such as steps and stairs to 50% with higher stairs height. In this way, our robot can step up to 16cm high stairs at most.
Commands Currieulum Learning: During training, we observed that robots tend walk in place on dangerously uneven terrains, even with high linear velocity tracking rewards. To address this, inspired by Margolis et al. [8], we implemented a curriculum learning strategy for velocity, which means the robot initially focuses on learning to follow the reference gait phase at low speeds. Once the average velocity reward exceeds a threshold r = 0.75, we gradually increase the linear velocity commands by Avz.y = 0.05 m/s and angular velocity commands by Awi = 0.05 rad/s, enabling effective speed tracking.
Multi-Observatie with Estimation: We have identified that single-frame frame observation augmented with estimator predicced information [9] [24], while capable of tracking velocity commands in simu- lation, often result in joint tremors and suboptimal performance on varied terrains. Conversely, rely- ing on multi- (r2m observation without estimator [29] integration leads to slow movement and weak action robustness. To address these issues, we developed our proprictary Muuuer--bssevaaiion with Estimation"" method to train policies. This approach effectively tracks velocity commands, navigates through diverse terrains, and demonstrates excellent robustness and adaptability when de- ployed actual hardware. It seamlessly transfers the policy from simulation to the real robot.
3.5 Sim-to-Real
To accomplish the sim-to-real transfer, we designed various randomizations and delays for instruc- tions. Firstly, we applied randomizations the observation and the robot's properties for domain randomization. In order to mimic the effects of real-world deployment as closely as possible, introduced a random delay of up to 15% in both action and torque to help bridge the sim2real gap- Besides, We adjusted the center of gravity offset and link mass to adapt to the gap in the real en- vironment. Additionally, we apply a random force to each link every time an action is outputted. These measures have significantly improved the situation where the robot's hardware undergoes some design adjustments, which necessitated retraining with past methods. A detailed summary of the domain randomization and the execution delays is presented in Appendix Table 5.
Experimental Results
We conduct Numerous comparative and ablation studies in simulation and reality. This section will provide a detailed account of the specific experimental designs and the final comparative results.
The comparative experiments we performed are as follows: Baseline: no estimator policy with frame stacks observation; DreamWaQ. the reproduced method presented in Nahrendra et al. [9]; A-RMA: the reproduced method presented in Kumar al. [7]: Ours w/o CC: Our policy without commands curriculum, fixed at [-1, 1] m/s; Ours w/o Ref: Our policy without rewards associated to reference motion; Ours w/o LVAE: Our policy without LVAE loss; Phase I: Our training strategy for the first phase with reference motion.
Reference motion: In experiment Ours w/o Ref. the significant differences in the contact forces between the robot's two feet can cause considerable strain on its motors and links, resulting in significant wear and tear, as shown in Fig. 3, the figure illustrates the contact forces for both the left and right feet, highlighting stark contrast between the policy executed with reference motion and one without it. The latter demonstrates substantial variability in force application, an issue absent in our policy. The absence of reference motion can result imbalances, potentially leading to motor damage in real robots, whereas our approach ensures consistent stability and motor protection. Additionally, the uncoordinated movements significantly increase the likelihood of the robot falling, underscoring an urgent need for synchronized actions and balanced force distribution enhance reliability and prevent such incidents. Consequently, we omitted this comparison in the following analysis.
5

---

2500
2000
2500
1000
500
100
1.5
20
25
time
Actions our policy
Fal
0.2
0.0
-0.2
(b) ctions of our policy without reference motion
-0.50
-0.25
position (rad)
Figure 3: The upper-left diagram reveals that the ground contact forces produced by the strategy, which lack reference motions, surpass the endurance limits of the motors extremely. The lower-left illustration indicates that the hip and knee joints, under the no-reference strategy, are inclined to move to more extreme positions, potentially causing damage to real robots. Right-side comparison strikingly outlines the difference gaits between those informed by reference motions and those without, underscoring the no-reference strategy's s challenge in sustaining stability.
Multi-fask: We conducted experiments comparing the robot's performance between Phase one and two. In the first experiment, the robot tested for its ability climb stairs at speed of 1 m/s, with success defined clearing a six-step staircase. The second experiment evaluated the robot's ability to traverse slopes with varying gradients at the same speed. Our approach, not constrained by predefined reference motion, showed effective traversal of steps and slopes, as detailed in Table 1. In contrast, other strategies exhibited instability on terrains deviating from reference motion param- eters. While alternative methods struggled to climb steps over em high, our system maintained an 85% success rate on 10 em steps. Similar proficiency was observed in traversing slopes steeper than 15 degrees, with Phase two being the only one navigate them effortlessly.
Table 1: The rate of successful stair ascents, evaluated across spectrum of increasing stair heights, correlates with the performance on inclines of varying degrees, showcasing the adaptability of the system diverse elevation challenges.
Sim
Real
Phase
Phase
D'WaQ
A-RMA
Phase
Phase
D'WaQ
A-RMA
0.03m
1.00
0.75
0.70
0.65
1.00
0.70
0.75
0.55
0.05m
1.00
0.15
0.15
0.20
0.95
0.05
0.10
0.10
0.08m
0.95
0.05
0
0.05
0.85
0
0
0
0. 0.100
0.85
0
0
0
0.80
0
0
0
5"
1.00
0.95
0.85
0.70
1.00
0.95
0.75
0.70
10"
1.00
0.80
0.85
0.55
1.00
0.85
0.8
0.40
15e
1.00
0.60
0.45
0.30
0.95
0.45
0.2
0.25
20%
0.95
0
0
0
0.95
0
0
0
Estimator loss: The estimator loss plays a crucial role in assessing the robot's ability to accurately predict linear velocity and latent vectors, which significantly aids in our ability to follow speeds and traverse various terrains effectively. We recorded all the loss curves of the methods mentioned

---

08
2.
3.10
230
3.0
2509 300
(a) Estimator loss
(b) MSE 1/ Joss
(c) MSE 0p+1 loss
(d) K1. divergence
Figure 4: Without our policy, the predictions for rendered by the Dream WaQ exhibit signifi- cantly inferior performance when juxtaposed with our approach. Furthermore, the subsequent o,+1 generated by DreamWaQ also needs to catch up in comparison to the outcomes yielded by our strategic implementation. The KL divergence across policies indicates a comparable level of learn- ing. However, upon comprehensive assessment, our strategy demonstrates superior performance in estimating the velocity Vi and latent variables Z₁.
above. The result is shown in Fig. 4. It shows that Multi-Ohservation with Estimation policy demonstrates superior predictive capabilities for v, and 2, compared to DreamWaQ, which relies solely single observation within its actor policy.
Stability: We conducted a comprehensive empirical analysis comparing our proposed strategy alongside DreamWaQ and standard baseline across continuum of simulated and actual robotic platforms. This battery of tests was designed rigorously evaluate the adaptability and versatility of each algorithm in diverse environmental conditions. Furthermore, to series of perturbations that simulate the unpredictable nature of real-world conditions, thereby evaluating their resilience and robustness. This final testing phase is instrumental in gauging the practical applicability and reliability of the algorithms under non-ideal circumstances.
subjected the algorithms
While all robotic systems encountered increased challenges on low-p surfaces, precipitating slip- page and elevated risk of collapse, strategy adeptly traversed the array of terrains shown in Table In the context of disturbance rejection trials, the intrinsic instability of the alternative al- gorithms led to an almost inevitable collapse under lateral forces and suboptimal success rates from anterior and posterior forces. Conversely, our strategy consistently exhibited unparalleled stability, withstanding forces applied across all directional vectors, resulting Table 3. Appendix shows that our robotic system has demonstrated the ability to traverse various terrains and obstacles.
Table 2: Success rate of passing various terrain.
Ours
A-RMA
D'WaQ
Ours w/o LVAE
Baseline
Urban
1.00
0.70
0.75
0.90
0.70
PVC floor
0.90
0.20
0.10
0.00
0.00
Grasslands
1.00
0.40
0.35
0.15
0.55
Wired-strewn
0.85
0.00
0.00
0.00
0.00
Table 3: Success rate of stability under external force.
Ours
A-RMA
D'WaQ
Ours w/o LVAE
Baseline
Front 1.00
0.45
0.40
0.40
0.10
Side 0.85
0.05
0.20
0.15
0.00
Back
0.90
0.20
0.05
0.35
0.05
Commands Tracking: The robotic system's primary objective is to track the velocity commands issued to it accurately. Consequently, the efficacy of commands tracking emerges as a pivotal metric for evaluating the success of our experimental endeavors. This performance is vividly illustrated Fig. 5. Observation have elucidated that when faced with arduous terrains, including stairs with heights exceeding 6 cm or 15-degree inclined uneven terrain within our training set. Baseline, Dream WaQ and A A-RMA tend prioritize safety, consequently adopting more cautious and slowed pace. Furthermore, the variant of our approach that forgoes commands curriculum exhibits di-
7

---

-227
time 55
Biee
time 65
time
VX ww.oo (009)
ann (nce)
(8) Plane
(b) Plane with obstacles
(e) Uneven Temain
(d) Stairs
Figure 5: The first tier of images presents a compendium of scenarios for testing, encompassing a flat plane, plane with obstacles, uneven terrain, and staircase. The second stratum delineates velocity profiles, illustrating the strategy's 's actual performance in tracking predefined commands trajectories, represented by black line. The third layer encompasses box plots that depict the distribution of velocity tracking errors within the simulated environment, the horizontal line indicating the median value.
minished capacity for velocity tracking, potentially due to the vast exploration space overwhelming the actor's learning process. Additionally, ours without LVAE have indicated that the Euclidean distance of the observations may not accurately represent the discrepancy from the target, resulting in suboptimal walking posture and inadequate speed following. In striking contrast, refined ap- proach showcases an enhanced aptitude for swiftly and adeptly acquiring and adjusting to velocity directives. Consequently, our strategy must incorporate commands curriculum to ensure proficient learning and adaptation to the velocity commands.
5 Limitation
Our policy, though effective in its intended tasks, is limited by the lack of perception for terrain movements. To address this, we plan to integrate perception systems into our future work, thereby enhancing our strategies' adaptability to more complex challenges and expanding the operational capabilities of our robotic systems. Additionally, we recognize our approach has limitations, par- ticularly parameter tuning for diverse robots, which complex engineering task. Adjusting reward parameters based robot's size and carefully selecting domain randomization parameters are essential; for instance, center of mass randomization exceeding 0.06 cubic meters can severely impact real-world performance. Time-related parameters need finely tuned to match the anatomical proportions of different robots to ensure optimal dynamic responses. To address these challenges and improve our strategies' adaptability, we plan to investigate solutions such as imitation learning, aiming enhance our system's robustness and applicability across various robotic platforms.
Conclusion
In this paper, we have presented a comprehensive study on the enhancement of steady humanoid robot locomotion, achieved through the refinement of estimator and actor policies. This research has broad implications, establishing foundation for versatile and adaptive locomotion across varied terrains and obstacles through two-phase training method that expands the robot's gait capabili- ties. Additionally, the study demonstrates an exceptional velocity tracking capability with velocity commands curriculum that underscores the system's 's precision and responsiveness.

---

Acknowledgments
We extend our heartfelt gratitude to the anonymous reviewers for their insightful feedback, which significantly contributed to the improvement of this manuscript. This work was supported by the Noetix Robotics
References
[1] J. Reher, W.-L. Ma, and A. D. Ames. Dynamic walking with compliance cassie bipedal robot. In 2019 18th European Control Conference (ECC), pages
IEEE, 2019,
[2] R. Bogue. Humanoid robots from the past to the present. Industrial Robot: the international journal of robotics research and application, 47(4):465-472, 2020.
[3] M. Chignoli, D. Kim, E. Stanger-Jones, and S. Kim. The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors. In 2020 IEEE-RAS 20th International Confer- ence Humanoid Robots (Humanoids), pages 1 8.. IEEE, 2021.
[4] Y.-H. Shin, S. Hong, S. Woo, J. Choe, H. Son, G. Kim, J.-H. Kim, K. Lee, J. Hwangbo, and H.- W. Park. Design of kaist hound, a quadruped robot platform for fast and efficient locomotion with mixed-integer nonlinear optimization of gear train. In 2022 International Conference on Robotics and Automation (ICRA), pages 6614-6620. IEEE, 2022.
[5] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 14341, 2023.
[6] Z. Wang. W. Wei, A. Xie, Y. Zhang. J. Wu, and Q. Zhu. Hybrid bipedal locomotion based on reinforcement learning and heuristics. Micromachines, 13(10):1688, 2022.
[7] A. Kumar, Z. Li, J. Zeng, D. Pathak, K. Sreenath, and Malik. Adapting rapid motor adapta- tion for bipedal robots. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1161-1168. IEEE, 2022.
[8] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via rein- forcement learning. The International Journal of Robotics Research, 43(4):572-587, 2024.
[9] I. M. A. Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal lo- comotion with implicit terrain imagination via deep reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5078-5084. IEEE, 2023.
[10] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47)xeabc5586, 2020.
[11] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXhv:210700403. 2021.
[12] G. Ji. J. Mun, H. Kim, and J. Hwangbo. Concurrent training of control policy and state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2):660044677, 2022.
[13] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint 200::200040078844 2020.
[14] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimie: Example-guided deep re- inforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 3744):1-14, 2018
[15] A. Escontrela. X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel. Adver- sarial motion priors make good substitutes for complex reward functions. In 2022 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 25-32. IEEE, 2022.

---

[16] S. Bohez, S. Tunyasuvunakool, P. Brakel, F. Sadeghi, L. Hasenclever, Y. Tassa, E. Parisotto,
J. Humplik, T. Haarnoja, R. Hafner, et al. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXi::220. 2022.
[17] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via rein- forcement learning, arXiv preprint Stti:220050022244 2022.
[18] Wu, A. Escontrela, D. Hafner, K. Goldberg, and Abbeel. Daydreamer: World models for physical robot learning. arXiv preprint arXiv:2206. 14176, 2022.
[19] L. Smith, I. Kostrikov, and Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint urXiv:2208.07866 2022.
[20] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal lo- comotion end-to-end with cross-modal transformers. arXiv preprint arXiv:2107.03996, 2021.
[21] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal. Learning to jump from pixels. arXiv preprint v7Xiv2211.113444 2021.
[22] T. Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab 2022.
[23] A. Sharma, M. Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu. Emergent real- robotic skills via unsupervised off-polliy reinforcement learning. arXiv preprint vrXiv:2004.12974 2020.
[24] Z. Wang, W. Wei, R. Yu, J. Wu, and Q Zhu. Toward understanding key estimation in learning robust humanoid locomotion. arXiv preprint 2403::2200305866. 2024.
[25] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-hody control for humanoid robots. arXiv preprint 24Xi::2402116777 2024.
[26] Q. Zhang, P. Cui, D. Yan, J. Sun, Y. Duan, A. Zhang, and R. Xu. Whole-body humanoid robot locomotion with human reference. arXiv preprint arXiv:2402.18294 2024.
[27] B. van Marum, A. Shrestha, H. Duan, P. Dugar, J. Dao, and A. Fern. Revisiting reward design and evaluation for robust humanoid standing and walking. arXiv preprint arXiv:2404.19173, 2024.
[28] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Learning humanoid locomotion with transformers. arXiv e-prints, pages arXiv-2303, 2023,
[29] X. Gu, Y.-J. Wang, and J. Chen. Humanoid-gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.05695, 2024.
[30] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 170.061777 2017.
10

---

A DETAILED EXPERIMENT SETUP
A.1 REWARD
In this section, we detail the reward function and the weights. The reward function is comprised of several critical components, each serving distinct purpose. The exponential function is rep- resented by exp(-), and the variance function is denoted by var(.). The terms (-)des and (.)amd are used to denote the desired and commanded values, respectively. The robot's body frame is defined by coordinates 2, y/, and 2, with x and oriented the forward and upward direc- tions. The rotation angles of the robot's coordinate system are given by roll, yaw, and pitch. Pf()) La(e): Le(e), Tair, V, w, h, y dR: g, 0, represent the height of the foot time the phase at time in the gait eycle, the contact status of the foot at time the aerial time of the foot, the lin- ear velocity of the robot's root, yaw rate, height, foot distance, knee distance, the projection of the gravity vector onto the robot's body frame, joint position, and joint torque.
Table 4: Setup of reward function and scales.
Reward
Equation (ra)
Scale (w;)
Feet clearance
(ogg) PJ())
(1 Id(1))
-0.01
Feet air time
T Toir Le(e)
-0,001
Follow gait phase
(1 a(t)) Ee(s)
-0.001
Feet slip
W c(l)
-0.005
Feet&Knee distance
exp(-100-jo.33
(0.3 3- d, + texpl 100-10.01100---00 125
0.4
Lin. velocity tracking
exp
(
ymd my
Vay
2.4
Ang. velocity tracking
exp
4 and
wyynw)
1.1
exp
10
v.
fr -( -illl s9ch }3
Velocity mismatch
0.5
Orientation
lg/ 2
1.0
Feet orientation
greet 2
1.0
Default joint
exp
(A
2ero
0.5
Body height
(he
0.6505)
2
-1.0
Root accelerations
exp
Ü Oreaa
0.2
Joint accelerations
-1 x 10-6
Joint velocity
0²
-5 x 10-3
Joint power
7²
-1 10-5
Action rate
(a: a 21-13 )
-0.01
Smoothness
(a. 2a,
+ a 2)
-0.01
Joint position tracking
exp 2 0 uurget
3.2
A.2 DOMAIN RANDOMIZATION
We leverage domain randomization during training to narrow the reality gap. Specifically, we set the range of parameters as shown in Table 5, mainly consisting of delay action and torque, ran- domization of position, velocity, friction, KP/KD factor, and CoM.
A.3 IMPLEMENTATION DETAILS
Our humanoid robot, named NI, is equipped with total of 18 degrees of freedom. In this work, have immobilized the 8 joints associated with the arms. focusing exclusively the 10 joints related
11

---

Table 5: Overview of Domain Randomization. Presented are the domain randomization terms and the associated parameter ranges. Additive randomization increments the parameter by value within the specified range while scaling randomization adjusts by a multiplicative factor from the same range.
Parameter
Unit
Range
Operator
Type
Joint Position
rad
[-0.05, 0.05]
additive
Gaussian (lo)
Joint Velocity
rad/s
[-1.5. 1.5]
additive
Gaussian (lo)
Angular Velocity
rad/s
[-0.2. 0.2]
additive
Gaussian (lo)
Linear Velocity
m/s
[-0.1, 0.1]
additive
Gaussian (lo)
Euler Angle
rad
[-0.06, 0.06]
additive
Gaussian (lo)
Action Delay
ms
[0, 10]
Uniform
Torque Delay
ms
[0, 10]
Uniform
Friction
10.1,2.0)
Uniform
Kp factor
%
[80, 120]
scaling
Gaussian (lo)
Kd factor
%
[80, 120]
scaling
Gaussian (lo)
Motor Strength
%
[80, 120]
scaling
Gaussian (lo)
Payload
kg
[-5, 5]
additive
Gaussian (lo)
CoM
[-0.02, 0.02]
additive
Gaussian (lo)
Link Mass
%
[0.9, 1.1]
scaling
Gaussian (lo)
to the legs. The motors' hip (pitch) and knee joint torque can reach up to 150 Nm, while the motors' torque of the foot joints is 36 Nm. This robot's total height and weight are 0.95 m and 23 kg.
Our RL control strategy operates 100 Hz, coupled with an internal PD controller that runs at 1000 Hz. ensures synchronization with the operational frequency of the actual hardware. We employ Isaac Gym for training and conduct sim-to-sim validation in various simulation environments, in- cluding MuJoCo, PyBullet, and Gazebo. This envliieeniiooment approach ensures the robustness and adaptability of our models. We utilize the Proximal Policy Optimization (PPO) algorithm [30]. The details of our training parameters are presented in Table 6, where we outline the specifics that contribute the enhanced performance of our model.
Table 6: Training hyperparameters
Parameter
Value
Number of environments
4096
Training epochs
2
Learning rate
10-5
Gamma ?
0,995
Lambda
0.95
Batch size
24
Episode Length
3000
Backbone hidden layers
[512, 512, 128]
Encoder hidden layers
[768, 256, 64]
Activation function
ELU
Decoder observation
19
Number of Observation frame stack
5
Number of Privileged frame stack
3
Number of Aggregated Observation
219
Number of Aggregated Privileged Observation
846
12

---

B ALGORITHM
We present our complete algorithmic process in the following algorithm, where four networks are continuously updated through sampling from simulation based on the process outlined below.
Input: Encoder network est,, decoder network dec,, policy network we, value function Ve, environment, number of epochs E, mini-batch size M
Output: Optimized encoder network est.. policy network To Initialize encoder network estØ, decoder network dec,, policy network vy and value function V
Initialize advantages A, = 0 and value targets V2 = for each epoch 1 F: do
for each mini-batch = 1 to M do Initialize observation og obtain frame stack observationol and collect rollouts for each step 1 to 7 do
Compute linear velocity and latent features 24 ~ csta(-lo,") Compute action =o0- 2t,
Execute in the environment and observe 0r41, of 1+1, Tr, de
Compute advantage estimate Å,, value target Vú linear velocity target 0 and next step observation targetó;41; Update advantages A, ptA:-1 A, and value targets V; = >V + 1/
end
Perform multiple PPO updates using A, and Ve to optimize and Vi and estimator updates using 0r+1, 2₁ and 0, to optimize esto and decii
end end
return the optimized policy network Ty and the encoder policy networkestg:
Algorithm 1: Training algorithm
C GENERALIZATION
This section presents a series of experimental images that vividly illustrate algorithm's intricate implementation details and robust generalization capabilities across various initial poses for robotic walking tasks, shown in the Fig. 6. We have achieved smooth locomotion across spectrum of initial postures, as evidenced by the images depicting the robot's initial stance.
Figure 6: Training with diverse initial poses: An illustrative analysis of robotic locomotion across varied starting configurations.
13

---

Within the simulated environment, the Fig. 7 display the robot's upright and stable gait, underscoring the algorithm's exceptional precision in control within the virtual platform. We have further sub- stantiated the algorithm's 's reliability, shown Fig. 8, and practical utility by advancing to real-world scenarios. The deployment our algorithm on an actual robot has endowed it with the ability to navigate in various straight-legged postures, as illustrated by the images portraying the robot's com- mendable equilibrium and stability. This rigorous process is testament to the algorithm's resilience and adaptability.
These outcomes furnish compelling validation for our algorithm's ongoing refinement and appli- cation, simultaneously presenting innovative perspectives and methodologies that hold significant promise for future research within related domains.
TOE
Figure 7: Diverse initial poses in simulation: A testament our algorithm's robustness and stability.
Figure 8: Simulated locomotion across diverse initial poses: Demonstrating algorithmic adaptability sim-to-real transitions.
14

---

D REAL-MACHINE EXPERIMENTATION
We conducted multiple experiments on the actual machine, testing our policy on various terrains, including grasslands, wire-strewn ground, slopes, and stairs. Our robot's traversal across grasslands testament its ability to handle the soft and unpredictable ground, while its passage through areas with wireless showcases its resilience against obstacles that could impede movement. The robot's ascent slopes highlights its dynamic balance and the algorithm's capacity adjust to inclines that require precise foot placement and torque control. Most notably, the robot's ability to climb and descend stairs indicates our algorithm's 's advanced control mechanisms, ensuring that each step is calculated for maximum efficiency and safety. The images reveal a robot that is not just mobile but one that can adapt to and stabilize on a wide array of environmental conditions, thereby proving the algorithm's robustness and stability comprehensive sim-to-real context.
These visual records more than just demonstrations of our robot's physical capabilities; they evidence of the sophisticated algorithms that enable it to interact intelligently with its surroundings, providing solid foundation for further research and development in robotics.
Figure 9: The sequence of images presented illustrates the diverse terrains our robot navigates with proficiency, ranging from the soft contours of grasslands to the challenging unevenness of wire- strewn areas, the inclines of varying gradients, and the ascents and descents of stairs. Each scenario, captured in vertical progression from top to bottom, demonstrates not only the robot's adaptability but also its ability to maintain equilibrium on surfaces that demand different levels traction and stability.
15


Reviews:
[{"review_id": "Buq8vIXLJw", "reviewer": "Reviewer_61BH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper extends studies the problem of robust blind biped locomotion. They extend prior work (DreamWaq) - an estimator network is trained to estimate velocities and map observation history to latent latent states. These condition a downstream policy which is trained via RL in simulation with domain randomizaton. They demonstrate locomotion on a real robot on small stairs, slopes, grass and flat ground with some perturbances.", "summary_of_recommendation": "The paper has nice real world results, however, I vote to weak reject this paper because of limited novelty in both results and method + insufficient comparison to prior work", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- In the video results, when going up / down the steps it is hard to tell whether the operator is applying a stabilizing force to the robot through the cord. Is the robot able to do the stairs reliably without support? Can the authors generate a video where the robot is held by a loose cord so that no force is being applied on it?\n- Is the only difference from DreamWaQ that the estimator gets a history of observations instead of a single observation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6yNjiqYoh", "reviewer": "Reviewer_rN9h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a two phase RL training in simulation with zero shot transfer to real for humanoid locomotion. The method relies on a learned estimator and a curriculum in terrain (two phase training) as well as velocity sampling to improve sim based RL learning. The proposed method shows high robustness in the real world and quantitatively superior performance to baselines.", "summary_of_recommendation": "The proposed method shows very compelling results in the real world on a humanoid robot. The experiments could be improved to make them more thorough", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- Estimator loss has two components. Could we ablate the importance of LVAE loss?\n- Figure 4: Could the authors define FHM used in the legend (point to where in the paper its defined)?\n- Are the terrains included in Phase one of the training or only in phase 2? If only included in phase 1, authors should include a comparison with terrains included in Phase 1. \n- L171: The sentence is not clear, based on what finding?\n- L237-L239: “This refined passage employs a more formal and academic 238 tone, with a clear and structured explanation of the experimental setup, observation, and the comparative efficacy of the proposed method against existing ones.” Has the passage been written by a language model?\n- Please address the limitations more thoroughly with discussion on things like: amount of simulation tuning required, the need to pick parameters of domain randomization which cannot be too aggressive, stability and reproducibility of the proposed method, etc.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJmBpUBvAk", "reviewer": "Reviewer_RdfP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method for humanoid robots to adapt over different terrains through the use of two-phased training.", "summary_of_recommendation": "The work is missing a clear overview of limitations which would improve the quality of the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Failure modes for the approach are the main questions that would improve the contribution of this work. In particular, where does the approach fail and why is something that is missing from this work.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduced a two-phase training approach to blind biped locomotion. They used an estimator network to predict velocities and map observation history to latent states. They then used these latent states to condition an RL policy trained in large-scale simulation environments and transfer the policy to the physical robot in real-world terrains. This paper received mixed initial ratings, with one weak reject and two weak accepts. The reviewers pointed out the strengths of this paper, particularly 1) the use of curriculum learning is an interesting idea, and 2) the real-robot deployments showed the robustness and practicality of this approach. In the meantime, they also mentioned several weaknesses, including 1) limited novelty: this work seems to be an incremental improvement over previous work, particularly DreamWaQ, and 2) experimental validation: comparisons to relevant methods, such as A-RMA, are missing and the experiments lack thorough ablation studies to validate the importance of specific components of the algorithm. In addition, all three reviewers found that the limitations were not well addressed. Overall, the AC thinks this work has good promises, but the reviewers' criticisms are valid and must be addressed before this paper passes the bar of publication.

**Post-rebuttal update**

The authors successfully addressed the concerns raised by the three reviewers. Toward the end of the rebuttal period, the reviewers had three unanimously positive recommendations. The AC felt pleased to see the positive changes, which improved the quality of this paper, and recommended accepting it for CoRL.
Decision: Accept