Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training

Authors: wenhao  cui
shengtao  li
huaxing  huang
bangyu  qin
tianchu  zhang
jinchao  han
liang  zheng
ziyang  tang
chenxu  hu
ning  yan
jiahao  chen
zheyuan  jiang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:14:53

Abstract: Humanoid robots are a key focus in robotics, with their capacity to navigate tough terrains being essential for many uses. While strides have been made, creating adaptable locomotion for complex environments is still tough. Recent progress in learning-based systems offers hope for robust legged locomotion, but challenges persist, such as tracking accuracy at high speeds and on uneven ground, and joint oscillations in actual robots.
    This paper proposes a novel training framework to address these challenges by employing a two-phase training paradigm with reinforcement learning. The proposed framework is further enhanced through the integration of command curriculum learning, refining the precision and adaptability of our approach. Additionally, we adapt DreamWaQ to our humanoid locomotion system and improve it to mitigate joint oscillations. Finally, we achieve the sim-to-real transfer of our method. A series of empirical results demonstrate the superior performance of our proposed method compared to state-of-the-art methods.

Main paper: 
 introduction: 
 The humanoid robot has gained significant interest due to its human-like capabilities in performing various tasks [1, 2, 3] . Recent advances in reinforcement learning-based legged locomotion systems [4, 5, 6, 7] have shown great potential in dealing with complex, challenging environments. In this paper, we aim to address the challenges of achieving robust walking for humanoid robots on difficult terrains. We propose a two-phase sim-to-real (Sim2Real) learning framework, integrated with novel learning-based state estimation and curriculum learning, for humanoid locomotion.The proposed framework divides the training process into two phases. In the first phase, the robots are trained on less challenging terrains with hand-crafted reference motion based on their structural and kinematic characteristics. Several reward functions are introduced to encourage the robots to imitate these motions quickly. A sinuous wave reference motion creates alternating leg movements for stable walking, encouraging quick imitation. However, these rewards may hinder exploration in phase two, which focuses on challenging terrains where adaptive gaits are necessary. Therefore, in phase two, by eliminating the reliance on predefined reference motions, our approach enables the robot to navigate more complex terrains with enhanced agility and efficiency.Besides the two-phase training paradigm, we incorporate a curriculum learning strategy and a learning-based state estimation method into our framework. We adopt curriculum learning on velocity command tracking, as proposed by Margolis et al. [8] . This progressive expansion in velocity enables the robot to develop robust and wide-ranging command tracking capabilities, essential for the overall training process. Our experimental validation shows that our robot can achieve a maximum speed of 1 m/s on diverse terrains. Furthermore, inspired by the DreamWaQ method [9] , we design an estimator network to estimate the robot's base velocity. Compared to the original method, we utilize multi-frame (20) observations rather than single-frame inputs to the actor policy.We design qualitative and quantitative experiments with various baselines to evaluate the proposed approach, as shown in Fig. 1 . Empirical results indicate that our method outperforms the baselines in terms of command tracking, task success rate, and robustness. The proposed approach demonstrates excellent robustness, even when transferred to novel terrains that are not initially present in the simulated environments. Our contributions can be summarized as follows:• We propose a simple yet effective two-phase training paradigm that enables us to learn robust, relatively high-speed locomotion controllers to traverse various challenging terrains on a real humanoid robot.• We improve the DreamWaQ method by changing the input for the policy network from single-frame to multi-frame (20) information. This straightforward yet practical approach enhances walking stability and significantly benefits the Sim2Real transfer.• We adopt a velocity command curriculum to our training framework, enabling the robot to work at a wide range of velocity commands over challenging terrains. process and make the resulting gait more natural. Peng et al. [13] introduced the use of Deep Mimic [14] to learn robotic locomotion skills by imitating animals. Escontrela et al. [15] further utilized Adversarial Motion Priors (AMP) to train control policies for a quadrupedal robot, highlighting that AMP can effectively substitute complex reward functions. Yang et al. Bohez et al. [16] trained a low-level locomotion controller for a quadruped robot by imitating actual animal data for various tasks. Curriculum learning, which can help with exploration, is essential in legged robotics. Margolis et al. [17] applied curriculum learning to train a locomotion controller for the Mini Cheetah robot, enabling it to achieve speeds of up to 3.9m/s, surpassing traditional controllers' speeds by a large margin. Other notable works include directly learning locomotion skills in the real world [18, 19] , learning from both proprioceptive observation and exterioceptive observation [20, 21, 22] , applying unsupervised skill discovery to discover locomotion skills [23] .Reinforcement Learning for Humanoid Locomotion. Humanoid locomotion is one of the most challenging tasks in legged locomotion and has been getting increased attention in recent years. Due to the embodiment difference and the inherent instability of bipedal robots, algorithms that work well on quadruped robots may need improvements to function fully on a bipedal robot. Kumar et al. [7] extended the A-RMA algorithm to the bipedal robot Cassie. Learning-based state estimation is further enhanced and studied in [24] . Imitating human motion is also of significant attention in bipedal locomotion. Cheng et al. [25] relaxes imitation constraints on legs and focuses on imitating the upper body movements of a humanoid robot. Zhang et al. [26] introduced AMP into humanoid locomotion. Other works include designing appropriate reward functions [27] , using transformers to represent locomotion policy [28] , Sim2Real learning of humanoid control policy [29] . 
method: 
 The overall architecture of our method can be depicted as illustrated in Fig. 2 . It provides a schematic representation of the network's composition, detailing the interconnected layers and components that constitute the system.Estimator Network: Similar to the works from Kumar et al. [11] and Margolis et al. [8] , an estimator network using the history observationo H t = [o t o t-1 . . . o t-H ] T, is necessary for estimating the latent variable. In contrast to Nahrendra et al. [9] , we utilize o H t as part of the input for the policy network, rather than inputting a single frame of observation o t , demonstrating considerably superior performance. The single observation o t is a 40-dimensional vector encom-passing a rich set of sensory inputs and robot dynamics. It is an n × 1 vector defined as follows:o t = [p t c t y t q t qt a t-1 ]T , Where p t (2D) is the current phase indicated a regular motion which is generated by a sinusoidal signal, c t (3D) is a velocity commands used to give the current desired speed direction of the body, y t (5D) is the body angle (only roll and pitch used) and angular velocity, q t (10D) is the amount by which the joint angles changes relative to the default joint angles, qt (10D) is the joint angular velocity, and a t-1 (10D) is the previous action.Policy Network: The latent variable obtained from the output of the estimator network are z t and body velocity v t , and then, they are coated together with the temporal proprioceptive observation o H t as the input of the policy network. Finally, the complete policy network, parameterized by ϕ, can be expressed as π ϕ (a t |o H t , z t , v t ). Value Network: The value network receives more information from the world's state, which can be obtained directly from the simulation environment. The privileged observation s t is defined as:s t = [o t v t h t f t ]T , where h t and f t means the topographical scans. Specifically, h t denotes the terrain map of the robot's surroundings, while f t represents a localized terrain map for the robot's left and right feet.The design of the reward function is pivotal in shaping the agent's policy, as it directly influences the actions selected and the resulting motion patterns exhibited by the humanoid robot. In humanoid locomotion, the reward signal reflects the efficiency and effectiveness of movement and embodies the safety and adaptability required for navigating unpredictable terrains and situations.In this work, we develop a reward system that propels the humanoid towards achieving agile, stable, and energy-efficient locomotion across diverse environments and tasks. The rewards delineated in [9, 29] facilitate the acquisition of basic ambulation; however, the stability of the robot's movements leaves room for improvement. The reward architecture comprises several components:r = r gait + r command + r root -r energy + p • r ref erencewhere r gait regulates gait patterns, r command promotes speed commands tracking, r root ensures the upright posture and correct orientation of the robot's base link, r energy penalize excessive energy consumption, r ref erence encourages adherence to reference motion, p ∈ {0, 1} equals one only in training phase one. The detailed reward function and weights are in Table 4 .For estimator, similar to Nahrendra et al. [9] , in our approach, we require an encoder-decoder framework where the decoder generates o ′ t+1 to compute the encoder's loss explicitly. The specific method involves calculating the mean-squared-error (MSE) loss between the estimated body velocity v ′ t and the ground truth v t from the simulator. Additionally, we employ the standard β-VAE loss to compute the loss for decoder result o ′ t+1 and the ground truth o t+1 , and sum these two losses. To further increase the sim-to-real robustness of the learned policy, we also utilize adaptive bootstrapping (AdaBoot), a technique similar to that used in Nahrendra et al. [9] , to stabilize the estimator's updates.L est = L MSE + L VAE L MSE = MSE(v t , v ′ t ), L VAE = MSE(o t+1 , o ′ t+1 ) + βKL(g(z t |o H t+1 )∥p(z t ))Two-phase training: The training is divided into two phases. In the first phase, the reference motion is incorporated as a reward to encourage the robot to learn to move quickly according to a standard sinusoidal gait. In the second phase, the reward for the reference motion is removed to allow the robot to learn better strategies to adapt to the set terrain. The terrain includes planes, steps, slopes, and stairs. During the first phase, the difficulty of the terrain is relatively low, with more simple terrains and lower stairs height. However, in the second phase, we increase the difficulty of the terrain, raising the proportion of high-difficulty terrains such as steps and stairs to 50% with higher stairs height. In this way, our robot can step up to 16cm high stairs at most.During training, we observed that robots tend to walk in place on dangerously uneven terrains, even with high linear velocity tracking rewards. To address this, inspired by Margolis et al. [8] , we implemented a curriculum learning strategy for velocity, which means the robot initially focuses on learning to follow the reference gait phase at low speeds. Once the average velocity reward exceeds a threshold r = 0.75, we gradually increase the linear velocity commands by △v x,y = 0.05 m/s and angular velocity commands by △ω z = 0.05 rad/s, enabling effective speed tracking.We have identified that single-frame observation augmented with estimator-predicted information [9] [24], while capable of tracking velocity commands in simulation, often result in joint tremors and suboptimal performance on varied terrains. Conversely, relying on multi-frame(20) observation without estimator [29] integration leads to slow movement and weak action robustness. To address these issues, we developed our proprietary "Multi-Observation with Estimation" method to train policies. This approach effectively tracks velocity commands, navigates through diverse terrains, and demonstrates excellent robustness and adaptability when deployed on actual hardware. It seamlessly transfers the policy from simulation to the real robot.To accomplish the sim-to-real transfer, we designed various randomizations and delays for instructions. Firstly, we applied randomizations to the observation and the robot's properties for domain randomization. In order to mimic the effects of real-world deployment as closely as possible, we introduced a random delay of up to 15% in both action and torque to help bridge the sim2real gap. Besides, We adjusted the center of gravity offset and link mass to adapt to the gap in the real environment. Additionally, we apply a random force to each link every time an action is outputted. These measures have significantly improved the situation where the robot's hardware undergoes some design adjustments, which necessitated retraining with past methods. A detailed summary of the domain randomization and the execution delays is presented in Appendix Table 5 . 
experimental results: 
 We conduct Numerous comparative and ablation studies in simulation and reality. This section will provide a detailed account of the specific experimental designs and the final comparative results.The comparative experiments we performed are as follows: Baseline: a no estimator policy with frame stacks observation; DreamWaQ: the reproduced method presented in Nahrendra et al. [9] ; A-RMA: the reproduced method presented in Kumar et al. Reference motion: In experiment Ours w/o Ref, the significant differences in the contact forces between the robot's two feet can cause considerable strain on its motors and links, resulting in significant wear and tear, as shown in Fig. 3 , the figure illustrates the contact forces for both the left and right feet, highlighting a stark contrast between the policy executed with reference motion and one without it. The latter demonstrates a substantial variability in force application, an issue absent in our policy. The absence of reference motion can result in imbalances, potentially leading to motor damage in real robots, whereas our approach ensures consistent stability and motor protection. Additionally, the uncoordinated movements significantly increase the likelihood of the robot falling, underscoring an urgent need for synchronized actions and balanced force distribution to enhance reliability and prevent such incidents. Consequently, we omitted this comparison in the following analysis. We conducted experiments comparing the robot's performance between Phase one and two. In the first experiment, the robot was tested for its ability to climb stairs at a speed of 1 m/s, with success defined as clearing a six-step staircase. The second experiment evaluated the robot's ability to traverse slopes with varying gradients at the same speed. Our approach, not constrained by predefined reference motion, showed effective traversal of steps and slopes, as detailed in Table 1 .In contrast, other strategies exhibited instability on terrains deviating from reference motion parameters. While alternative methods struggled to climb steps over 5 cm high, our system maintained an 85% success rate on 10 cm steps. Similar proficiency was observed in traversing slopes steeper than 15 degrees, with Phase two being the only one to navigate them effortlessly. Estimator loss: The estimator loss plays a crucial role in assessing the robot's ability to accurately predict linear velocity and latent vectors, which significantly aids in our ability to follow speeds and traverse various terrains effectively. We recorded all the loss curves of the methods mentioned above. The result is shown in Fig. 4 . It shows that our Multi-Observation with Estimation policy demonstrates superior predictive capabilities for v t and z t compared to DreamWaQ, which relies solely on single observation within its actor policy.We conducted a comprehensive empirical analysis comparing our proposed strategy alongside DreamWaQ and a standard baseline across a continuum of simulated and actual robotic platforms. This battery of tests was designed to rigorously evaluate the adaptability and versatility of each algorithm in diverse environmental conditions. Furthermore, we subjected the algorithms to a series of perturbations that simulate the unpredictable nature of real-world conditions, thereby evaluating their resilience and robustness. This final testing phase is instrumental in gauging the practical applicability and reliability of the algorithms under non-ideal circumstances.While all robotic systems encountered increased challenges on low-µ surfaces, precipitating slippage and an elevated risk of collapse, our strategy adeptly traversed the array of terrains as shown in Table 2 In the context of disturbance rejection trials, the intrinsic instability of the alternative algorithms led to an almost inevitable collapse under lateral forces and suboptimal success rates from anterior and posterior forces. Conversely, our strategy consistently exhibited unparalleled stability, withstanding forces applied across all directional vectors, resulting in Table 3 . Appendix B shows that our robotic system has demonstrated the ability to traverse various terrains and obstacles. Commands Tracking: The robotic system's primary objective is to track the velocity commands issued to it accurately. Consequently, the efficacy of commands tracking emerges as a pivotal metric for evaluating the success of our experimental endeavors. This performance is vividly illustrated in Fig. 5 . Observation have elucidated that when faced with arduous terrains, including stairs with heights exceeding 6 cm or a 15-degree inclined uneven terrain within our training set. Baseline, DreamWaQ and A-RMA tend to prioritize safety, consequently adopting a more cautious and slowed pace. Furthermore, the variant of our approach that forgoes a commands curriculum exhibits a di- minished capacity for velocity tracking, potentially due to the vast exploration space overwhelming the actor's learning process. Additionally, ours without LVAE have indicated that the Euclidean distance of the observations may not accurately represent the discrepancy from the target, resulting in suboptimal walking posture and inadequate speed following. In striking contrast, our refined approach showcases an enhanced aptitude for swiftly and adeptly acquiring and adjusting to velocity directives. Consequently, our strategy must incorporate a commands curriculum to ensure proficient learning and adaptation to the velocity commands. 
limitation: 
 Our policy, though effective in its intended tasks, is limited by the lack of perception for terrain movements. To address this, we plan to integrate perception systems into our future work, thereby enhancing our strategies' adaptability to more complex challenges and expanding the operational capabilities of our robotic systems. Additionally, we recognize our approach has limitations, particularly in parameter tuning for diverse robots, which is a complex engineering task. Adjusting reward parameters based on a robot's size and carefully selecting domain randomization parameters are essential; for instance, center of mass randomization exceeding 0.06 cubic meters can severely impact real-world performance. Time-related parameters need finely tuned to match the anatomical proportions of different robots to ensure optimal dynamic responses. To address these challenges and improve our strategies' adaptability, we plan to investigate solutions such as imitation learning, aiming to enhance our system's robustness and applicability across various robotic platforms. 
conclusion: 
 In this paper, we have presented a comprehensive study on the enhancement of steady humanoid robot locomotion, achieved through the refinement of estimator and actor policies. This research has broad implications, establishing a foundation for versatile and adaptive locomotion across varied terrains and obstacles through a two-phase training method that expands the robot's gait capabilities. Additionally, the study demonstrates an exceptional velocity tracking capability with a velocity commands curriculum that underscores the system's precision and responsiveness. 
 
 Appendix: We present our complete algorithmic process in the following algorithm, where four networks are continuously updated through sampling from the simulation based on the process outlined below. This section presents a series of experimental images that vividly illustrate our algorithm's intricate implementation details and robust generalization capabilities across various initial poses for robotic walking tasks, shown in the Fig. 6 . We have achieved smooth locomotion across a spectrum of initial postures, as evidenced by the images depicting the robot's initial stance. Within the simulated environment, the Fig. 7 display the robot's upright and stable gait, underscoring the algorithm's exceptional precision in control within the virtual platform. We have further substantiated the algorithm's reliability, shown in Fig. 8 , and practical utility by advancing to real-world scenarios. The deployment of our algorithm on an actual robot has endowed it with the ability to navigate in various straight-legged postures, as illustrated by the images portraying the robot's commendable equilibrium and stability. This rigorous process is a testament to the algorithm's resilience and adaptability. These outcomes furnish compelling validation for our algorithm's ongoing refinement and application, simultaneously presenting innovative perspectives and methodologies that hold significant promise for future research within related domains. We conducted multiple experiments on the actual machine, testing our policy on various terrains, including grasslands, wire-strewn ground, slopes, and stairs. Our robot's traversal across grasslands is a testament to its ability to handle the soft and unpredictable ground, while its passage through areas with wireless showcases its resilience against obstacles that could impede movement. The robot's ascent on slopes highlights its dynamic balance and the algorithm's capacity to adjust to inclines that require precise foot placement and torque control. Most notably, the robot's ability to climb and descend stairs indicates our algorithm's advanced control mechanisms, ensuring that each step is calculated for maximum efficiency and safety. The images reveal a robot that is not just mobile but one that can adapt to and stabilize on a wide array of environmental conditions, thereby proving the algorithm's robustness and stability in a comprehensive sim-to-real context.These visual records are more than just demonstrations of our robot's physical capabilities; they are evidence of the sophisticated algorithms that enable it to interact intelligently with its surroundings, providing a solid foundation for further research and development in robotics.Figure 9 : The sequence of images presented illustrates the diverse terrains our robot navigates with proficiency, ranging from the soft contours of grasslands to the challenging unevenness of wirestrewn areas, the inclines of varying gradients, and the ascents and descents of stairs. Each scenario, captured in a vertical progression from top to bottom, demonstrates not only the robot's adaptability but also its ability to maintain equilibrium on surfaces that demand different levels of traction and stability. 


Decision: This paper introduced a two-phase training approach to blind biped locomotion. They used an estimator network to predict velocities and map observation history to latent states. They then used these latent states to condition an RL policy trained in large-scale simulation environments and transfer the policy to the physical robot in real-world terrains. This paper received mixed initial ratings, with one weak reject and two weak accepts. The reviewers pointed out the strengths of this paper, particularly 1) the use of curriculum learning is an interesting idea, and 2) the real-robot deployments showed the robustness and practicality of this approach. In the meantime, they also mentioned several weaknesses, including 1) limited novelty: this work seems to be an incremental improvement over previous work, particularly DreamWaQ, and 2) experimental validation: comparisons to relevant methods, such as A-RMA, are missing and the experiments lack thorough ablation studies to validate the importance of specific components of the algorithm. In addition, all three reviewers found that the limitations were not well addressed. Overall, the AC thinks this work has good promises, but the reviewers' criticisms are valid and must be addressed before this paper passes the bar of publication.

**Post-rebuttal update**

The authors successfully addressed the concerns raised by the three reviewers. Toward the end of the rebuttal period, the reviewers had three unanimously positive recommendations. The AC felt pleased to see the positive changes, which improved the quality of this paper, and recommended accepting it for CoRL.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper extends studies the problem of robust blind biped locomotion. They extend prior work (DreamWaq) - an estimator network is trained to estimate velocities and map observation history to latent latent states. These condition a downstream policy which is trained via RL in simulation with domain randomizaton. They demonstrate locomotion on a real robot on small stairs, slopes, grass and flat ground with some perturbances.
main_review: main_review: This paper has nice real-world results, although I think prior work exists which shows superior results albeit on different hardware. The method itself seems to be a minor change to DreamWaq (1) adding velocity prediction to the estimator and (2) using observation history instead of just a single observation. This implies that the method has limited novelty. Moreover, the authors have not compared against some important baselines, and it is unclear to me what the benefit of this method is over the ones proposed in prior work. 

- Why have the authors not compared against A-RMA [1], which also uses privileged observations in sim, and has shown results on real biped robots?
- How many frame stacks does the baseline get? Did the authors also try RNNs which have a much longer context length and have shown success in prior work [2]? 

I would encourage the authors to position their paper better in relation to prior work and add more thorough evaluation.

[1] Kumar, Ashish, et al. "Adapting rapid motor adaptation for bipedal robots." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.
[2] Siekmann, Jonah, et al. "Blind bipedal stair traversal via sim-to-real reinforcement learning." arXiv preprint arXiv:2105.08328 (2021).
questions: questions: - In the video results, when going up / down the steps it is hard to tell whether the operator is applying a stabilizing force to the robot through the cord. Is the robot able to do the stairs reliably without support? Can the authors generate a video where the robot is held by a loose cord so that no force is being applied on it?
- Is the only difference from DreamWaQ that the estimator gets a history of observations instead of a single observation?
limitations: limitations: 1
review_summary: review_summary: The paper has nice real world results, however, I vote to weak reject this paper because of limited novelty in both results and method + insufficient comparison to prior work
title: title: Strong results but missing ablations
paper_summary: paper_summary: The paper proposes a two phase RL training in simulation with zero shot transfer to real for humanoid locomotion. The method relies on a learned estimator and a curriculum in terrain (two phase training) as well as velocity sampling to improve sim based RL learning. The proposed method shows high robustness in the real world and quantitatively superior performance to baselines.
main_review: main_review: - The proposed method extends existing method in an important way to achieve superior performance compared to baseline methods. 
- The paper will likely have a reasonable impact given the increased importance of humanoid research and that locomotion and stability is central to any downstream research on humanoids. 
- The direction taken in the paper of learning in simulation via RL and then zeroshot transfer to real for locomotion in itself is not particularly new, however, there is no compelling recipe widely used for humanoids, hence making the contribution of this paper important.     
- The push recovery in the video results is the most impressive and I would suggest the authors show more video and simulation based quantitative results of push recovery which is known to be a harder task for bipeds than quadrupeds.
- The paper is well written, easy to follow and understand. The figures are done well and its clear to understand the point of the method figure and the result tables. 
- Some ablations that validate the importance of the algorithmic design are either missing from the comparison tables, or not clearly explained. More details below on this. 
- The paper is lacking a limitations section and only superficially mentions "no vision" as the only limitation. A more detailed discussion on the limits of the current approach (like the amount of simulation tuning required, the need to pick parameters of domain randomization which cannot be too aggressive, stability and reproducibility of the proposed method), would help a lot.
questions: questions: - Estimator loss has two components. Could we ablate the importance of LVAE loss?
- Figure 4: Could the authors define FHM used in the legend (point to where in the paper its defined)?
- Are the terrains included in Phase one of the training or only in phase 2? If only included in phase 1, authors should include a comparison with terrains included in Phase 1. 
- L171: The sentence is not clear, based on what finding?
- L237-L239: “This refined passage employs a more formal and academic 238 tone, with a clear and structured explanation of the experimental setup, observation, and the comparative efficacy of the proposed method against existing ones.” Has the passage been written by a language model?
- Please address the limitations more thoroughly with discussion on things like: amount of simulation tuning required, the need to pick parameters of domain randomization which cannot be too aggressive, stability and reproducibility of the proposed method, etc.
limitations: limitations: 1
review_summary: review_summary: The proposed method shows very compelling results in the real world on a humanoid robot. The experiments could be improved to make them more thorough
title: title: Review
paper_summary: paper_summary: The paper presents a method for humanoid robots to adapt over different terrains through the use of two-phased training.
main_review: main_review: The paper presents a curriculum-like approach for training humanoid policies. The paper presents impressive results that are deployed on the physical robot. In addition, the organization of the work was well done and made it easy to follow the paper. The background literature motivates the work well and provides a nice overview of previous approaches and where this work fits. 

In terms of weaknesses, there are a few that the paper should address. First, how does the paper change with respect to the PD controller and variability on the controller when transferred to the physical robot? As practitioners it would be good to see how much low-level control depends on the success of the approach. Some intuition for what the specific change of curriculum training should be in simulation. For instance, how quick of a change in difficulty would the approach fail or the learned policy become unstable or lose performance? This would be interesting as a means to provide intuition on how quickly the policy can adapt as difficulty is increased.
questions: questions: Failure modes for the approach are the main questions that would improve the contribution of this work. In particular, where does the approach fail and why is something that is missing from this work.
limitations: limitations: 1
review_summary: review_summary: The work is missing a clear overview of limitations which would improve the quality of the paper.