## 2 Double-Layer Reinforcement Learning Algorithm Design
### The Design Structure of the Double-Layer Structure
**A1**–**A3** is the adaptive speed adjustment method proposed in this paper, and **B1**–**B3** is the fixed speed method of 3.5 m/s. In the method proposed in this paper, the robot walks at a speed of 3.5 m/s before and after climbing the obstacle, and the speed is adjusted to 1.5 m/s when climbing the obstacle in order to maintain stability and pass the obstacle at a high speed. In the fixed speed method, the robot is tripped by the step and loses balance due to the excessive moving speed. especially the balance between speed and stability. The key findings and contributions of this study are summarized as follows:

1. The algorithm combines DDQN and PPO to dynamically adjust the walking speed based on real-time terrain information and robot states, ensuring efficient and stable movement across different terrains. 2. A detailed design of the double-layer reinforcement learning algorithm is provided, including the speed command generation and motion trajectory control modules. 3. Experimental results on the NVIDIA Isaac Gym simulation platform demonstrate that the algorithm significantly improves both the walking distance and stability of the robot, particularly on dynamic and irregular terrains. 4. The innovative combination of DDQN and PPO presents a widely adaptable control method, validated through comprehensive experiments. 5. Future research can focus on optimizing the real-time performance of the algorithm, expanding its application to multi-task scenarios, and testing its effectiveness in real-world environments. Conceptualization, Y.Z. ; methodology, Y.Z. ; software, Y.Z. and J.Z. ; validation, Y.Z. and H.S. ([PERSON]); formal analysis, J.Z. ; investigation, Y.Z. ; resources, K.H. ; data curation, Y.Z. ; writing--original draft preparation, Y.Z. ; writing--review and editing, H.S. ([PERSON]); visualization, Y.Z. ; supervision, K.H. ; project administration, K.H. ; funding acquisition, K.H. All authors have read and agreed to the published version of the manuscript. This study was conducted with the support of the Information, Production and Systems Research Center, Waseda University; Future Robotics Organization, Waseda University, and as part of the humanoid project at the Humanoid Robotics Institute, Waseda University. This work was supported by JSPS KAKENHI Grant Number JP21H05055 and a Waseda University Grant for Special Research Projects (Project number: 2024C-188). This work was also supported by JST SPRING, Grant Number JPMJSP2128. Not applicable. The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author. The authors declare no conflicts of interest. The following abbreviations are used in this manuscript: \begin{tabular}{l l} DDQN & Deep Double Q-Network \\ PPO & Proximal Policy Optimization \\ MPC & Model Predictive Control \\ ZMP & Zero Moment Point \\ DRL & Deep Reinforcement Learning \\ DQN & Deep Q-Network \\ MDP & Markov Decision Process \\ RL & Reinforcement Learning \\ \end{tabular} 

## References

* [1] [PERSON] [PERSON]; [PERSON] [PERSON]; [PERSON] [PERSON] Comparative anatomy of quadruped robots and animals: A review. _Adv. Robot._**2022**, 36, 612-630. [CrossRef]
* [2] [PERSON]; [PERSON]; [PERSON]; [PERSON] Bigdog, the rough-terrain quadruped robot. _Ijac Proc. Vol._**2008**, _41_, 10822-10825. [CrossRef]
* [3] [PERSON]; [PERSON] [PERSON]; [PERSON] [PERSON]; [PERSON] [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON] Design of the hydraulically actuated, torque-controlled quadruped robot HyQ2 Max. _Ieee/Asme Trans. Medatron._**2016**, _22_, 635-646. [CrossRef]* [PERSON] et al. (2019) [PERSON]; [PERSON]; [PERSON] Probabilistic inferences on quadruped robots: An experimental comparison. In Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China, 6-8 December 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1892-1898. * [PERSON] and [PERSON] (2016) [PERSON]; [PERSON] Towards a walking, turning, and jumping quadruped robot with compliant mechanisms.