## 2 Double-Layer Reinforcement Learning Algorithm Design
### The Design Structure of the Double-Layer Structure
### Experimental Results and Analysis

To verify the effectiveness of the method proposed in this study, this study chose to let the robot walk forward for 10 s and compared the effects of different models based on the distance the robot walked forward. During the training process, the different effects when using a single model and using four specially designed models for training were compared. The single model selected was designed for a speed of 2.5 m/s during training. No conditions for stopping training were set, and the training would end after 500 episodes. The characteristics of the reward changing with the episode during training are shown in the Figure 4. From these figures, it is observable that, as training progresses, the agents gradually receive increasing rewards, indicating that the agents are progressively optimizing their adaptation to the environment through learning. At the same time, we conducted 10 experiments in random terrains and took the average, and compared the distances walked by the robot using six different methods within 10 s. The results are shown in the table. It can be seen from the table that the proposed method takes into account the fast moving speed while maintaining stability by adjusting the walking speed in different terrains. The specific results are shown in Table 4.

1. Using a fixed speed command of 0.5 m/s. Figure 4: Experimental result comparison. The above figure shows the results of the method proposed in this paper, and the lower figure shows the training results of the single model at 2.5 m/s. The horizontal axis in these figures represents the number of training rounds, while the vertical axis shows the average reward obtained by the agents in each episode. The darker part in the figures is the moving average reward in 50 episodes, and the lighter part is the reward of each episode. Higher rewards represent that the robot may walk at a faster speed. Using a fixed speed command of 1.5 m/s. Using a fixed speed command of 2.5 m/s. Using a fixed speed command of 3.5 m/s. Using a 2.5 m/s motion model and dynamically adjusting the speed using the speed generation module. The double-layer reinforcement learning algorithm proposed in this paper. To more intuitively reflect the advantages of the dynamic adjustment method proposed in this paper, this paper compares the gait differences between using the fixed speed method of 3.5 m/s and the method proposed in this paper when climbing an obstacle with a height of 0.5 m. The result is shown in Figure 5. It can be clearly seen that the adaptability of the method proposed in this paper to complex terrains is much better than the method of choosing a fixed speed, and it has the longest walking distance among all the methods. ## 4 Discussion

This paper proposes a double-layer control algorithm based on deep reinforcement learning to address the motion control problem of quadruped robots in complex terrains, \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Control Method Description** & **Distance (m)** \\ \hline
1 & Using 0.5 m/s constant speed command and corresponding model & 4.8 \\
2 & Using 1.5 m/s constant speed command and corresponding model & 13.8 \\
3 & Using 2.5 m/s constant speed command and corresponding model & 21.1 \\
4 & Using 3.5 m/s constant speed command and corresponding model & 20.5 \\
5 & Using 2.5 m/s motion model with speed generation module to dynamically adjust speed command & 24.2 \\
6 & Proposed dual-layer reinforcement learning algorithm & 25.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of different methods for controlling robot’s 10-second movement distance. Figure 5: Gait differences between using the fixed speed method of 3.5 m/s and the method proposed in this paper. Both methods are tested on the same terrain and the same step obstacle. The two rows of pictures show the robot walking methods based on the two models, respectively, and the process of each model is from number 1 to 3.