## 2 Double-Layer Reinforcement Learning Algorithm Design
### The Design Structure of the Double-Layer Structure  
The double-layer reinforcement learning algorithm framework proposed in this paper combines DDQN and PPO to improve the speed adjustment ability and stability of the quadruped robot Unitree Go1 in complex terrains. In the initial stage of the algorithm, it is necessary to initialize the parameters of the DDQN and PPO models, including setting the terrain information and the state of the robot as observation variables, and initializing the robot environment and the simulation platform Isaac Gym. Then, the algorithm collects the current terrain information and the state of the robot, such as position, speed, and posture, and inputs this data into the DDQN network to generate the walking speed command. The speed command serves as a high-level instruction, determining how fast the robot should move based on its environment and internal status. It provides a target speed that ensures the robot maintains balance and avoids obstacles, while optimizing its overall movement efficiency. Subsequently, the PPO model takes this speed command, along with the current observation information, to generate specific robot actions, such as joint angles and gaits. These actions control the precise movement of the robot's legs and joints. The PPO model continuously refines and adjusts these actions through policy optimization, helping the robot adapt to uneven or complex terrain while maintaining stability and smooth locomotion. After the robot executes the actions generated by PPO, the system updates its state, and collects new observation information and feeds it back to the DDQN and PPO models for further optimization and adjustment. Through training and update, the DDQN algorithm updates the Q value to optimize the walking speed strategy, and the PPO algorithm updates the policy network to optimize the action control strategy. The above steps are executed continuously in a loop until the model converges or reaches the predetermined number of training times. The step can be illustrated as Algorithm 2. ```
1:Initialization:
2:Initialize parameters for DDQN and PPO models
3:Set terrain information and robot state as observations
4:Initialize the environment and simulation platform (Isaac Gym) for Unitree Go1
5:loop
6:Terrain Information and State Observation:
7:Collect current terrain information and robot state (position, speed, posture, etc.) 8:Input observation information into the DDQN network
9:DDQN Speed Command Generation:
10:DDQN network outputs a walking speed command based on the input observations
11:PPO Action Control:
12:PPO model generates specific robot actions (joint angles, gait, etc.) based on the speed command and current observations
13:PPO model continuously adjusts actions through policy optimization to adapt to complex terrains and maintain stability
14:Execute Action and Update State:
15:Execute actions generated by PPO and update the robot's state in the environment
16:Collect new observations and feed back to DDQN and PPO models
17:Training and Updating:
18:Use DDQN algorithm to update Q-values and optimize the walking speed strategy
19:Use PPO algorithm to update the policy network and optimize the action control strategy
20:if convergence or maximum training iterations reached then
21:Model Validation:
22:Perform detailed validation of the trained model in Isaac Gym
23:Evaluate the model's speed adjustment ability and stability in various terrains
24:Exit loop
25:endif
26:endloop
27:Performance Evaluation:
28:Assess the model's performance, ensuring high walking speed and stability across different terrainsThe schematic diagram of the method proposed in this paper can be represented by Figure 1. The dual-layer structure offers several advantages by separating high-level decision making from low-level control, improving both learning efficiency and motion precision.