## 2 Double-Layer Reinforcement Learning Algorithm Design
### The Design Structure of the Double-Layer Structure
Through the real-time fusion of the above multi-source sensor data, the robot can accurately perceive its own state and the surrounding environment, thereby performing stable and efficient movement in complex terrains. In this study, in the process of determining the robot's speed command based on DDQN and the process of generating the robot's actions based on PPO, the robot's observations use all the observation information. ### Speed Command Generation Based on DDQN

To achieve efficient motion control of the quadruped robot in complex terrains, this paper proposes a speed command generation method based on DDQN. In complex terrains, a single speed command cannot maintain good terrain adaptability. In rough terrain environments, an overly fast speed command will make the robot unable to maintain balance, while in flat terrain environments, an overly slow speed command will make the robot walk slowly. Therefore, it is necessary to dynamically adjust the speed, and DDQN can effectively judge the terrain and output corresponding commands. The Q-network structure in the structure is shown in Figure 2:

Figure 2: The structure of the speed command generation network, where the input is 235-dimensional observation information and the output is the speed command. The numbers in the figure are the quantities of neurons in each layer. In previous verifications, even on flat ground, the robot has difficulty walking at a speed exceeding 3.5 m/s. Therefore, the output of the network is four discrete speeds: 0.5, 1.5, 2.5, and 3.5, with a unit of m/s. Since the task of the DDQN model is to maintain the fastest speed as possible, the reward function of this task is set as Equation (4), where the reward is designed to encourage actions that maximize speed while penalizing unstable or inefficient movements. This structure helps the DDQN model prioritize behaviors that achieve high velocity while maintaining balance and control, ensuring the robot performs optimally in dynamic environments. \[R=\exp(-\frac{\sum_{i=1}^{2}(3.5-v_{i,\text{actual}})^{2}}{\sigma_{\text{ tracking}}}) \tag{4}\]

where \(v_{x}\) is the horizontal speed of the robot walking along its own orientation, and \(\sigma_{\text{tracking}}\) is a parameter that determines the reward difference corresponding to the difference in different speeds. Taking the speed command as the input, the corresponding action planning is carried out using the control module based on PPO. In each training step, the robot selects the appropriate speed command by perceiving the changes in the environment and updates the Q-value function according to the actual walking result. To avoid the model overfitting to a specific terrain during training, this paper uses a variety of different terrain types for mixed training during training, including flat ground, slopes, steps, and complex gravel roads, etc. ### Reward Function

In reinforcement learning, the reward function is one of the core elements guiding the agent's learning process. It provides positive or negative feedback, helping the agent make optimal decisions in its environment. The reward function design is key in this algorithm. It directly affects the robot's stable walking in complex terrains. Since different factors contribute variably to the robot's overall performance, a series of weighted reward functions are used to guide the robot's behavior in both positive and negative ways [33]. To determine the most effective combination of weights for these reward functions, we employed the grid search method, a widely used hyperparameter tuning technique. Grid search involves defining a grid of possible values for each weight and systematically evaluating the performance of the model for every possible combination of these values. In our case, we defined a range of candidate values for each reward component, such as penalties for instability or incentives for speed, and tested the algorithm across these values.