## 2 Double-Layer Reinforcement Learning Algorithm Design
### The Design Structure of the Double-Layer Structure
\begin{table}
\begin{tabular}{l c c} \hline
**Reward Term** & **Weight** & **Function** \\ \hline tracking\_lin\_vel & 1.0 & \(\exp\left(-\frac{(\mathbf{v}_{\text{max}}^{y}-\mathbf{v}_{\text{max}}^{y})^ {2}}{\sigma_{\text{relax}}}\right)\) \\ tracking\_ang\_vel & 0.5 & \(\exp\left(-\frac{(\mathbf{v}_{\text{max}}^{i}-\mathbf{v}_{\text{max}}^{i})^ {2}}{\sigma_{\text{relax}}}\right)\) \\ lin\_vel\_z & \(-2.0\) & \(v_{\text{base}}^{z}\cdot v_{\text{base}}^{z}\) \\ ang\_vel\_xy & \(-0.05\) & \(\sum\left(\mathbf{a}_{\text{base}}^{x}\right)^{2}+\left(\mathbf{v}_{\text{ base}}^{y}\right)^{2}\) \\ torques & \(-0.00001\) & \(\sum\tau^{2}\) \\ dof\_acc & \(-2.5\times 10^{-7}\) & \(\sum\left(\frac{d_{\text{prev}}-\hat{d}}{\Delta t}\right)^{2}\) \\ feet\_air\_time & 1.0 & \(\sum(t_{\text{air}}-0.5)\times f_{\text{contact}}\times(\|\mathbf{v}_{\text{cmd }}^{\text{xy}}\|>0.1)\) \\ collision & \(-1.0\) & \(\sum(1\times(\|\mathbf{F}_{\text{contact}}\|>0.1))\) \\ action\_rate & \(-0.01\) & \(\sum(\mathbf{a}_{\text{prev}}-\mathbf{a})^{2}\) \\ \hline \end{tabular}
\end{table} 
Table 1: Reward function and weight factors. \begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Speed** & **Random Speed Command (m/s)** & **Terrain Slope** & **Platform Size (m)** & **Obstacles Quantity** & **Obstacles Size (m)** & **Step Size (m)** \\ \hline
0.5 & \([-1,1]\) & 0.39 & 1 & 20 & 0.5–1 & 0.30 & 0.2 \\ \hline
1.5 & \([-1,2]\) & 0.30 & 1 & 20 & 0.3–0.8 & 0.56 & 0.1 \\ \hline
2.5 & \([-1,3]\) & 0.15 & 1 & 20 & 0.2–0.5 & 1.00 & 0.1 \\ \hline
3.5 & \([-1,3.5]\) & 0 During the training process, the duration of each training episode is 10 s, while the frequency of the command output structure based on DDQN is 2.5 Hz. The output frequency of the position output structure based on PPO is 50 Hz, meaning that the speed command is re-determined every 20 position outputs. At the same time, between every two adjacent positions, the PD controller performs five torque control outputs. This study uses a \(7\times 7\) terrain grid, with each terrain having a size of \(8\times 8\) m. The entire terrain layout is shown in the upper part of Figure 3. The specific size parameters of the terrain are obtained through random sampling of the parameters used in training the four different models. These four types of terrains, as illustrated in the lower part of Figure 3, are concave slope, convex slope, cuboid obstacle, and stepping with gap, with proportions of 2:3:2:3. At the beginning of each training cycle, the robot's initial position is randomly set at the center of one of the terrains, and it moves forward in the direction it faces. \begin{table}
\begin{tabular}{l c r r} \hline
**Joint** & **Dimensions (m)** & **Lower Limit (rad)** & **Upper Limit (rad)** \\ \hline Hip Joint (Front Right) & \(0.28\times 0.16\times 0.11\) & \(-1.047\) & \(1.047\) \\ Thigh Joint (Front Right) & \(0.213\times 0.0245\times 0.034\) & \(-0.663\) & \(2.967\) \\ Calf Joint (Front Right) & \(0.213\times 0.016\times 0.016\) & \(-2.723\) & \(-0.838\) \\ Hip Joint (Front Left) & Radius: 0.01 & \(-1.047\) & \(1.047\) \\ Thigh Joint (Front Left) & \(0.213\times 0.0245\times 0.034\) & \(-0.663\) & \(2.967\) \\ Calf Joint (Front Left) & \(0.213\times 0.016\times 0.016\) & \(-2.723\) & \(-0.838\) \\ Hip Joint (Back Right) & Radius: 0.01 & \(-1.047\) & \(1.047\) \\ Thigh Joint (Back Right) & \(0.213\times 0.0245\times 0.034\) & \(-0.663\) & \(2.967\) \\ Calf Joint (Back Right) & \(0.213\times 0.016\times 0.016\) & \(-2.723\) & \(-0.838\) \\ Hip Joint (Back Left) & Radius: 0.01 & \(-1.047\) & \(1.047\) \\ Thigh Joint (Back Left) & \(0.213\times 0.0245\times 0.034\) & \(-0.663\) & \(2.967\) \\ Calf Joint (Back Left) & \(0.213\times 0.016\times 0.016\) & \(-2.723\) & \(-0.838\) \\ \hline \end{tabular}
\end{table} 
Table 3: Joint limits and dimensions. Figure 3: The entire terrain layout used for training (**upper**) and the specific terrain types (**lower**).