## 3 Locomotion Learning Algorithm
### Deep Reinforcement Learning Algorithm  
In this work, DRL is used to train the rhythm controller parameters to ensure the stable motion of the quadruped robot under different conditions. The DRL algorithm used in this work is the Soft Actor-Critic (SAC) [32; 33]. The learning process of a quadruped robot can be regarded as the Markov decision process (MDP), which can be represented by a tuple \((S,A,P,r,\gamma)\), where \(S=\{s_{1},s_{2},\cdots,s_{T}\}\) is the set of states, \(A=\{a_{1},a_{2},\cdots,a_{T}\}\) represents the set of actions, \(P\) is the state transition probability from the current state to the next state, \(r\) is the reward value for each step and \(\gamma\in[0,1]\) represents the discount factor. The usual mechanism for reinforcement learning is to find a strategy that maximizes cumulative reward; its expression is shown in Equation (6):  
\[\pi^{*}=\arg\max_{\pi}\mathrm{E}_{\pi}[\sum_{t}r(s_{t},a_{t})], \tag{6}\]  
where \(\pi\) is the policy, \(\pi^{*}\) is the optimal policy, E is the expected reward, \(t\) is time and \(r(s_{t},a_{t})\) is the reward for choosing action \(a_{t}\) in state \(s_{t}\). The SAC algorithm, of course, also follows this rule, but the difference is that the SAC algorithm incorporates an entropy, which, in addition to maximizing the cumulative reward, makes the strategy more random; the expression is then shown in Equation (7):  
\[\pi^{*}=\arg\max_{\pi}\mathrm{E}_{\pi}[\sum_{t}r(s_{t},a_{t})+\alpha H(\pi( \cdot|s_{t}))], \tag{7}\]  
where \(H(\pi(\cdot|s_{t}))\) is the entropy representing the degree of randomness of policy \(\pi\) in state \(s\) and \(\alpha\) is the regularization coefficient, which mainly controls the importance of the entropy. In this work, the selected state space includes the position of the center of gravity of the robot, the motion speed of the robot on the X, Y and Z axes, the pitch angle, yaw angle, roll angle and the rotation angle of the eight joints during the robot's movement; so, the state space can be expressed as \(\{v_{x},v_{y},v_{z},q_{pitch},q_{yaw},q_{roll},\xi_{1},\cdots,\xi_{8}\}\). From Equation (3), we know that the parameters of the rhythm controller include \(A_{k},A_{h},T,\varphi_{0},\varphi_{1},\varphi_{2},\varphi_{3},\varphi_{4}, \varphi_{5},\varphi_{6},\varphi_{7}\). Since the gait used in this work is the trot gait, \(\varphi_{0}=\varphi_{6}=0,\varphi_{1}=\varphi_{7},\varphi_{2}=\varphi_{4}= \pi,\varphi_{3}=\varphi_{5}\), the action space is \(a=[A_{k},A_{h},T,\varphi_{1},\varphi_{3}]\), which directly passes to the cosine rhythm controller to ensure that the controller works correctly, where \(A_{k}\in[-0.5,0.5],\)\(A_{h}\in[-0.5,0.5],\)\(T\in(0,2],\varphi_{1}\in[-2\pi,2\pi],\varphi_{3}\in[-2\pi,2\pi]\). In order to speed up the learning process, we set a penalty value, as long as the robot falls, where a punishment \(r_{p}\) will be given, as shown in Equation (8), and the current round of training is then over. \[r_{p}=-100. \tag{8}\]  
At the same time, in order to encourage the robot to move, a reward \(r_{s}\) is given to the robot for every step of movement, as shown in Equation (9):  
\[r_{s}=10. \tag{9}\]  
In order to encourage the robot to move forward, a forward reward is given to the robot. In this work, the forward direction of the robot is the Y direction (in the world coordinate system), so the forward reward function is  
\[r_{y}=w_{1}(y_{now}-y_{last}), \tag{10}\]where \(y_{now}\) is the position of the robot in the Y direction at the current moment, \(y_{last}\) is the position of the robot in the Y direction at the previous moment and \(w_{1}\) is the forward reward function weight. Similarly, a reward function is given to the robot for the X direction, aiming to reduce the yaw of the robot; the reward function is shown in Equation (11):  
\[r_{x}=-w_{2}|x_{now}-x_{last}|, \tag{11}\]  
where \(r_{x}\) is the reward for yawing, \(x_{now}\) is the position of the robot in the X direction at the current moment and \(x_{last}\) is the position of the robot in the X direction at the previous moment.