# A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot
## 3 Method
### Policy Training  
Six DRL algorithms were used in training the cart-table model, namely, the PG, MC-based AC, TD-based AC, PPO, DDPG and model-based MPC methods were used to train the agent to control the movement of the cart so that the table leg is upright. In the PG method, the policy was first parameterized, and the parameters were then updated using gradients. Finally, optimal parameters, expressing the optimal policy, were found. The regression network was used as the policy network because the output of the cart-table model is a continuous variable. Gaussian policies were used for agent training, and the mean and standard deviation of the Gaussian distribution were parameterized. The network model of the policy is shown in Figure 4. In the cart-table model, the state of six DRL algorithms is the cosine of the table angle, sine of the table angle, table rotation speed, cart position and cart speed. The output of the network is the driving force of the cart wheels. This state-action pair was used in the cart-table model with six algorithms described in this section. Figure 3: Biped robot in the simulation environment. (**a**) At the start of simulation, the pendulum is in its reset state, i.e., it is inclined upward at an angle with the ground of thirty degrees. (**b**) The pendulum with an iron ball accelerates and falls under the influence of gravity. When the back of the robot detects that the ball has struck the robot, the agent starts to interact with and learn the environment. Compared with the classic REINFORCE [24] algorithm of reinforcement learning, we used a batch-based method instead of updating for each state \(s\). When using the current policy to collect \(N\) trajectories, each trajectory contains \(T\) state-action pairs. These \(N\times T\) data points are used to estimate the gradient. This helps improve the stability of the PG method. The final stochastic policy gradient formula is thus  
\[\
abla_{\theta}U(\theta)\approx\frac{1}{m}\sum\limits_{i=1}^{m}\sum\limits_{t= 0}^{T}\
abla_{\theta}\log\pi\Big{(}a_{t}^{i}|s_{t}^{i}\Big{)}\Bigg{[}\sum \limits_{k=t}^{T}R\Big{(}s_{k}^{i}\Big{)}\Bigg{]} \tag{9}\]  
where \(R(s_{k})\) is the cumulative discounted reward from this state, owing to the current action having no relationship with the past return. In Equation (9), \(\sum\limits_{k=t}^{T}R\big{(}s_{k}^{i}\big{)}\) estimated using the MC method in the PG algorithm is adopted to judge the effect of the action; it is usually called the critic. However, online estimation methods are usually inaccurate, and they cannot solve the problem according to long-term interaction with the environment. \(\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\) is related to policy and usually called the actor. In the AC algorithm, we used the function approximation method to estimate the critic, and make the critic as close to the true value as possible through continuous learning. The AC algorithm adopted in this research is thus written as  
\[\mathrm{g}=\mathrm{E}[\sum\limits_{t=0}^{\infty}\delta_{t}\
abla_{\theta}\log \pi_{\theta}(a_{t}|s_{t})], \tag{10}\]  
where \(\delta_{t}=\hat{Q}-V(s_{t};w)\) is the critic, and the state value function \(V(s)\) is parameterized as \(V(s_{t};w)\). \(\hat{Q}\) is the estimated value of the current action-value function. If a single-step update is adopted, the parameter update rule is  
\[\theta\leftarrow\theta+\alpha^{\theta}\delta_{t}\
abla_{\theta}\log\pi_{ \theta}(a_{t}|s_{t}) \tag{11}\]  
The purpose of this review is to improve the estimation of the value function. The loss function is constructed as  
\[\mathrm{loss}=\big{(}\hat{Q}-V(s_{t};w)\big{)}^{2} \tag{12}\]  
and the single-step update \(w\) is  
\[w\leftarrow\mathrm{w}+\alpha^{w}\delta_{t}\
abla_{w}V(s_{t};w) \tag{13}\]  
Figure 4: Policy network of the policy gradient (PG) method in cart–table model. In this paper, we used two methods to estimate \(\hat{Q}\) in Equation (12) for comparison; one is based on the MC method, while the other is based on the TD method.