# A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot
## 3 Method
### Policy Training
We used Equation (14) to estimate the critic using the MC method, with the batch update method adopted to estimate \(\hat{Q}\):  
\[\hat{Q}_{t}=\sum_{t^{\prime}=t}^{T^{\prime}-1}\tau^{t^{\prime}-t}r_{t^{\prime}}+ \gamma^{T^{\prime}-t}V(s_{T};w) \tag{14}\]  
Using the TD method to estimate \(\hat{Q}\), Equation (14) is modified as  
\[\hat{Q}_{t}=r+V\big{(}s^{\prime};w\big{)} \tag{15}\]  
We also used the PPO algorithm to improve the training stability and efficiency, because the PPO uses a variable step size of the update instead of a fixed step size while also ensuring monotonicity. The theoretical basis of the PPO algorithm is to use the state distribution of the old policy \(\pi\) to approximate the state distribution of the new policy \(\widetilde{\pi}\), as expressed by  
Figure 3:0  
The advantage function \(A^{\pi}(s,a)\) is calculated as \(Q_{\pi}(s_{t},a_{t})-V_{\pi}(s_{t})\), where \(V_{\pi}(s_{t})\) is calculated using the output of the critic network of the PPO. The objective function to be optimized for the PPO algorithm in this paper is  
Figure 3:1  
Figure 3:2  
Although the on-policy PPO algorithm based on the stochastic policy has higher efficiency and better convergence, the biggest disadvantage of an on-policy algorithm, including the on-policy PPO algorithm, is low data efficiency. Deterministic off-policy algorithms make full use of data generated by other policies and try to avoid sampling through a large number of random actions to explore the environment. These algorithms are thus more data efficient than on-policy algorithms. We used the DDPG algorithm as a representative deterministic policy algorithm. The DDPG algorithm combines elements of value-function-based and PG-based algorithms. With a DNN, adopting the actor-critic architecture, \(\mu(s\,|\,\theta^{\mu})\) is a parameterized actor function maintained by the deterministic PG, and the critic \(Q(s,a)\) is learned through Q-learning to use the Bellman equation. The DDPG method is shown in Figure 3:3  
On the basis of the actor-critic framework with a chain rule, the actor is updated using Equation (19). The DDPG method uses \(Q(s,a)\) as a critic to evaluate the policy  
Figure 3:4 Vanilla deep deterministic policy gradient (DDPG) method. \(\mu(s\,|\,\theta^{n})\). The cumulated reward is defined as the sum of the discounted future reward \(R_{t}=\sum\limits_{i=t}^{T}\gamma^{(1-t)}r(s_{i},a_{i})\) with discounting factor \(\gamma\in[0,1]\):  
Figure 3:5  
where \(\beta\) is a stochastic behavior policy different from \(\mu\). A deterministic policy is used in the DDPG method, and the value function thus does not depend on any policy. The DDPG method does not need to perform importance sampling like in the case of a stochastic policy. The exploratory strategy of the off-policy method is adopted by the agent to generate data, and the gradient of policy is calculated using the data. It is, therefore, assumed that the samples come from the policy \(\rho^{\beta}\). \(\theta^{n}\) is a parameter that generates the actor network and \(\theta^{Q}\) is that for a critic network. The critic network is trained by minimizing the loss:  
Figure 3:6  
where  
Figure 3:7  
The above algorithms are all reinforcement learnings of model-free methods. A model-free method only uses previous data to optimize and improve the current behavior, and we, therefore, used the model-based method to train the model and thus further improve the learning speed and generalization of the agent. We use a method described in the literature [25] to train the agent, i.e., we used a neural network to fit the dynamic model and thus avoid the design of complex MPC controllers. In the model of an agent training of the biped robot, the state of DDPG and model-based MPC algorithms are the x and y positions of robot ZMP, as well as the waist joint angle and angular velocity.