# A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot
## 5 Conclusions  
We studied the possibility of combining two cutting-edge technologies of advanced artificial intelligence: the humanoid robot and DRL. Adopting co-simulation, we first used several DRL algorithms to train the agent's policies, so that the table in the classic cart-table model adopted in the field of humanoid robots can maintain balance for as long as possible. The cart-table model is the theoretical basis of the model ZMP control method for whole-body stability control of the humanoid robot. We then adopted DDPG and model-based MPC algorithms, which performed well in two tasks of the table fixed at an initial angle and had a random initial angle in training in a second experiment. The experimental results were generally satisfactory, as shown by learning curves and simulation results. Our research shows that the DDPG algorithm has important advantages in robot control problems with a high-dimensional state and action space.  
\begin{table}
\begin{tabular}{c c} \hline \hline (a) DDPG algorithm \\ \hline
**DDPG Setup Hyper-Parameters** \\ \hline Actor/Critic learning rate & \(1\times 10^{-3}\) \\ Reward discount factor \(\gamma\) & 0.9 \\ Soft replacement \(\tau\) & 0.01 \\ Batch size & 8 \\ Running episodes & 80 \\ Experience buffer & 5000 \\ Number of neuron on actor network & 400 \\ Number of neuron on critic network & 200 \\ \hline (b) Model-based MPC algorithm \\ \hline
**Model-based MPC Setup Hyper-Parameters** \\ \hline learning rate & \(1\times 10^{-2}\) \\ Batch size & 16 \\ Running episodes & 100 \\ Buffer size & 10,000 \\ Number of candidate action sequences & 200 \\ Number of actions per candidate action sequence & 20 \\ Number of neuron on input layer & 800 \\ Number of neuron on hidden layer & 400 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hyperparameters of the two algorithms.  
Figure 14: Comparison of training times.  
In future work, we will change the weight and height of the iron ball to simulate different external forces in the x-direction of the biped robot. We will record the ZMP and pitch angle and speed information of the upper body of the robot and map them to the actions output by the trained strategy in the simulation, so that the state space and robot action space can be quickly mapped and real-time performance and effectiveness ensured.  
Since we considered the rigid connection of the arm and the body as a whole, we do not know whether the model ZMP control method is enhanced or weakened by the arm movement. Our next task also includes increasing the arm movement to make the humanoid robot closer to the actual situation when the robot needs to maintain a balance. The situation when there is no external force impact on the axis of the robot also needs to be studied, because this may cause the robot to rotate around the z axis.  
Owing to the limitations of cart-table model theory and the mechanical structure of the humanoid robot, the robot cannot take effective upper body control actions in the y-direction. In the next stage of research, in addition to adding a method of model ZMP control for walking, it will be necessary to study the anti-disturbance control method of the robot under an external force in the y-direction, so that the biped robot can use DRL algorithms to resist disturbances in all directions when standing upright and walking.  
[PERSON] is responsible for data curation, methodology, writing-original draft, visualization and software. [PERSON] is respondsible for funding acquisition. [PERSON] is responsible for software. [PERSON] is responsible for writing-review & editing. [PERSON] is responsible for formal analysis. [PERSON] is responsible for validation. All authors have read and agreed to the published version of the manuscript.  
This research was funded by the National Key Research and Development Project under Grant 2019 YFB1309501, National Natural Science Foundation of China under Grant 91748202 and 61903038.  
The authors declare no conflict of interest.