# A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot
## 2 Related Work
### Biped Robot Controlled by DRL  
Humanoid robots are high-dimensional non-smooth systems with many physical constraints. Nowadays, an increasing number of humanoid robot control algorithms are being developed adopting DRL in research on, for example, robot balance [3; 4; 5], the dynamic performance after the training of a legged robot [6] and the passive dynamic walking robot [7]. [PERSON] et al. [8] used motion capture with a model-free reinforcement learning algorithm to reduce differences in humanoid robots learning human actions. [PERSON] et al. [9] proposed a biped robot control method based on stability training and reinforcement learning by applying random disturbance to the robot. [PERSON] et al. [10] designed a passive biped robot based on control with a policy gradient (PG) algorithm that can learn to walk while adapting to the terrain itself. [PERSON] et al. [11] proposed a multilevel system and used the Q-learning method in this system to make the NAO robot walk quickly in simulation. [PERSON] et al. [12] proposed a hybrid reinforcement learning method to keep the NAO robot balanced on an oscillation platform with different frequencies and amplitudes in the simulation. [PERSON] et al. [13] proposed a learning framework for central pattern generation (CPG)-based biped locomotion with a policy gradient method and applied it to real robots. [PERSON] et al. [14] applied actor-critic (AC) algorithms to a humanoid robot learning to crawl and a puppy robot learning to gallop, and compared two types of AC algorithm (i.e., policy search and PG algorithms). [PERSON] et al. [15] designed a learning framework using the proximal policy optimization (PPO) algorithm; the biped robot developed on this basis can recover from large external forces. [PERSON] et al. [16] used an improved DDPG algorithm to study the walking stability of a passive biped robot. [PERSON] et al. [17] proposed a model-based method for MPC, which is based on a framework that generates biped robot actions. Simulation results show that the framework can generate actions robustly.  
In addition, many researchers are now conducting research on momentum-based robot control. [PERSON] et al. [18] proposed a control framework based on momentum and quadratic programming for the humanoid robot \"Atlas\", and demonstrated its ability to walk on rough terrain. Using a reformulation of existing algorithms, [PERSON] et al. [19]proposed a simplification method based on the cascades of quadratic programs and hierarchical inverse dynamics. The experimental results prove the effectiveness of the method for feedback control under a real robot. [PERSON] et al. [20] introduced a new regressor-based observer method for improving the collision detection sensitivity of the serial manipulators. Compared to the state of the art, this method improves collision detection accuracy and sensitivity.  
All the above studies considered the improvement of robot stability control from the perspective of improving algorithms. To solve disturbance rejection from a practical point of view, the present paper develops a biped robot x-direction disturbance rejection training environment. Different algorithms can be used to test the disturbance rejection ability in the simulation, so that actual problems encountered in the research and development of a simulation robot can be solved quickly and conveniently.