# A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot
## 4 Experiments and Results
### Agent Training for the Cart-Table Model  
We used the six classic DRL algorithms introduced in Section 3 to train the model in the simulation. The condition for the end of each episode of simulation is that the angle between the desktop and the horizontal direction is greater than \(15^{\circ}\) or less than \(-15^{\circ}\). This design indicates that when the biped robot tilts forward or backward more than \(15^{\circ}\), it is basically impossible to adjust the tilt of the upper body of the biped robot to rebalance the robot. Figure 8 shows the state when the table is about to lose its balance, when the absolute value of the table tilt angle is greater than or equal to \(15^{\circ}\). Figure 7: Two tasks in the evaluation of the proposed scheme. (**a**) Task 1: the initial angle between the leg of the table and the vertical direction is a fixed value of \(1^{\circ}\). (The red dashed line is the table leg, and the black dashed line is the vertical direction.) (**b**) Task 2: the initial table angle is set as a random value between \(-0.5^{\circ}\) and \(0.5^{\circ}\) for each episode. In this experiment, the reward function of reinforcement learning is  
\[r_{t}=\frac{1}{(\theta_{t}^{2}+1)}, \tag{23}\]  
where \(\theta_{t}\) is the tilt angle of the table at each step. The simulation time step was 0.01 s.  
The pseudo-code for training the cart-table model with reinforcement learning algorithms is presented in Algorithm 1. We executed each of the algorithms to train tasks 1 and 2. The use of the same program framework ensures that the results of different algorithms can be compared. ```
Build a cart-table model in V-REP Choose a DRL algorithm Randomly initialize a set of weights \(w\) and biases \(b\) of the deep neural network of this algorithm for\(ep=1\), EPISODEdo Initialize the simulation environment: set the Cart to zero position to \(x_{0}=0\) (the midpoint of the desktop), set the speed to \(v_{0}=0\) and acceleration to \(a_{0}=0\) Set the table in angle \(\theta_{0}\) to a random value within the range from \(-0.5^{\circ}\) to \(0.5^{\circ}\) (task 1) or a fixed value of \(1^{\circ}\) (task 2). Get initial state \(s_{0}\) by sorting (\(\theta_{0}\), \(x_{0}\), \(v_{0}\)) for\(t=1\), STEPdo Select action \(a_{t}\) according to current policy Execute \(a_{t}\) to obtain reward \(r_{t}\) and observe new state \(s_{t}(\theta_{t},x_{t},v_{t})\) Add \(r_{t}\) to reward \(r\) Obtain loss according to network output Update network parameters using information such as the loss and gradient: \(w_{t}\gets v_{t}+\Delta_{t}^{pv}\) \(b_{t}\gets b_{t}+\Delta_{t}^{b}\) endfor endfor
```  
**Algorithm 1**Cart-table model with DRL algorithms. Figure 9 shows the training curves of the six algorithms for the two tasks. The blue line is the curve for task 1 and the green line the curve for task 2. Each task for each algorithm was trained three times. The shaded area of each graph is the range between the maximum and minimum scores in the three experimental runs. The thick line represents the mean of the average scores of the three experimental runs, which can also be called the average rising curve of scores obtained from the three experimental runs. Figure 10 is the result of extracting all the thick lines in Figure 9 and putting them into their respective tasks for comparison. Figure 9 shows the performance comparison of each algorithm on  
Figure 8: End of simulation of one episode when the table tilt angle is too large. two tasks. Figure 10 shows the performance comparison of the six algorithms deployed in each task. The figures clearly show that deploying the DDPG- and model-based MPC algorithms for task 1 achieved better training results. In task 2, the policy trained adopting the model-based MPC method quickly maintained the table balance, and the balance was maintained for a relatively long time. The PPO method had similar performance. The reason for this may be that the initial angle of the desktop in task 2 was smaller than that in task 1, and it was thus easier to achieve long-term stability.