# Control Strategy of Speed Servo Systems Based on Deep Reinforcement Learning
## 4 Result and Discussion
### Experimental Results  
Test case #1  
The following figure shows the data of the successful training of each model. The performance test of MODEL_1 is shown in Figure 7a, in which the tracking amplitude of agent 1 is 1000, the absolute value of the slope is 13,333.3, and the isosceles trapezoid tracking effect presents an inconsiderable error. The reinforcement learning output action values of Kp, Ki present a straight line. The optimal control parameters are \(K_{p}=0.9318\) and \(K_{i}=1.4824\). It can be seen from the Figure 7b that the improvedPID (Improved_PID) with good parameters can achieve better control performance than Classic_PID. Figure 7c indicates that 800 episode values are trained to acquire the average reward for agent 2_1. The training is started by exploring a large degree, where in the average reward value is very low. As the training steps increase, the average reward increases gradually, and MODEL_2 eventually converges. Figure 7d depicts the average reward of agent 2_2 when training 4000 episode values. The average reward eventually increases and tends to be stable. The correctness and effectiveness of the agent model of the proposed method in this study are verified according to the above results. Test case #2  
The tracking command signal of the setting system is set with the amplitude value of 1000, the slope absolute value of 13,333.3 isosceles trapezoid signal, amplitude of 0.1, zero mean, variance of 1, and random white Gaussian noise signal. When tracking 0.5 s, system 1 is selected as the servo system in the tracking process. The reinforcement learning adaptive PID current compensation method is compared with PID control. The MODEL_2 control and the performance comparison of the PID approaches are shown in Figure 8.  
Figure 8a presents the tracking effect diagram of 0.50 s for the reinforcement learning adaptive PID current compensation method. A good tracking performance is obtained. The circle-part amplification is shown in the figure. Agent 2_1 tracking effect (RL_PID) is compared with Classic_PID and Improved_PID. In the absence of external disturbances, the proposed method can rapidly and accurately track the command signal, and the Classic_PID control structure produces a large tracking error. The control performance of Improved_PID is between Classic_PID and RL_PID. The parameter tuning method proposed in this paper can achieve better control performance than the Classic_PID. Figure 8b depicts the IAE index for RL_PID, Improved_PID, and Classic_PID. The IAE of the reinforcement learning adaptive PID control algorithm is smaller than that of the PID algorithm under the condition of the same system. The results prove that the proposed control strategy is feasible in tracking trapezoidal signal and can achieve a good control performance. Figure 7: Result of test case #1. (**a**) Agent 1 tracking renderings; (**b**) Integral of absolute error (IAE) index comparison diagram; (**c**) Average reward value of agent 2_1; (**d**) Average reward value of agent 2_2. Test case #3  
The tracking command signal of the setting system has an amplitude value of 2000, aslope absolute value of 26,666.6 isosceles trapezoid signal, an amplitude of 0.1, zero mean, variance of 1, and a random white Gaussian noise signal. When tracking 0.5 s, system 1 is selected as the servo system. The inertia and moment disturbance of the random system are added between 0.075 and 0.375 s. The reinforcement learning adaptive PID current compensation method (RL_PID) is compared with Improved_PID control, and Classic_PID. The MODEL_2 control and the performance comparison of the PID approaches are shown in Figure 9.  
Figure 8: Result of test case #2. (**a**) Tracking signal comparison diagram; (**b**) IAE index comparison diagram. Figure 9: Result of test case #3 (**a**) Tracking signal comparison diagram; (**b**) System selection chart; (**c**) IAE index comparison diagram; (**d**) Agent 2_1 output current value and tracking error graph.