# Control Strategy of Speed Servo Systems Based on Deep Reinforcement Learning
## 4 Result and Discussion  
### Experimental Setup  
In the actual application process, the speed servo system instruction is generally trapezoidal or positive rotation. Therefore, only the performances of the servo control system in the two types of instruction cases are analyzed and compared in this study on the basis of system control experiments under the instructions of different amplitudes, frequencies, and rise and fall rates. In the control process of the servo system, PID parameters play a key role in the control performance of the speed servo system. The classic PID (Classic_PID) controller is adopted to compare with the algorithm proposed in this paper. The control parameters turned based on the empirical knowledge from experts are \(K_{p}=0.5\) and \(K_{i}=2.985\). Reinforcement learning agent 1 is constructed on the basis of the method in 3.4.1 to realize automatic tuning of the PID parameters of the servo system. The relevant parameters are set as follows: the capacity of the memory \(M_{1}\) is \(C_{1}\) = 50,000, the actor target network updates the steps by 11,000, and  
Figure 6: Structure diagram of the critic network. the critic target network updates the steps by 10,000. The action range is [0 to 100]. The learning rate of actor and critic \(\alpha_{a}=\alpha_{c}=0.0001\), and the batch size \(N=256\). Gaussian distribution is used to increase noise, and the output value of the neural network is used as the distribution mean. Parameter var 1 is set as the standard deviation, and the initial action noise var 1 = 10. The noise strategy is adjusted dynamically with the increase in learning frequency. Finally, var 1 = 0.01. In this paper, the control precision of the controller is adopted as the control target. In consideration of the limitation of control servo motor current and power consumption in practical engineering, the reward \(r\) is defined as:  
\[\mathrm{r}_{t}=\mathrm{e}^{-\frac{1}{2}\frac{\left[0.9\mathrm{s};\mathrm{s} (r)\right]+0.1\mathrm{s}-\left[1_{t}\right]\right|^{2}}{\sigma^{2}}} \tag{13}\]  
where \(\sigma=10\). Table 1 presents that with the control servo system 1, the tracking amplitude is 1000, the slope absolute value is 13,333.3, the isosceles trapezoidal signal is 0.5 s, the loop iteration is 5000 times, and the environment interaction object is reinforcement learning agent 1. The trained agent model is called MODEL_1. Reinforcement learning agent 2_1 is constructed on the basis of the method in 3.4.2 to realize the adaptive compensation of PID output current. The optimal control parameters learned by the adaptive PID control algorithm in MODEL_1 are \(K_{p}=0.9318\) and \(K_{i}=1.4824\). The system performance of the fixed PID control structure under these parameters is compared with the control performance of the adaptive current compensation method proposed in this study. The relevant parameters of the reinforcement agent 2_1 are set as follows: the capacity of the memory \(M_{2}\) is \(C_{2}=50\),000, the actor target network updates the steps by 11,000, and the critic target network updates the steps by 10,000. The action range is [\(-2\) to 2]. The learning rate of actor and critic \(\alpha_{a}=\alpha_{c}=0.0001\), and the batch size \(N=256\), and the initial action noise var 2 = 2. Finally, var 2 = 0.01. The control output of agent 2_1 needs to compensate the PID current. Therefore, the angular velocity error of PID control is constructed to build the reward function \(r\), and the reward \(r\) is defined as:  
\[\mathrm{r}_{t}=0.9*\mathrm{e}^{-\frac{1}{2}\frac{\sigma(t+1)^{2}}{\sigma^{2}} }+0.1*\mathrm{e}^{-\left|1_{t}\right|} \tag{14}\]  
where \(\sigma=2\). The environment interaction object of agent 1 is adopted to the reinforcement learning agent 2_1. Eight hundred episodes are trained to acquire the agent model, called MODEL_2. Agent 2_2 is constructed on the basis of the structure of agent 2_1.