# Vector Control of PMSM Using TD3 Reinforcement Learning Algorithm
## 4 Comparison of Simulation Results  
When the number of iterations reaches 300, the training ends. It can be seen from the Figure 14a, the average reward at the end of TD3 was 541 and DDPG was 520. As the estimate of long-term discount at the beginning of each episode, \(Q_{0}\) is closer to the real long-term value. As can be seen from the Figure 14b, \(Q_{0}\) under TD3 algorithm training is 545, while the maximum value of \(Q_{0}\) under DDPG algorithm is 210, which proves that PMSM current loop control under TD3 algorithm training will have better results. In terms of training time, TD3 training ended in 3 min and 56 s and DDPG training ended in 5 min  
Figure 13: RL learning and training structure.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter** & **Symbol** & **Value** \\ \hline Random seed & \(\alpha_{r}\) & 1 \\ Maximal set & \(M\) & 2000 \\ Maximum sub-step size per episode & \(T\) & 5000 \\ Sample Time & \(T_{s}\) & \(2\times 10^{-4}\) \\ Time of simulation & \(T_{f}\) & 3 \\ Experience Buffer Length & \(B\) & \(2\times 10^{6}\) \\ Quantity of batch & \(N\) & 250 \\ Threshold of gradient & \(\varepsilon\) & 1 \\ Learning rate of Actor network & \(L_{a}\) & 0.001 \\ Learning rate of Critic network & \(L_{c}\) & 0.0002 \\ Noise of exploration & \(e\) & 0.1 \\ Delayed updating & \(D\) & 2 \\ L2 Regularization Factor & \(L_{2}\) & 0.001 \\ Target Update Frequency & \(w_{l}\) & 10 \\ Factor of discount & \(\gamma\) & 0.995 \\ Rate of soft renewal & \(\tau\) & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The setting of hyperparameters.  
and 55 s. From the training results and training time, TD3 training reward value is higher and training time is shorter, and therefore, more suitable for RL control in the current loop.  
To evaluate the real control performance of the training network, the above TD3-RL control is compared with DDPG-RL control, linear quadratic Gauss (LQG) control, and BP network controller in simulation tests.  
The simulation is mainly to verify whether this control method can make the motor stable in the starting stage, during increasing and decelerating processes, and the loading and unloading of Work reliably. To better verify the reliability of the algorithm, a variety of working conditions are reflected in a simulation experiment. First, let the motor start no-load to the given speed of 1000 r/min, the start time is very short, the stepped speed rises, and then increase the load torque of 0.03 N\(\cdot\)m at 2 s, reduce the given load to 0 N\(\cdot\)m at 4 s, and increase the speed to 3100 r/min. Then, the speed drops step by step. The simulation experiment results are shown in Figure 15.  
As can be seen from Figure 16, the error fluctuation is particularly obvious, and the cumulative error of TD3 (the error integral under each simulation step) is smaller than that of the other three methods. As can be seen below from Table 4, compared with BP and LQG, the error of the two RL algorithms in signal tracking is smaller, while the step  
Figure 14: Training result. From left to right, (**a**) DDPG training results; (**b**) TD3 training results.  
Figure 15: Motor speed under different algorithms.  
The stator current amplitude is constant during the motor starting process, because the deviation between the given speed and the current speed value is too large, so that the outer loop PI of the speed is saturated. Due to the limiting effect of the controller, the given current of the output Q-axis is the limiting value, and the current of the D-axis is controlled by \(i_{d}=0\), so the amplitude of the stator three-phase current is constant. It can be seen from Figure 17 that TD3 algorithm has the smallest \(i_{d}\) fluctuation when torque is applied, and \(i_{q}\) response speed is faster than other algorithms. TD3-FOC has faster speed and torque current response than the other three algorithms in the start-up stage, loading stage, and unloading stage, and has better control performance.