# Vector Control of PMSM Using TD3 Reinforcement Learning Algorithm
## 2 PMSM Model and RL
### Reinforcement Learning
#### 2.2.2 TD3 Algorithm  
TD3 (Twin Delayed Deep Deterministic Policy Gradient Algorithm), an online off-policy deep RL method that is upgraded with DDPG is utilized to handle continuous control issues [34]. Essentially, the purpose of the TD3 algorithm is to include the Double Q-Learning algorithm into the DDPG algorithm. Combined with the concept of Double DQN, there are six networks in TD3, whose network structure is shown in Figure 7.  
The TD3 algorithm, which is based on the DDPG algorithm, proposes three fundamental technologies:  
1. Double network: Two sets of Critic networks are adopted, and the smaller value of the two is taken when calculating the target value, to suppress the overestimation problem of the network. According to Equation (3), the update mode of the target value \(Q^{\prime}\) can be known. The following Equation (7) represents the overestimation between the target value \(Q^{\prime}\) and the actual value \(Q^{*}\), when \(Q^{\prime}\) is close to \(y\): \[Q^{\prime}\Big{(}s^{\prime},a^{\prime}\Big{|}\theta_{i}^{Q^{\prime}}\Big{)} \geq Q^{*}(s^{\prime},a^{\prime}).\] (7)
2. Target policy smoothing regularization: when the target value is calculated, the perturbation is added to the action of the next state, so that the value evaluation is more accurate: \[y=r+\gamma\text{min}_{i=1,2}Q\Big{(}s^{\prime},a^{\prime}\Big{|}\theta_{i}^{Q^ {\prime}}+\epsilon\Big{|}\theta_{i}^{Q^{\prime}}\Big{)},\] (8) \[\epsilon\sim clip(N(0,\sigma),-c,c)\]
3. Soft update: After the Critic network is updated for several times, the Actor network is updated, to ensure that the Actor network training is more stable. A learning rate \(\tau\) is introduced, the old target network parameters and the new corresponding network parameters are weighted averaged, and then assigned to the target network: \[\begin{array}{c}\theta Q_{i}^{\prime}=\tau\theta^{Q_{i}}+(1-\tau)\theta^{Q_{ i}^{\prime}}(i=1,2)\\ \theta^{\prime\prime}=\tau\theta^{\prime\prime}+(1-\tau)\theta^{\prime\prime} \end{array}.\] (9)  
The update process of TD3 algorithm is not much different from that of DDPG algorithm, and the main difference lies in the calculation method of the target value (Equation (3)). Where Actor networks are updated by maximizing cumulative expected returns (deterministic policy gradient), Critic1 and Critic2 networks are updated by minimizing the error between the evaluated value and the target value (MSE). All target networks are updated using soft update (Exponential Moving Average (EMA)). In the training phase, we sample a Batch size of data from the Replay Buffer, assuming that one sample of data is (\(s\), \(a\), \(r\), \(s^{\prime}\)), the update process of all networks is as follows:  
Figure 7: TD3 network structure.  
Update the Critic1 and Critic2 network parameters, calculate the actions under the state \(s^{\prime}\) using the Target Actor network:  
\[a^{\prime}=\mu^{\prime}(s^{\prime}\Big{|}\theta^{\mu^{\prime}}). \tag{10}\]  
Then, smooth regularization based on the target strategy, and add noise to the target action \(a^{\prime}\), and calculate the target value based on the idea of dual network (Equation (8)). The parameters in the Critic1 and Critic2 networks are updated using the gradient descent algorithm to minimize the error between the evaluated value and the target value:  
\[\theta_{i}\leftarrow\text{argmin}_{\theta_{i}}N^{-1}\sum{(y-Q_{\theta_{i}}(s, a))}^{2}. \tag{11}\]  
After updating the Critic 1 and Critic 2 networks step \(d\), start updating the Actor network and calculate the actions in state \(s\) using the Actor network, The gradient ascent algorithm is used to maximize \(q_{new}\) and update the Actor network. The TD3 network update process is shown in Figure 8.  
\[\left\{\begin{array}{l}a_{new}=\mu(s|\theta^{\mu})\\ q_{new}=Q_{1}(s,a_{new}|\theta^{\text{Q}_{1}})\end{array}\right.. \tag{12}\]