# Vector Control of PMSM Using TD3 Reinforcement Learning Algorithm
## 2 PMSM Model and RL
### Reinforcement Learning
#### 2.2.1 DDPG Algorithm  
DDPG (Deep Deterministic Policy Gradient) is a depth deterministic strategy gradient algorithm. It is also a way to solve the problem of continuity control. It is a model-free, off-policy, or policy-based method. It solves the shortcoming that there is correlation before and after each parameter update of Actor-Critic neural network, which leads to the neural network only viewing the problem unilaterally, solving the shortcoming that DQN cannot be used for continuous action [30; 31; 32].  
The structure of DDPG is like to Actor-Critic. DDPG can be divided into two major networks: strategy network and value network. DDPG continues the idea of fixed target network with DQN, and each network is subdivided into target network and reality network. However, the update of the target network is somewhat different, and its network structure is shown in Figure 5.  
DDPG consists of four networks: Actor current network, which is responsible for the iterative update of policy network parameters \(\theta\) and the selection of current action \(A\) according to the current state \(S\), which is used to interact with the environment to generate \(S\)' and \(R\)', the network update process is shown in Figure 6.  
Figure 4: Conventional control methods.  
Figure 5: DDPG network structure.  
Actor target network: it is responsible for selecting the optimal next action according to the next state sampled in the empirical playback pool, updating the network parameter \(\theta^{\prime}\) by soft update. Critic network: responsible for the iterative update of value network parameter \(w\) and the calculation of \(Q\) value. Critic target network: responsible for calculating the \(Q^{\prime}\) part of the target \(Q\) value and soft updating the network parameter \(w^{\prime}\):  
\[y_{i}=R+\gamma Q^{\prime}(S^{\prime},A^{\prime},w^{\prime}). \tag{3}\]  
The neural network uses gradient backpropagation to change the parameters of the Actor and Critic networks:  
According to the mathematical derivation in [PERSON]'s DPG paper [33], the strategy gradient update algorithm derived by adopting the off-policy training method according to the Monte Carlo method, when we randomly sample mini-batch data from replay memory buffers into the policy gradient formula described above, we can make an unbiased estimate of the expected value.  
\[\
abla J(\theta)=\frac{1}{m}\sum_{j=1}^{m}\left[\
abla_{a}Q(s_{i},a_{i},w) \right|_{s=s_{i},a=\pi_{\theta(s_{i})}}\
abla_{\partial}\pi_{\theta}(s)\big{|} _{s=s_{i}}\big{]}. \tag{4}\]  
The loss function in the network is calculated using a method like supervised learning, which is usually calculated as the mean square error (MSE) calculation method.  
\[J(w)=\frac{1}{m}\sum_{j=1}^{m}\big{(}y_{i}-Q(\phi(S_{j}),A_{j},w)\big{)}^{2}. \tag{5}\]  
DDPG uses a soft update method for network parameters, using the update coefficient \(\tau\), and only slightly modifies some parameters each time. Each iteration will modify the target network, and the algorithm can still maintain a certain stability.  
\[\begin{array}{l}w^{\prime}\leftarrow\tau w+(1-\tau)w^{\prime}\\ \theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}\end{array}. \tag{6}\]  
The target network parameter changes little and is used to calculate the gradient of the online network in the training process, which is relatively stable and easy to converge in training. However, with small parameter changes, the learning process is slow, and the  
Figure 6: DDPG network update process.  
use of a slowly updated target network can easily cause overestimation of \(Q\) value, which makes it difficult to converge the strategy. This defect is solved in TD3.