# Vector Control of PMSM Using TD3 Reinforcement Learning Algorithm
## 1 Introduction  
Due to their simple form, high power density, and high efficiency, PMSMs are widely employed in numerous industrial control applications [1; 2; 3]. The speed of PMSM is mainly regulated by frequency conversion. Closed-loop speed constant voltage frequency ratio control (V/F), vector control (VC), and direct torque control are widely used in frequency conversion speed control [4]. The most common is VC, also known as magnetic Field-Oriented Control (FOC), which uses a variable-frequency drive (VFD) control three-phase alternating current (AC) motor to control the output of the motor by adjusting the frequency converter's output frequency, output voltage size, and angle [5]. In this paper, the \(i_{d}\)-axis and \(i_{q}\)-axis of the current loop are accurately controlled by reinforcement learning method under the FOC framework, so that the inverter can output a suitable PWM signal and obtain accurate motor speed, providing a new current loop control method. PMSM is a complex object with multiple variables, strong coupling, and nonlinear and variable parameters. To obtain better control performance, many scholars have designed a reasonable controller [6]. [PERSON], [PERSON] et al. [7] proposed a non-singular fast terminal sliding mode (NNFFSM) control strategy based on extended state observer (ESO) and tracking differential (TD), in which PMSM has strong robustness to parameter changes and external load disturbances. [PERSON], [PERSON] et al. [8] adopted a nonlinear adaptive control (NAC) method of PMSM to estimate the concentrated disturbance terms through the observer, thus achieving better dynamic performance of the system. [PERSON], [PERSON] et al. [9] proposed a current restraint controller based on interference observer for PMSM speed control to reduce theinterference of current constraints and external conditions. [PERSON], [PERSON] et al. [10] constructed a special nonlinear gain to directly establish the Q-axis current penalty mechanism in the PMSM control action, which solves the problem of overcurrent protection under fast dynamic conditions. With the rapid development of computer technology, many scholars have done more research on the intelligent control of PMSM. [PERSON], [PERSON] et al. [11] adopted NSGA-II (Non-dominated Sorting Genetic Algorithm-II) to optimize the PID controller parameters of PMSM. [PERSON], [PERSON] et al. [12] proposed a new energy efficiency-oriented sliding mode controller (APIDSMC-PALC) compensation method to suppress the influence of PMSM servo system torque ripple. Deep reinforcement learning (DRL) is an ideal multi-objective optimization method with a high searching ability and a quick convergence rate, and it has demonstrated high application value in the control sector in recent years. [PERSON], [PERSON] et al. [13] made a control autonomous underwater vehicle (AUV) by model-free RL based on data informed domain randomization (DDR) that enables the controller to adapt to sudden changes in dynamics. [PERSON], [PERSON] et al. [14] used the RL framework with Actor-Critic network as a new path ordering algorithm (PRA) to carry out effective relationship learning and path finding, reducing the dependence on large-scale training data sets. [PERSON], [PERSON] et al. [15] integrated input and output data and cyclic neural networks, established an observer to approximate unknown system dynamics, and solved the problem of optimal stability of unknown nonlinear systems affected by uncertain input constraints. [PERSON], [PERSON] et al. [16] used DRL to enable unmanned aerial vehicles (UAVs) to perform navigation tasks randomly and dynamically in a multi-obstacle environment. [PERSON], [PERSON] et al. [17] proposed a control method of reinforcement learning-PI based on genetic algorithm. The model was built, and the initial parameters of PI controller were optimized by genetic algorithm.