# An Intelligent Control Method for Servo Motor Based on Reinforcement Learning
## 1 Introduction  
A servo motor plays an important role in the composition of automation equipment. It can convert the voltage signal into torque and speed to drive the controlled object with high control precision. Because servo motors can meet the torque and speed control requirements of most tasks, they have been widely used in many manufacturing fields, such as in aerospace, automotive manufacturing, and robotics, among others [1]. In the servo motor control system, the most widely used and stable control method is proportion-integration-differentiation (PID), which can generate the control strategy only by the tracking error. This control strategy bears the advantage of a control mechanism which is completely independent of the mathematical model of the controlled object [2]. However, the PID parameters can generally only be set manually. Thus, the actual control effect not only depends on the debugging experience of engineering personnel, but also requires another time-consuming process to verify the control effect of parameters. Moreover, during the operation of the servo system, the change in magnetic saturation degree, temperature, load and other conditions of the motor will degrade the performance of the PID controller designed with the original parameters. Therefore, the PID controller parameters should be adjusted in a timely manner according to the actual working conditions, which will increase the difficulty of manual parameter setting. In order to reduce the burden of manual parameter setting and improve the control effect, the servo system needs an adaptive control algorithm which is able to effectively solve the problems caused by the changes in servo motor working conditions by identifying some related parameter values online and modifying the control strategy in real time [3; 4; 5; 6; 7]. In recent years, reinforcement learning, as a major branch of artificial intelligence, has gained increased attention and made breakthroughs in fields such as autonomousdriving [8; 9], path planning [10; 11], and robotics [12; 13]. Reinforcement learning imitates humans' self-learning ability and can continuously optimize decision-making strategies through a trial and error mechanism to realize its own evolution [14]. By applying the reinforcement learning algorithm, it may prompt the servo motor control system to adapt to the complex control environment by automatically adjusting the output of the control system through the imitation of human decision-making behavior [15; 16; 17; 18]. The existing methods are all based on deterministic strategies, and their learned optimal strategies are uniquely determined action values which lack the ability to explore the interaction environment. To explore other actions around the current optimal action, noise needs to be artificially added to the policy. The performance is affected. Moreover, in the existing algorithm, the tracking error is taken as the basis of reward, and the error values in a period of time are uniformly input to the neural network model. Then, the coefficients of proportion, integral, and differential of the PID algorithm are learned via the neural network. This method only allows the agent to control the motor by tracking errors, although it improves the generalization performance of the algorithm to a certain extent, but also causes the loss of the motor running-state information, which is not conducive to further improving the control accuracy. In view of the problems faced by traditional industrial control systems, this study intends to design a deep reinforcement learning controller based on reference trajectory coding to implement intelligent servo motor control. In this, the running state of the servo motor and the target trajectory in the control process are regarded as two different items of input information, so that the agent can make better use of the external information to make control decisions. Therefore, the control precision of the reinforcement learning motor control algorithm is more optimized.