# An Intelligent Control Method for Servo Motor Based on Reinforcement Learning
## 3 Problem Description
### Task of Agent  
To enable the motor to operate at a given speed or torque, the task of the reinforcement learning agent is to output an appropriate duty cycle according to the servo error, which not only depends on the state of the motor but also depends on the target value.  
In order to reduce the influence of observation errors on the control strategy and the learning process of the agent, and to capture some motor running states that cannot be represented by instantaneous observations, each model input of the algorithm not only includes the current motor running state observations, but also combines historical information within a period of time. This includes the observation of the motor running state and the control output of the agent in the past period. The state input \(s_{t}\) of each neural network model can be specifically expressed as (4).  
\[s_{t}=[o_{t-h},a_{t-h},o_{t-h+1},a_{t-h+1},...,o_{t}] \tag{4}\]  
where \(o_{t}\) represents all operational state observations of the motor at time \(t\), i.e., Equations (1)-(3); \(a_{t}\) represents the action output by the agent at time \(t\); and \(h\) indicates the length of the history information contained in the status input.  
The action output of the reinforcement learning agent is the duty cycle. For the externally excited DC motor, it includes duty cycles of the armature voltage and excitation circuit voltage, which are denoted as \(a_{ExtEx}=[a_{A},a_{E}]\). For a series DC motor, it only includes one duty cycle of the input voltage and is denoted as \(a_{Series}\). For the permanent magnet synchronous motor, it includes three duty cycles of the A, B, and C phases, respectively, which is denoted as \(a_{PMSM}=[a_{A},a_{B},a_{C}]\). Each output action is a real number between 0 and 1, and as the value increases from 0 to 1, the duty cycle gradually increases.  
Therefore, the task of the agent is to output the appropriate duty cycle \(a_{t}\) at time \(t\) based on the state input \(s_{t}\) in order to control the motor running under the set target \(g_{t}\). \(g\) denotes the value of the set speed or torque.  
The Interaction between Agent and Environment  
Given a task space \(G\) including all possible speeds or torques, the reward function of the model is expressed as \(r(s,a,g)\), which calculates the real reward according to the environment state \(s\), the duty cycle \(a\) output by the agent, and the target \(g\). The reward is given to the agent by the environment after the agent outputs the duty cycle in a specific state under this target. At the beginning of each turn, the environment obtains an initial state of the environment \(s_{0}\) and the target \(g_{0}\). In each subsequent turn, the agent makes decisions according to the environment state \(s_{t}\) and the target \(g_{t}\) to be tracked in the current time step \(t\). When the agent output duty cycle occurs, the environment feeds back a reward signal \(r_{t}=r(s_{t},a_{t},g_{t})\) to the agent. At the same time, the environment transitions to the next state \(s_{t+1}\), and gives a new target \(g_{t+1}\). The reward function is calculated based on the negative of the value of the current servo error.