# A Risk-Sensitive Intelligent Control Algorithm for Servo Motor Based on Value Distribution
## 3 Problem Description
### Task of Agent  
The objective of a servo control system is to operate the motor at a specified speed or torque. To achieve this, a reinforcement learning agent must output an appropriate duty cycle to minimize the servo error, which is the difference between the actual state and the target [27].  
To mitigate the impact of observation errors, and to capture some motor running states that cannot be represented by instantaneous observations, in this work, both the current observations of motor operation and the historical information over a period are taken as the input of the proposed control algorithm. Specifically, it includes the observations of the motor's running state and the control outputs of the agent in the past period. Therefore, the input state \(s_{t}\) of the reinforcement learning agent at time \(t\) can be expressed as Equation (6).  
\[s_{t}=[o_{t-h},a_{t-h},o_{t-h+1},a_{t-h+1},\ldots,o_{t}] \tag{6}\]  
where \(o_{t}\) represents all operational state observations of the motor at time \(t\), i.e., Equations (3)-(5); \(a_{t}\) represents the action output by the agent at time \(t\); and \(h\) indicates the length of the history information contained in the status input.  
Then, the reinforcement learning agent takes \(s_{t}\) as input, and takes the duty cycle as the output action. For an ExtEx motor, the action includes the armature voltage's duty cycle \(a_{A}\), and the excitation circuit voltage's duty cycle \(a_{E}\), which is denoted as \(a_{ExtEx}=[a_{A},a_{E}]\). For a series DC motor, the action only includes the duty cycle of the input voltage \(a_{Series}\). For a PMSM, all the duty cycles of the A, B, and C phases are taken as the output action, which is denoted as \(a_{PMSM}=[a_{A},a_{B},a_{C}]\). Each output action is a real number between 0 and 1. The duty cycle gradually increases with the increase in the action value.  
Therefore, at time \(t\), to control the motor running under the set target \(g_{t}\), the agent has to output the appropriate duty cycle \(a_{t}\) based on the state input \(s_{t}\), where \(g_{t}\) denotes the target speed or target torque.  
The Interaction between Agent and Environment  
Given a target space \(G\) including all possible speeds or torques, the reward function evaluating the performance of the control strategy is expressed as \(r(s,a,g)\), which calculates the real-time reward using the input state \(s\), the output duty cycle \(a\), and the set target speed or torque \(g\). After the agent outputs a duty cycle in the current state under a given target, the environment gives a reward to the agent to reflect the performance of the agent. In detail, the environment obtains an initial state of the environment \(s_{0}\) and the target \(g_{0}\) at the beginning of each turn. In each subsequent turn, the agent makes decisions according to the environment state \(s_{t}\) and the target \(g_{t}\) to be tracked at time \(t\). When the agent outputs the duty cycle, the environment feeds back a reward signal \(r_{t}=r(s_{t},a_{t},g_{t})\) to the agent. At the same time, the environment transitions to the next state \(s_{t+1}\), and gives a new target \(g_{t+1}\).