# A Risk-Sensitive Intelligent Control Algorithm for Servo Motor Based on Value Distribution
## 4 The Proposed Risk-Sensitive Intelligent Control Method
### The Structure of the Control Method
#### 4.1.1 Structure and Loss Function of Value Network  
The value network of the proposed risk-sensitive intelligent control method includes two parts: the coding part and the integrated part. The former outputs the hidden-layer codes of the quantiles, and the latter calculates the value of each action. As shown in Figure 6, the inputs of the value network also include two parts: The first one is the motor running state \(s_{t}=[o_{t},a_{t}]\), target speed, or torque \(g_{t}\), and the duty ratio output of agent \(a_{t}\), and all of this information is input into the fully connected network and activation function. The second one is a real value \(\tau_{i}\) sampled from a uniform distribution \(\tau\sim U(0,1)\), which represents the position of the \(i\)th quantile. After being cosine-encoded, this real value can translate to an output of the same length as the first part through a fully connected network. Each time an estimate is needed, the positions of the quantiles are sampled uniformly and act as an input to the neural network, which outputs the action value corresponding to each quantile. After receiving the input of the two parts, the neural network adopts a bit-multiplication calculation method, and transmits the data to the next fully connected layer and activation function to obtain the hidden-layer codes of the quantiles. With this input method, the location of each quantile can be different, which improves the accuracy of fitting the probability distribution of the neural network. On the other hand, the network can use arbitrary precision quantiles to describe the probability distribution, and the number of quantiles can be flexibly adjusted according to the actual needs and the computing performance of the specific computing equipment. After that, the coding part will generate the corresponding hidden-layer coding information for each input quantile position, and the amount of coding information is consistent with the number of quantile positions [30,31]. Figure 5: The structure of the control method. After extracting all the hidden-layer coding information of the quantiles, the value network collects and concatenates them together, and then, inputs them into the integrated part of the value network. As shown in Figure 7, there are two branches in the integrated part. In the first branch, all the hidden-layer coding information of the quantiles is successively passed through a fully connected layer and softmax layer. Then, through a cumulative summation operation, a set of monotonically increasing non-negative real \(\psi\) between 0 and 1 is obtained, which is used to represent the proportional relationship of the distance between the quantiles. In the second branch, the hidden-layer coding information of the quantiles passes through the fully connected network layer and outputs two real values. The first real value is exponentiated to obtain a proportional coefficient \(\alpha(a\geq 0)\), which is used to represent the proportional relationship between the output in the first branch and the actual quantiles. The second unprocessed real value, as the offset coefficient \(\beta\), represents the offset of the whole set of component values on the number line. Using the two coefficients \(\alpha\) and \(\beta\), the estimate of the probability distribution of cumulative discount returns for the entire network is obtained by linearly transforming the whole set of quantiles in the first branch. That can be expressed as follows:  
\[Z_{i}(s,a,g)=\alpha(s,a,g)\times\psi_{i}+\beta(s,a,g),i=1,2,\ldots,N \tag{7}\]  
where \(Z_{i}(s,a,g)\) denotes the output of the \(i\)th quantile. When using the value network, to solve the overvaluation problem of the duty ratio the algorithm adopts a pair of twin networks with the same structure, and takes the minimum value of the two networks as the final output every time the quantile is calculated. When training the value network, the positions of \(N\) quantiles as input are first obtained by uniform sampling in the range of \([0,1]\).