# Automatic Control Optimization for Large-Load Plant-Protection Quadrotor
## 2 Materials and Methods
### Attitude Controller Design for Large-Load Plant-Protection Quadrotors
#### 2.2.2 Network Structure and Training Algorithm  
The structure of the RL controller is an actor-critic [21] based structure, as presented in Figure 5. The critic network is built to estimate the value of the state and action, and the actor network is built to compute the control values with the state. Figure 4: Control diagram for large-load plant-protection quadrotors. The RL method in this article is the extended-state RL proposed in our previous work [20], where system parameters can be extended into the state list, which enables the controller to fit the changes in these parameters. As a result, the state list consists of the attitude errors \(att_{e}=[q_{0},q_{1},q_{2},q_{3}]\) and the current angular velocities in the three axes \(\big{[}\omega_{x},\omega_{y},\omega_{z}\big{]}\), along with the MOIs in the three axes \(J_{xx},J_{yy},J_{zz}\) and the response constant of the rotors \(T_{m}\). The action list consists of control values in the three axes \(a_{x},a_{y},a_{z}\). The training method in this article is the DPG method [21], consists of two actor networks and two critic networks. The actor networks include the target actor network and the online actor network, both with the same structure, and two hidden layers of 128 nodes are included. The tanh activation function is set to limit the output value between \(-1.0\) and \(1.0\) in the output layer. The online actor network is trained in each time step, the gradients are computed with the sampled policy gradients, and the weights are updated using the Adam optimizer [23]. The target actor network is soft updated [21] using the online actor network. The critic networks also include the target critic network and the online critic network. The two critic networks share the same structure as well, and two hidden layers of 128 nodes are designed. The cost function of the online critic network is a squared error function, and we lower the cost with the Adam optimizer. We soft update the weights of the target critic network using the online critic network. The reward function design is critical work in the RL algorithm. Squared error style reward functions are widely used in the RL-based control domain, as Equation (12) presents:  
\[r(s)=-p_{1}*\Theta^{2}-p_{2}*\Big{(}\omega_{x}^{2}+\omega_{y}^{2}+\omega_{z}^{ 2}\Big{)}, \tag{12}\]  
where \(p_{1}\) and \(p_{2}\) are positive parameters and \(\Theta=2 arccos(q_{0})\) is the total rotation angle of the attitude error. This reward function encourages the agent to stably reach the zero point. However, in many RL studies using these squared error reward functions, controllers show obvious steady-state errors [24]. We analyze the cause of steady-state errors and present a reward shaping method to solve this problem in article [20]. The reward shaping skill is also applied in this article, and we construct a piecewise function as:  
\[fr(x,\epsilon,\varrho)= \begin{cases}-|x|/\epsilon+1,&(|x|\leq\epsilon)\\ (|x|-\epsilon)/(\epsilon-\varrho),&(\epsilon<|x|<\varrho),\end{cases}  \tag{13}\]  
where \(x\) is the input state value (attitude error or angular velocity in this article), \(\epsilon\) and \(\varrho\) are constants. With Equation (13), the reward function is designed as:  
\[r(s)=p_{1}*fr(\Theta,\delta_{1},\rho_{1})+p_{2}*\big{(}fr(\omega_{x},\delta_{ 2},\rho_{2})+fr(\omega_{y},\delta_{2},\rho_{2})+fr(\omega_{z},\delta_{2},\rho _{2})\big{)}, \tag{14}\]  
Figure 5: The RL networks structure. where \(p_{1},p_{2},\delta_{1},\rho_{1},\delta_{2}\) and \(\rho_{2}\) are positive parameters to be tuned. Compared to the reward functions like Equation (12), this function output positive reward for close-to-zero states. In a real flight environment, sensor noise can decrease the performance of the controller. The attitude angle data are estimated and filtered using gyroscope data, accelerometer data and magnetic compass data, so the errors of attitude angle data cannot be ignored. The angular velocities from sensors are relatively accurate, so errors of angular velocity data are ignored in this research.