# Single-Parameter-Tuned Attitude Control for Quadrotor with Unknown Disturbance
## 3 Quadrotor Attitude Control System Design
### Estimation of the CGPs
#### 3.3.1 Network Structures

The RL controller in this paper is based on the actor-critic scheme [26], in which the critic network judges the value of a state-action-pair, and the actor network outputs the control values according to the states input. The state in the attitude control process consists of the error between the desired attitude and the current attitude, along with the current angular velocity. The CGP is added to the state in this paper, so the state list is written as: \(\mathbf{s}=[\theta,\omega,b_{0}]\). The action list is expressed as \(\mathbf{a}=[a]\), where \(a\in[-1,1]\) is the control value in the controlled axis. We have two critic networks with the same structure: target critic network and online critic network [26]. The critic networks are fully connected networks (FCNs), with a hidden layer of 32 nodes and ReLU activation function. All of the experience is used to train the online critic network to fully observe the system and compute the value \([V]\). After that, we define the cost function as a squared error function and lower the cost using the Adam optimizer. The target critic network is soft updated with an online critic network. The online actor network and target actor network are also FCNs with a 32-node ReLU hidden layer. The tanh activation function is set in the output layer to limit the output between \(-1\) and \(1\). The state lists are sent directly to the actor network and action lists are computed. The gradients of the online actor network are computed with sampled policy gradients directly and then update the weights with the Adam optimizer [27]. The target actor network is soft updated [26] with the online actor network as well. Figure 3 presents the structures of actor networks and critic networks. #### 3.3.2 Reward Function Shaping and Mirror Sampling Methods

Reward function shaping and mirror sampling methods are presented in order to eliminate the steady-state error problem of RL algorithms in this section. Traditionally, the reward functions are squared error functions such as \(r\left(s\right)=-a_{1}*\theta^{2}-a_{2}*\omega^{2}\), where \(a_{1}\) and \(a_{2}\) are positive parameters. However, these squared error reward functions lead to slow convergence rates and severe steady-state error problems in our former experiments. From the perspective of this paper, the state with a low angle error or low angular velocity should be rewarded positively rather than be punished with less penalty. This paper presents a simple but powerful piecewise function \(fr\left(x,\delta,\rho\right)\) to meet this demand:

\[fr\left(x,\delta,\rho\right)= \begin{cases}-\frac{x}{\delta}+1,&\left(\left|x \right|\leq\delta\right)\\ \frac{x-\delta}{\delta-\rho},&\left(\delta<\left|x\right|<\rho\right)\end{cases}  \tag{12}\]

where \(x\) is the angle error or angular velocity to be rewarded, \(\rho\) is the upper level of \(x\), and \(\delta\) is the boundary line. If \(\left|x\right|\leq\delta\), then \(x\) should be rewarded positively. With \(fr\left(x,\delta,\rho\right)\), the reward function is designed as:

\[r\left(s\right)=-a_{1}*fr\left(\theta,\delta_{1},\rho_{1}\right)-a_{2}*fr\left( \omega,\delta_{2},\rho_{2}\right) \tag{13}\]

where \(a_{1},a_{2},\sigma_{1},\rho_{1},\sigma_{2}\) and \(\rho_{2}\) are positive parameters to be tuned. Because the quadrotor attitude control system is a symmetrical system, whenever a list of experience \(E=\left(s,\mathbf{a},\mathbf{s}^{\prime}\right)=\left(\left[\theta,\omega,b_{0}\right],\left[a\right],\left[\theta^{\prime},\omega^{\prime},b_{0}\right]\right)\) is explored, the mirror sample experience \(E_{m}=\left(s_{m},\mathbf{a}_{m},\mathbf{s}_{m}^{\prime}\right)=\left[-\theta,-\omega,b_{0}\right],\left[-a\right],\left[-\theta^{\prime},-\omega^{\prime},b_{0}\right]\) is also correct.