# Single-Parameter-Tuned Attitude Control for Quadrotor with Unknown Disturbance
## 3 Quadrotor Attitude Control System Design
### Estimation of the CGPs  
In this paper, the CGP is the proportional relation between the control signal \(a_{x},a_{y},a_{z}\) and the angular accelerations in three axes \(\dot{\omega}_{x},\omega_{y},\dot{\omega}_{z}\). The accuracy of the CGPs is critical to both the RL controller and ESO feedback. The CGPs of the F550 quadrotor can be simply calculated with Equation (8), but the measurement of different quadrotors or platforms can be difficult and expensive and, thus, a method to estimate the CPGs is presented in this section. As Equation (7) shows, the CGP can be calculated given the state of a quadrotor, including angular acceleration, control value, and disturbance. When considering the ill-conditioned equation problem and noise of sensors, the system should obtain enough excitation during the estimation process, but too much excitation may cause a quadrotor crash. To precisely estimate angular acceleration data, a TD is designed to calculate the differential of angular velocity data in Algorithm 2:
```
1:\(v_{0}\): angular velocity data from sensors
2:\(v_{1}\): angular velocity output
3:\(v_{2}\): angular acceleration output
4:\(fhan\): a non-linear function [14]
5:\(e\), \(fh\): intermediate variables
6:\(h\): time step
7:\(h_{0}\): adjustable parameters, usually two or three times \(h\)
8:
9:\(e=v_{1}-v_{0}\)
10:\(fh=fhan(e,v_{2},r_{0},h_{0})\)
11:\(v_{1}=v_{1}+h*v_{2}\)
12:\(v_{2}=v_{2}+h*fh\) ```
**Algorithm 2** Tracking differentiators for angular velocity data. The estimation process starts when the quadrotor hovers stably above an open and wide ground. The controller produces an excitation \(a_{e}\in[0.3,0.5]\) for \(t_{e}=0.1\)s on a single axis, and then the CGP in this axis can be calculated with these flight data. ### Single-Parameter-Tuned Attitude Control Method

The RL method is a frame with these key factors: the agent, environment, state \(\mathbf{s_{t}}\), policy \(\pi\left(\mathbf{s_{t}}\right)\), action \(\mathbf{a_{t}}\), reward \(r_{t}\left(\mathbf{s_{t+1}},\mathbf{a_{t}}\right)\), and state-action value \(V_{t}\left(\mathbf{s_{t}},\mathbf{a_{t}}\right)\). During the training process, the agent takes action \(\mathbf{a_{t}}\) according to state \(\mathbf{s_{t}}\) and policy \(\pi\) in the environment, and obtains a reward \(r_{t}\left(\mathbf{s_{t+1}},\mathbf{a_{t}}\right)\) in return. Value \(V_{t}\) is the sum of the expected future rewards:

\[V_{t}=r_{t}+\lambda r_{t+1}+\lambda^{2}r_{t+1}+...=r_{t}+\lambda V_{t+1} \tag{10}\]

where discount factor \(\lambda\in[0,1)\) is introduced to prevent the infinite sum of future rewards. However, the future reward is unknown, so we assume that an optimal policy \(\pi_{opt}\) can maximize \(V_{t+1}\):

\[ \begin{split} V_{t}\left(\mathbf{s_{t+1}},\mathbf{a_{t}}\right)& =r_{t}\left(\mathbf{s_{t+1}},\mathbf{a_{t}}\right)+\lambda max_{\mathbf{a_{t}} +1\in A}\left(V_{t+1}\left(\mathbf{s_{t+1}},\mathbf{a_{t+1}}\right)\right)\\ &=r_{t}\left(\mathbf{s_{t+1}},\mathbf{a_{t}}\right)+\lambda V_{t+1}\left( \mathbf{s_{t+1}},\pi_{opt}\left(\mathbf{s_{t+1}}\right)\right)\end{split}  \tag{11}\]

where \(A\) is the action space. During the training process, we use the current policy \(\pi\) instead, and this process can make the performance of \(\pi\) gradually close to \(\pi_{opt}\). With the ADRC methods, the quadrotor attitude control is divided into three second-order linear systems with different CGPs and disturbances, so we lock the pitch and yaw axis of a quadrotor as a standard second order linear rotation system rather than a full quadrotor. The agent is similar to an inverted pendulum, which explores the rotation space while discovering the optimal policy and tracking the desired roll angle. Accordingly, the system state variables in training process are represented as: \(\theta\) is the attitude angle, \(\omega\) is the angular velocity, \(b_{0}\) is the CGP of this axis and \(a_{0}\) is the control output. Actuator delay is also considered as a first order system in Equation (4). During the training process, no disturbance is added in the simulation.