# Robust Quadrotor Control through Reinforcement Learning with Disturbance Compensation
## 4 Experiments
### RL Controller Training  
In the RL training process, we followed the dynamic equations in Section 2.1 and constructed a simulation environment in Python to generate training data. In the simulation environment, the state space of the MDP comprised the position, velocity, attitude, and angular velocity of the quadrotor. Moreover, the four motors thrust outputs were chosen as the action space. The training process follows the work in [22] using two processes, one for data collection and another for value and policy network update. The update is based on off-policy training, and the main difference between on-policy is the on-policy only uses the collected data once and be cleaned up after each time neural network updates. On the contrary in the off-policy training, the collection thread keeps generating the trajectory data and neural network updating thread can reuse the data from collection which accelerates the learning process. In data collection process, the quadrotor was randomly launched in a 2 m cubic space with random states. The training data, which comprised the quadrotor states, action, probability, and reward, were recorded in each episode which contained 200 steps in two seconds flight, and then saved as a single data trajectory in a memory buffer. The normalized reward function for evaluating the current state of the quadrotor is as follows:  
\[r=-(0.002\big{\|}e_{q}\big{\|}+0.002\big{\|}e_{p}\big{\|}+0.002\big{\|}a\big{\|}), \tag{20}\]  
where \(e_{q}\) is the vehicle angle error, \(e_{p}\) is the vehicle position error, and \(a\) is the motor thrust command for constraining the energy cost. When the number of data trajectories in memory buffer exceeded 10, the training process starts. In training process, trajectories were randomly sampled from the memory buffer. The advantage and value functions were defined recursively and calculated in reverse direction which depend on the future time \(t+1\). The functions were estimated according to  
\[A_{t}^{trace}=A_{t}+\gamma\min(1,\frac{\pi_{t+1}}{\mu_{t+1}})A_{t+1}^{trace}, \tag{21}\]  
and  
\[V_{t}^{trace}=V_{t}+\min(1,\frac{\pi_{t}}{\mu_{t}})A_{t}^{trace}. \tag{22}\]  
With the two equations above, we use stochastic gradient descent to optimize the objectives as follows:  
\[ \begin{split} maximize\ L_{policy}&=\sum_{(s,a)\in T }\min\bigg{[}\bigg{(}\frac{\pi(a|s)}{\mu(a|s)}-1\bigg{)}A^{trace},\epsilon|A^{ trace}|\bigg{]}\\ minimize\ L_{value}&=\frac{1}{|T|}\sum_{(s,a)\in T} \big{(}V(s)-V^{trace}\big{)}^{2}.\end{split}  \tag{23}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Weight (g)** & \(\mathbf{I_{xx}},\mathbf{I_{yy}},\mathbf{I_{zz}}\) (**kgm2**) \\ \hline
665 & 0.0023, 0.0025, 0.0037 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Physical parameters of the quadrotor. To approximate the function \(\pi(a|s)\) for proposing actions and \(V(s)\) for predicting the state value, two neural network were formulate as following equations, where stochastic Gaussian policy was used for the actor network:  
\[\theta(s) =\bar{h}_{1}^{32}\circ\bar{h}_{0}^{32}(s) \tag{24}\] \[\alpha(s) =\sin\circ\underline{y}_{2}^{4}\circ\theta(s)\] (25) \[\sigma(s) =\frac{1}{2}+\frac{1}{2}\cos\circ\underline{y}_{2}^{4}\circ \theta(s)\] (26) \[\pi(a|s) =\frac{1}{\sqrt{2\pi\sigma^{2}(s)}}e^{-\frac{(\alpha-\alpha(s))^{ 2}}{2\sigma^{2}(s)}}. \tag{27}\]  
The state value function can be approximated as follows:  
\[V(s)=y_{2}^{1}\circ h_{1}^{128}\circ h_{0}^{128}(s), \tag{28}\]  
where \(h_{i}^{j}\) and \(y_{i}^{j}\) are the \(i\)th hidden layer and output layer of neural networks with width \(j\), \(\circ\) is the fully connected activation function. In both the actor and critic networks, the input state \(s\) is the quadrotor's position, velocity, attitude and angular velocity. The sin and cos functions are used to constrain the output range. A rectified linear unit (ReLU) is used as the activation function due to its characteristic of fast calculation and easy implementation in a microcontroller unit.