## 3 Quadcopter Control
### Attitude Control  
To control the attitude of the robot, an architecture comprising of RL-based adaptive controllers is proposed in this study. The outer loop PID controllers were designed to generate the \(u1_{x}\) and \(u1_{y}\) virtual control signals as described in Equations (25)-(28). Equation (29) is used to convert the control commands from outer loop PIDs to the necessary roll and pitch reference values for the inner-loop PID controllers. \[e_{x}=x_{des}-x \tag{25}\]  
\[e_{y}=y_{des}-y \tag{26}\]  
\[u1_{x}=\left(Kp_{x}\cdot e_{x}\right)+\left(Kd_{x}\cdot\dot{e}_{x}\right)+ \left(Ki_{x}\int e_{x}\;dt\right) \tag{27}\]  
\[u1_{y}=\left(Kp_{y}\cdot e_{y}\right)+\left(Kd_{y}\cdot\dot{e}_{y}\right)+ \left(Ki_{y}\int e_{y}\;dt\right) \tag{28}\]  
\[\left[ \begin{array}{c}\varphi_{d}\\ \theta_{d}\end{array} \right]=\frac{1}{g}\left[ \begin{array}{c}\sin(\psi_{ds })-\cos(\psi_{ds})\\ \cos(\psi_{ds})-\sin(\psi_{ds})\end{array} \right]\left[ \begin{array}{c}u1_{ x}\\ u1_{y}\end{array} \right] \tag{29}\]  
As indicated in Equations (30)-(32), three PID controllers were implemented in the inner loop PID control block to provide manipulated variables for robot attitude control. \[u_{2}=Kp_{2}(\varphi_{d}-\varphi)+Kd_{2}\big{(}\dot{\varphi}_{d}-\dot{\varphi }\big{)}+Ki_{2}\int(\varphi_{d}-\varphi)\;dt \tag{30}\]  
\[u_{3}=Kp_{3}(\theta_{d}-\theta)+Kd_{3}\Big{(}\dot{\theta}_{d}-\dot{\theta} \Big{)}+Ki_{3}\int(\theta_{d}-\theta)\;dt \tag{31}\]  
\[u_{4}=Kp_{4}(\psi_{d}-\psi)+Kd_{4}\Big{(}\dot{\psi}_{d}-\dot{\psi}\Big{)}+Ki_ {4}\int(\psi_{d}-\psi)\;dt \tag{32}\]  
The optimal control gains for the inner loop PID controllers were obtained, based on several trials and errors. The obtained gain coefficients are listed in Table 1. The control gains of outer loop PID controllers are actively estimated and adjusted by the RL agent. ### Attitude Control  
In this study, the proposed altitude controller utilizes a linear model predictive controller and a gravity compensator in its controlling architecture. The gravity compensator is responsible for alleviating the effect of forces that arise from fluctuations in the weight of the robot. The scaling factor of the compensator is adaptively updated using the RL policy. The proposed algorithm aimed to mitigate the impact of disturbances arising from changes in the weight of the robot on the performance of the aerial robot in trajectory tracking and altitude stabilization. The MPC is based on an iterative, finite-horizon robot model optimization. The present states of the quadcopter are sampled at time t, and a cost-minimizing control strategy for a relatively short time horizon in the future \([\text{t},\text{t}+\text{T}]\) is computed (using a numerical minimization technique). At each control interval, model predictive control solves an optimization problem, a quadratic program (QP). Until the next control interval, the solution generated a sequence of manipulated variables to be \begin{table}
\begin{tabular}{c c c c} \hline \hline
**PID Control** & \(Kp\) & \(Ki\) & \(Kd\) \\ \hline Roll (\(\varphi\)) & 0.021 & 0.011 & 0.003 \\ Pitch (\(\theta\)) & 0.014 & 0.03 & 0.001 \\ Yaw (\(\psi\)) & 0.002 & 0.07 & 0.013 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Control gain values of the inner-loop PID controllers. applied to the robot. A series of online optimizations are run to estimate possible state trajectories that would arise from the present states. Furthermore, the solution identifies a cost-minimizing control strategy (by the solution of Euler-Lagrange equations) from t until time ran out at t + T. Although the MPC computes a series of manipulated variables, only the first step of the computed control strategy is applied to the quadcopter, after which the updated states of the robot are sampled again and the computations are repeated using the updated states, resulting in computation of fresh control inputs and a new anticipated state route. Figure 3 shows the relationship between the prediction horizon and generated control inputs.