<figure><img src="image_1.png" /></figure>

HAL
open science

# Model-Based Reinforcement Learning for Robust End-to-End UAV Control from Simulation to Real System Application

Maximilian Mehl, Alberto Dalla Libera, Ruggero Carli, Marco Tognon

<figure><img src="image_2.png" /></figure>

## To cite this version:

Maximilian Mehl, Alberto Dalla Libera, Ruggero Carli, Marco Tognon. Model-Based Reinforcement Learning for Robust End-to-End UAV Control from Simulation to Real System Application. ERF 2025 - European Robotics Forum, Mar 2025, Stuttgart, Germany. pp.1-6. hal-05021288

HAL Id: hal-05021288

https://hal.science/hal-05021288v1

Submitted on 4 Apr 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Model-Based Reinforcement Learning for
Robust End-to-End UAV Control from
Simulation to Real System Application

Maximilian Mehl<fnref n="1" />,<fnref n="2" />, Alberto Dalla Libera<fnref n="3" />, Ruggero Carli<fnref n="3" />, and
Marco Tognon<fnref n="1" />

**Abstract.** Among optimal control policies for aerial robots, Reinforce-ment Learning promises to improve robustness with low computational effort during deployment. Nevertheless, gathering data to learn the sys-tem's behavior represents a major issue. Ideally, training occurs on the real system, but crashes during early training stages make this impractical. Simulators are preferable but imperfect, creating a *sim2real gap*. More detailed simulators add computational burden, making RL infeasible. We bridge this gap by encapsulating the behavior of a detailed simulator into a Gaussian Process model. We use this computationally light model for training an RL-based control policy, which we test on a real quadrotor. Additionally, the simulator and the corresponding learned model is compared with real flight data showing their accuracy. Our re-sults show the feasibility of conducting policy search in the simulation pipeline and motivate future use of the model-based RL algorithm di-rectly on the real system.

**Keywords:** Aerial Robotics, Model-based RL, Gaussian Process Regession

# 1 Introduction

Learning-based control is gaining more and more traction for deployment on Unmanned Aerial Vehicles (UAVs). The usage ranges from agile flight, outperforming Drone-racing world champions [7], to aerial physical interaction [5, 12]. In those cases, a common approach for learning a control policy is Reinforcement Learning (RL). Here, the training is conducted by deploying the (current) policy on the system and improving according to the observed reward (or costs) of the executed trajectory. In model-based RL, not only the policy, but also a model

<footnote n="1">¹ Univ Rennes, CNRS, Inria, IRISA - UMR 6074, F-35000 Rennes. *</footnote>

<footnote n="2">² maximilian.mehl@inria.fr,</footnote>

<footnote n="3">³ Dipartimento di Ingegneria dell'Informazione, Università degli Studi di Padova**</footnote>

<footnote>* This work was supported by the project ANR-23-CE33-0013 "FlyHandyBot".</footnote>

<footnote>** Alberto Dalla Libera was supported by PNRR research activities of the consortium iNEST (Interconnected North-Est Innovation Ecosystem) funded by the European Union Next GenerationEU (Piano Nazionale di Ripresa e Resilienza (PNRR) –alla sione 4 Componente 2, Investimento 1.5 – D.D. 1058 23/06/2022, ECS_00000043).</footnote>
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Robot used for experimental validation (left) and for training (right).</figcaption></figure>

is trained by a data-driven approach. A dynamics model for mechanical systems can be derived by first principles and identifying its parameters [1], or recently by regression using Gaussian Processes (GPs) [4]. Several works were conducted to incorporate GP modelling into adaptive control of quadrotors, like in [3]. A model for the dynamics of a quadrotor was learned in [9], with application in a predictive control algorithm. In [6], a quadrotor policy for a point-to-point movement is trained in simulation, taking only seconds for the training to generate robust RL policies. In contrast to [6], our focus lies rather on obtaining robust policies using only a few (less than 10) rollouts in the simulation environment, than on fast policy learning. The GP model used in this work takes longer to forward simulate the system, but model and policy learning is possible using only a small number of interactions with the simulator, improving the data efficiency. By doing so, our long-term goal is to enable future learning directly on the real system. To ensure safe learning from the start, the policy obtained in the simulator can be used as initialization.

The presented work is an application of the algorithm from [2]. In this, it is related to [10], where the algorithm is used to teach a robotic manipulator to toss objects. With this algorithm, it is possible to learn a policy with few rollouts on the system. We seek to show the deployment of the algorithm to an aerial robotic system. The contributions are twofold: First, we show the bridging of the sim2real gap by learning a GP model, leveraging the accurate simulation environment from the TELEKYB3 project<fnref n="1" />. The used model-based RL algorithm considers model-uncertainty. By leveraging these two approaches, we show sim2real transfer of a policy, trained using model-based RL. Second, the resulting policy, only trained in simulation, is tested on the real system. With the data from the real-world experiments, the obtained model is validated, to show the accuracy of the used simulator for learning a GP model. Both the robot in the simulator and the real quadrotor are depicted in Fig. 1.

## 2 Model-based RL Algorithm

In this section, the setup for the used Monte Carlo Probabilistic Inference for Learning Control (MCPILCO) [2] Model-based RL algorithm is outlined. The quadrotor state $\boldsymbol{x} = [\boldsymbol{p}^{\top}, \boldsymbol{r}^{\top}, \boldsymbol{v}^{\top}, \dot{\boldsymbol{r}}^{\top}]^{\top}$ is described by the position $\boldsymbol{p} = [x, y, z]^{\top}$, the attitude expressed by the Euler angles $\boldsymbol{r} = [\alpha, \beta, \gamma]^{\top}$ (around the x, y, z axes

<footnote n="1">¹ additional information at: https://git.openrobots.org/projects/telekyb3</footnote>
of the rotated coordinate system), the linear velocities given in the world frame $\mathbf{v} = [u, v, w]^T$ and the rate of change of the Euler angles $\dot{\mathbf{r}} = [\dot{\alpha}, \dot{\beta}, \dot{\gamma}]^T$. Actuation is performed by $N_u = 4$ spinning velocities of the rotors $\mathbf{u} = [\omega_{r0}, \omega_{r1}, \omega_{r2}, \omega_{r3}]^T$.

## 2.1 Gaussian Process Model

To compute the state evolution of the system, a model is required. For GP modelling, $\mathbf{x}_{GP} = [\mathbf{r}^T, \mathbf{v}^T, \dot{\mathbf{r}}^T]^T$ of the robot's state is considered. To complete the input for the GPs, the state is concatenated with $\mathbf{u}$, giving $\bar{\mathbf{x}}_{GP} = [\mathbf{x}_{GP}^T, \mathbf{u}^T]^T$. GP regression is explained in detail in [8]. The model is trained by supervised learning. A dataset of $N_{train}$ training examples $X = [\bar{\mathbf{x}}_{GP,1}, \dots, \bar{\mathbf{x}}_{GP,N_{train}}]^T$ is built together with training targets $\bar{\mathbf{\Delta}}_i = [\mathbf{\Delta}_{i,1}, \dots, \mathbf{\Delta}_{i,N_{train}}]^T$. The main concept is to assume that the training and the test output, $\mathbf{\Delta}_i$ and $\mathbf{\Delta}_i^*$, respectively, are drawn from the same Gaussian distribution with covariance defined by a kernel function $k(\star)$:

$$ \begin{bmatrix} \bar{\mathbf{\Delta}}_i \\ \mathbf{\Delta}_i^* \end{bmatrix} \sim \mathcal{N} \left( \mathbf{0}, \begin{bmatrix} K(X, X) + \sigma_n^2 \mathbf{I} & K(X, \bar{\mathbf{x}}_{GP}^*) \\ K(\bar{\mathbf{x}}_{GP}^*, X) & K(\bar{\mathbf{x}}_{GP}^*, \bar{\mathbf{x}}_{GP}^*) \end{bmatrix} \right) \quad i = 1 \dots N_{GP}, \quad (1) $$

where the elements of matrices $K(\star)$ are the kernel function evaluated at the correspondent input locations. The model to be learned is the speed integration model from [2]. We aim at the mapping from the current state and inputs to the difference in the velocities of the current and the next time step, $\mathbf{\Delta}_\star$. The differences are of the form

$$ \mathbf{\Delta} = [\mathbf{\Delta}_x^T, \mathbf{\Delta}_r^T]^T = [\Delta_u, \Delta_v, \Delta_w, \Delta_\alpha, \Delta_\beta, \Delta_\gamma]^T, \quad (2) $$

where $\mathbf{\Delta}_\star = \star_{t+1} - \star_t$. Equation (1) is used to model $\mathbf{\Delta}$ as $N_{GP} = 6$ independent GPs. The modelling decision to be made is about the covariance in (1), i.e., which kernel function is to be used for the submatrices $K(\star)$. In the presented work, we use the squared exponential kernel

$$ k(\bar{\mathbf{x}}_{GP,i}, \bar{\mathbf{x}}_{GP,j}) = \sigma_s^2 \exp \left( -\|\bar{\mathbf{x}}_{GP,i} - \bar{\mathbf{x}}_{GP,j}\|^2_{\Lambda^{-1}} \right). \quad (3) $$

Replacing (3) into (1), the model depends on the hyperparameters modelling Gaussian noise $\sigma_n$, a scaling parameter $\sigma_s$, and a diagonal matrix of length scales $\Lambda$. These hyperparameters can be optimized using the log likelihood (4) to get a well fitting model.

$$ \log p(\bar{\mathbf{\Delta}}_i | X) = -\frac{1}{2} \bar{\mathbf{\Delta}}_i^T [K + \sigma_n^2 \mathbf{I}]^{-1} \bar{\mathbf{\Delta}}_i - \frac{1}{2} \log |K + \sigma_n^2 \mathbf{I}| - \frac{n}{2} \log 2\pi. \quad (4) $$

The expected value and the variance of the velocity delta for a test input can be computed by (5) and (6) respectively.

$$ \mathbb{E}[\mathbf{\Delta}_i^*] = K(\bar{\mathbf{x}}_{GP}^*, X) [K + \sigma_n^2 \mathbf{I}]^{-1} \bar{\mathbf{\Delta}}_i \quad i = 1 \dots N_{GP} \quad (5) $$

$$ \mathbb{V}[\mathbf{\Delta}_i^*] = K(\bar{\mathbf{x}}_{GP}^*, \bar{\mathbf{x}}_{GP}^*) - K(\bar{\mathbf{x}}_{GP}^*, X) [K + \sigma_n^2 \mathbf{I}]^{-1} K(X, \bar{\mathbf{x}}_{GP}^*). \quad (6) $$
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Results of the experiment in sim (left) and real (right). Commanded points: [1.0, 0.0, 1.0], [0.0, 0.0, 1.5], [-0.5, 1.0, 1.0], [-0.5, 0.5, 0.8] /m.</figcaption></figure>

Equations (5) and (6) can be used in the speed integration model to forward simulate the system, taking into account uncertainty. A downside of using a GP model is the cubic scaling w.r.t. the number of used data points. To address this, a subset of the generated data is used for model learning. For additional information, please refer to [2].

## 2.2 MCPILCO

To derive a control policy, we use the model-based RL algorithm [2] which consists of three steps: i) rollout in the simulator using the current policy, ii) model optimization using (4), and iii) policy optimization by optimizing the cumulative costs via Monte Carlo sampling. The reference provides further details about the algorithm. The used algorithm propagates model uncertainty by using particles, other approaches show propagation by using an Unscented Kalman Filter [11]. In particular, we aim at a control policy for point-to-point movement that maps the controller state $x_{\pi,t} = [x_t^\top, e_t^\top]^\top$ to desired rotor speeds. The entries of the error vector $e_{x_{\pi,t}} = \star_t - \star_d$ are computed by comparing a controllable Degree of Freedom (DOF) at a time step with its desired value. As suggested in [2] the policy is a Radial Basis Function network, thus the parameters to optimize are the centers and weights on the Gaussians, and the cost function is chosen to be a saturating function of the error variables. Since the cost consists only of the cumulated errors, the policy learns to minimize the control error trying to find a time optimal trajectory.

An upside of using RL for a point-to-point movement is that trajectory planning and control is encapsulated into the same policy. Nevertheless, it is important to point out that the movement considered here is just a first step to policies for more complex tasks, e.g. aerial physical interaction.
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Comparison of predicted and measured velocity deltas. Mean squared errors for the DOF are computed to: MSE(Δₓ) = 3.2 × 10⁻⁶, MSE(Δᵧ) = 2.5 × 10⁻⁶, MSE(Δz) = 1.7 × 10⁻⁵, MSE(Δα) = 4.6 × 10⁻³, MSE(Δβ) = 3.1 × 10⁻³, MSE(Δγ) = 3.5 × 10⁻³.</figcaption></figure>

# 3 Experiments

To test and validate the capabilities of the obtained policy, several consecutive point-to-point movements are performed in the simulator and with the real quadrotor. The used policy is trained using 400 particles for Monte Carlo sampling. During training, these particles are spawned in a cube around the goal-point with a side length of 0.5 m around the goal-point $x_d = [x_d, y_d, z_d, \gamma_d]^\top = [0, 0, 1.7, 0]^\top$. This way, each particle is initialized with a different state, and different movements to reach the goal are learned.

## 3.1 Deployment of the policy in simulation and real experiments

In Fig. 2 we compare the simulation and experimental results. In both cases, the quadrotor is commanded to the same set of points. The policy allows for zero shot sim2real transfer, with no additional fine-tuning necessary. Due to the limited size of the flight arena, the maximal displacement in z-direction is reduced w.r.t. the training case. The same goal command was issued for 7 s to test the stabilization capabilities.

The achieved accuracy is similar in the simulation and experiment case. In both cases, the policy is more accurate when going to $x = y = 0.0$ m compared to the accuracy to different points. This is because during training this fixed goal-point was considered. On the attitude and rotor speed level, the obtained results lie in a similar range. A clear difference between the simulation and experiment results is visible at the first commanded point. This difference arises because at this point, the quadrotor is commanded close to a wall. In the simulation, no walls or other obstacles are considered, and therefore the changed aerodynamics due to the proximity to the wall aren't visible.

## 3.2 Experimental validation of the GP model

State and input data from experimental flight is collected and fed to the model learned in simulation. Results are visualized in Fig. 3 where we show the error between the velocity deltas predicted by the GPs and the measurement. The GPs can predict the mean of the real data accurately, except for the Δα DOF, where a small bias is visible. This bias can be attributed to the results at the first point, where the aerodynamic effects of the proximity to a wall are not considered by the GP. The higher variance in the results for the attitude part can be explained by higher proportion of noise present in the measured deltas compared to the measurement for the position part.
## 4 Conclusion

Comparison of the results achieved in simulation and on the real system, and testing the obtained model on the real flight data, shows the capabilities of the used simulator. The accurate simulator, integrating software in the loop of the real system, allows the training of a GP model which is precise while being computationally efficient for model-based RL. This allowed the zero shot deployment of the MCPILCO algorithm on a real aerial robot. Future work will include refinement of the learning directly on the real system. In fact, when considering more difficult tasks than a point-to-point movement, such as aerial physical interaction, additional training on the real system could be beneficial for robust task execution.

## References

1. Abbeel, P., Coates, A., Ng, A.Y.: Autonomous Helicopter Aerobatics through Apprenticeship Learning. The International Journal of Robotics Research 29(13), 1608–1639 (2010)

2. Amadio, F., Dalla Libera, A., Antonello, R., Nikovski, D., Carli, R., Romeres, D.: Model-Based Policy Search Using Monte Carlo Gradient Estimation With Real Systems Application. IEEE Transactions on Robotics 38(6), 3879–3898 (2022)

3. Cen, R., Jiang, T., Tang, P.: Modified Gaussian process regression based adaptive control for quadrotors. Aerospace Science and Technology p. 106483 (2021)

4. Deisenroth, M.P., Fox, D., Rasmussen, C.E.: Gaussian Processes for Data-Efficient Learning in Robotics and Control. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(2), 408–423 (2015)

5. E. Cuniato, E., Geles, I., Zhang, W., Andersson, O., Tognon, M., Siegwart, R.: Learning to open doors with an aerial manipulator. In: 2023 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (2023)

6. Eschmann, J., Albani, D., Loianno, G.: Learning to Fly in Seconds. IEEE Robotics and Automation Letters 9(7), 6336–6343 (2024)

7. Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., Scaramuzza, D.: Champion-level drone racing using deep reinforcement learning. Nature 620(7976), 982–987 (2023)

8. Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2005)

9. Smith, A.J., AlAbsi, M., Fields, T.: Heteroscedastic Gaussian Process-based System Identification and Predictive Control of a Quadcopter. In: 2018 AIAA Atmospheric Flight Mechanics Conference. American Institute of Aeronautics and Astronautics, Kissimmee, Florida (2018)

10. Turcato, N., Libera, A.D., Giacomuzzo, G., Carli, R.: Teaching a Robot to Toss Arbitrary Objects with Model-Based Reinforcement Learning. In: 2023 9th International Conference on Control, Decision and Information Technologies (CoDIT), pp. 1126–1131. IEEE, Rome, Italy (2023)

11. Wu, X., Wedernikow, E., Huber, M.F.: Data-Efficient Uncertainty-Guided Model-Based Reinforcement Learning with Unscented Kalman Bayesian Neural Networks. In: 2024 American Control Conference (ACC), pp. 104–110. IEEE, Toronto, ON, Canada (2024)

12. Zhang, W., Ott, L., Tognon, M., Siegwart, R.: Learning Variable Impedance Control for Aerial Sliding on Uneven Heterogeneous Surfaces by Proprioceptive and Tactile Sensing. IEEE Robotics and Automation Letters 7(4), 11,275–11,282 (2022)