DOI: 10.5061/dryad.3tx95x6md
title: Reaching the limit in autonomous racing: Optimal control versus reinforcement learning

description: A central question in robotics is how to design a control system for an
 agile mobile robot. This paper studies this question systematically,
 focusing on a challenging setting: autonomous drone racing. We show that a
 neural network controller trained with reinforcement learning (RL)
 outperformed optimal control (OC) methods in this setting. We then
 investigated which fundamental factors have contributed to the success of
 RL or have limited OC. Our study indicates that the fundamental advantage
 of RL over OC is not that it optimizes its objective better but that it
 optimizes a better objective. OC decomposes the problem into planning and
 control with an explicit intermediate representation, such as a
 trajectory, that serves as an interface. This decomposition limits the
 range of behaviors that can be expressed by the controller, leading to
 inferior control performance when facing unmodeled effects. In contrast,
 RL can directly optimize a task-level objective and can leverage domain
 randomization to cope with model uncertainty, allowing the discovery of
 more robust control responses. Our findings allowed us to push an agile
 drone to its maximum performance, achieving a peak acceleration greater
 than 12 times the gravitational acceleration and a peak velocity of 108
 kilometers per hour. Our policy achieved superhuman control within minutes
 of training on a standard workstation. This work presents a milestone in
 agile robotics and sheds light on the role of RL and OC in robot control.
description: This dataset contains the data collected in both simulation and
 real world. In the simulation, we run experiments with three different
 approaches, including trajectory tracking using model predictive control,
 contouring control using model predictive contouring control, and gate
 progress using reinforcement learning. We compare all methods using a
 nominal simulator and a realistic simulator called BEM (Blade element
 momentum). We developed a high-performance drone to push the limit of our
 drone.  We collect trajectory flew using this high-performance drone in
 the real world. We compare our policy against three professional human
 pilots. We collected the best trajectory from these three
 pilots. 
description: The structure of the dataset shows are following: (1) RLvsOC: This folder
 contrains the simulation experiment for the comparison between optimal
 control and reinforcement learning. \- MPC: Trajectory tracking using an
 nonlinear model predictive control \(MPC\)\. It contains 50 trials\,
 indicated by the folder name\, from trial\_0 to trial\_49\. Inside each
 folder\, there are 4 different csv files: \- \*ref\_bem\.csv\* is the
 reference trajectory in BEM simulation \- \*ref\_nominal\.csv\* is the
 reference trajectory in the nominal simulation \- \*traj\_bem\.csv\* is
 the executed trajectory in the BEM simulation \- \*traj\_nominal\.csv\* is
 the executed trajectory in the nominal simulation \- MPCC: Path following
 using a model predictive contouring control \(MPCC\)\. It also contains 50
 trials\, indicated by the folder name\. Inside each folder\, there are 4
 different csv fildes: \- \*ref\_bem\.csv\* is the reference path in BEM
 simulation \- \*ref\_nominal\.csv\* is the reference path in the nominal
 simulation \- \*traj\_bem\.csv\* is the executed trajectory in the BEM
 simulation \- \*traj\_nominal\.csv\* is the executed trajectory in the
 nominal simulation \- PPO: Gate progress maximization using proximal
 policy optimization \(PPO\)\. It contains two csv files\. Since we use
 parallezied environment\, each file contains all the 50 trials\, the trail
 number is indicated by the \*env\_id\* inside the csv\. PPO does not
 require reference trajectory or path\. \- \*bem\_traj\.csv\* is the
 executed trajectory in the BEM simulation \- \*nominal\_traj\.csv\* is the
 executed trajectory in the nominal simulation (2)
 HumanPilots\_MarvTrack\_6sDrone: This folder contains the best trajectory
 flew by the three professional human pilots. All human pilot uses a 6s
 Drone. \- best\_pilot1\.csv: the best lap \(trajectory\) by Pilot 1 \(Alex
 Vanover\) \- best\_pilot2\.csv: the best lap \(trajectory\) by Pilot 2
 \(Thomas Bitmatta\) \- best\_pilot3\.csv: the best lap \(trajectory\) by
 Pilot 3 \(Marvin Schaepper\) (3) OptimizationMethods: This folder contains
 the data for the comparison between MPC and RL for trajectory tracking \-
 MPC: Using an nonlinear MPC to track a time\-optimal trajectory\.
 \*mpc\_bem\_\* means the experiment in the BEM simulation\,
 \*mpc\_nominal\_\* indicates the experiment in the nominal simulation\.
 \*\_ref\.csv\* is the reference trajectory\, \*\_traj\.csv\* is the
 executed trajectory\. \- PPO: Using a reinforcement learning controller to
 track the same time\-optimal trajectory\. \*rl\_bem\_\* means the
 experiment in the BEM simulation\, \*rl\_nominal\_\* indicates the
 experiment in the nominal simulation\. \*\_ref\.csv\* is the reference
 trajectory\, \*\_traj\.csv\* is the executed trajectory\. (4)
 OptimizationObjectives: This folder contains the data for the comparison
 between optimization objectives using RL. \- PPO\_Racing: Using a
 reinforcement learning controller to optimize a gate progress for drone
 racing\. \*rl\_bem\_traj\.csv\* means the experiment in the BEM
 simulation\, \*rl\_nominal\_traj\.csv\* indicates the experiment in the
 nominal simulation\. \- PPO\_Tracking: Using a reinforcement learning
 controller to track a time\-optimal trajectory\. \*rl\_bem\_\* means the
 experiment in the BEM simulation\, \*rl\_nominal\_\* indicates the
 experiment in the nominal simulation\. \*\_ref\.csv\* is the reference
 trajectory\, \*\_traj\.csv\* is the executed trajectory\. (5)
 RL\_MarvTrack\_6sDrone: This folder contains the best trajectory
 (best.csv) using the 6s drone in the Marv track for the competition. (6)
 RL\_SplitS\_6sDrone: This folder contains the real-world experiment data
 using the 6s drone in the SplitS track. It has 5 different folders, run
 with different batteries. The name of each folder indicates the experiment
 time, such as date and time. Inside each folder, it contains two files,
 one is the battery log (battery.csv) and another one is the vehicle states
 (states.csv). (7) README.md: a copy of the readme.md
 ==============================CSV
 headers================================================ The column headers
 for the CSV files are explained as follows: \- t: Time in seconds \(s\)\.
 simulation timestamp\. \- px\, py\, pz: Position coordinates in meters
 \(m\) in 3D space\, representing the x\, y\, and z positions\. \- qw\,
 qx\, qy\, qz: Quaternion components which are unitless\. Quaternions
 represent rotations in 3D space with the four components being w\, x\, y\,
 and z\. \- vx\, vy\, vz: Velocity in meters per second \(m/s\) for the x\,
 y\, and z directions\. \- omex\, omey\, omez: Angular velocities in
 radians per second \(rad/s\) around the x\, y\, and z axes\. \- accx\,
 accy\, accz: Accelerations in meters per second squared \(m/s^2\) for the
 x\, y\, and z directions\. \- taux\, tauy\, tauz: Torques \(or moments\)
 in Newton meters \(N·m\) around the x\, y\, and z axes\. \- jerkx\,
 jerky\, jerkz: Jerk values in meters per second cubed \(m/s^3\)\,
 representing the rate of change of acceleration in the x\, y\, and z
 directions\. \- snapx\, snapy\, snapz: Snap values in meters per second to
 the fourth power \(m/s^4\) for the x\, y\, and z directions\. \- bomex\,
 bomey\, bomez: \(placeholder only\, not used\, the value can be ignored\)
 \- baccx\, baccy\, baccz: \(placeholder only\, not used\, the value can be
 ignored\) \- mot1\, mot2\, mot3\, mot4: \(placeholder only\, not used\,
 the value can be ignored\) \- motdex1\, motdex2\, motdex3\, motdex4:
 \(placeholder only\, not used\, the value can be ignored\) \- f1\, f2\,
 f3\, f4: \(N\, newton\)\. forces associated with each of the four motors\.
 \- env\_id: Environment ID for parallelized simulation in Flightmare\, no
 unit \- done: a binary value indicating completion\. 0 \- not done\, 1 \-
 done \- flightmode: The mode of flight \(0 \- flying mode\, 1 \- Passing
 the gate\, 2 \- crashing into the gate\, 3 \-\- crashing into the tube\, 4
 \- crashing on the ground\, 5 \- crashing into the world box\, 6 \-
 episode done\, 7 \- inside the gate\)\, no unit \- flightlap: The lap
 number of the flight test\. \- laptime: \(seconds\, s\) Time taken for a
 specific lap\. \- reward: A metric used in a reinforcement learning
 setup\\\, indicating the reward for a particular action given a state\. no
 unit \- value: A general value that represents the output of the state
 value function\, no unit\.