DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control
Authors: Kevin Huang
Rwik Rana
Alexander Spitzer
Guanya Shi
Byron Boots
Venue: CoRL 2023 Oral (2023)
Keywords: Quadrotor
Reinforcement Learning
Adaptive Control

Abstract: Precise arbitrary trajectory tracking for quadrotors is challenging due to unknown nonlinear dynamics, trajectory infeasibility, and actuation limits. To tackle these challenges, we present DATT, a learning-based approach that can precisely track arbitrary, potentially infeasible trajectories in the presence of large disturbances in the real world. DATT builds on a novel feedforward-feedback-adaptive control structure trained in simulation using reinforcement learning. When deployed on real hardware, DATT is augmented with a disturbance estimator using $\mathcal{L}_1$ adaptive control in closed-loop, without any fine-tuning. DATT significantly outperforms competitive adaptive nonlinear and model predictive controllers for both feasible smooth and infeasible trajectories in unsteady wind fields, including challenging scenarios where baselines completely fail. Moreover, DATT can efficiently run online with an inference time less than 3.2ms, less than 1/4 of the adaptive nonlinear model predictive control baseline.

DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control
Kevin Huang
Rwik Rana University of Washington rwik2000@uw edu
University of Washington
kehuang@cs washington edu
Alexander Spitzer University of Washington spi tzzereess... www.:/etss..aiingnoo.eeeeee
Guanya Shi
Carnegie Mellon University guanyas@andrev. cmu. edu
Byron Boots
University of Washington
bbootsecs, washington edu edu
Abstract: Precise arbitrary trajectory tracking for quadrotors is challenging due to unknown nonlinear dynamics, trajectory infeasibility, and actuation limits. To tackle these challenges, we present Deep Adaptive Trajectory Tracking (DATT). learning-based approach that can precisely track arbitrary, potentially infeasible trajectories in the presence of large disturbances in the real world. DATT builds on novel wewp://wwdd-eeee---addieeeee
control structure trained in simulation us- ing reinforcement learning. When deployed on real hardware, DATT is augmented with a disturbance estimator using & adaptive control in closed-loop, without any tine-tuning. DATT significantly outperforms competitive adaptive nonlinear and model predictive controllers for both feasible smooth and infeasible trajectories unsteady wind fields, including challenging scenarios where baselines completely fail. Moreover, DATT can efficiently run online with inference time less than 3.2 ms, less than 1/4 of the adaptive nonlinear model predictive control baseline!.
Keywords: Quadrotor, Reinforcement Learning, Adaptive Control
1 Introduction
Executing precise and agile flight maneuvers is important for the ongoing commoditization of un- manned aerial vehicles (UAVss, in applications such drone delivery, rescue and search, and urban air mobility. In particular, accurately following arbitrary trajectories with quadrotors is among the most notable challenges to precise flight control for the following reasons. First, quadrotor dynamics are highly nonlinear and underactuated, and often hard model due to unknown system parameters (e.g., motor characteristics) and uncertain environments (c.g., complex aerodynamics from unknown wind gusts). Second, aggressive trajectories demand operating at the limits of system performance, requiring awareness and proper handling of actuation constraints, especially for quadrotors with small thrust-to-weight ratios. Finally, the arbitrary desired trajectory might not dynamically fea- sible (i.c., impossible to stay on such trajectory), which necessities long-horizon reasoning and optimization in real-time. For instance, stay close to the five-star trajectory in Fig. 1, which is infeasible due to the sharp changes of direction, the quadrotor must predict, plan, and react online before the sharp turns.
Traditionally, there two commonly deployed control strategies for accurate trajectory follow- ing with quadrotors: nonlinear control based on differential flatness and model predictive control
Videos and demonstrations in Attps://siten Beaiites.ggoee and code in http:: httt:::///itthue comm/Keeriifffunngg/DDr....
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

(a) Ours
(p) MPC
(c) Nonlinear
(d) Ouus
(e)
0.0
0.4
0.2
0.5
0.0
0.0
-0.5
-0.50
0.50
(m)
Time
(0.11)
Nonlineer (0.30)
(0.05)
Figure 1: Trajectory visualizations for example infeasible trajectories. (a-c) Long-exposure photos photo of
of different methods for an equilateral triangle reference trajectory. (d) method for five-pointed star reference trajectory. (e) Quantitative comparisons between approach and baselines for the five-pointed star. Numbers indicate the tracking error in meters.
(MPC). However, nonlinear control methods, despite their proven stability and efficiency, are con- strained to differentially flat trajectories (i.e., smooth trajectories with bounded velocity, accelera- tion, jerk, and snap) satisfying actuation constraints [1, 2, 3]. On the other hand, MPC approaches can potentially incorporate constraints and non-smooth arbitrary trajectories [4, 5J, but their perfor- mances heavily rely on the aceuracy of the model and the optimality of the solver for the underlying nonconvex optimization problems, which could also be expensive to run online.
Reinforcement learning (RL) has shown its potential flexibility and efficiency in trajectory tracking problems [6. 8]. However, most existing works focus on tracking smooth trajectories in stationary environments. In this work, we aim to design RL-based flight controller that can (1) follow feasi- ble trajectories accurately traditional nonlinear controllers and MPC approaches; (2) accurately follow arbitrary infeasible and dynamic trajectories to the limits of the hardware platform; and (3) adapt to unknown system parameters and uncertain environments online. Our contributions are:
We propose DATT, a novel pipeline for RL-based controllers to track arbitrary trajectories. training, this policy is condi- tioned on ground-truth translational disturbance in simulator, and such disturbance is estimated in real using L adaptive control in closed-loop:
policy architecture and training
On real, commercially available, lightweight, and open-sourced quadrotor platform (Crazyflie 2.1 with upgraded motors), show that our approach can track feasible smooth trajectories with 27%%38% smaller errors than adaptive nonlinear or adaptive MPC baselines. Moreover, our approach can effectively track infeasible trajectories where the nonlinear baseline completely fails, with 39% smaller error than MPC and 1/4th the computational time;
On the real quadrotor platform, we show that our approach can adapt zero-shot to unseen turbulent wind fields with an extra cardboard drag plate for both smooth desired trajectories and infeasible trajectories. Specifically, for smooth trajectories, our method achieves up 22% smaller emoos than the state-of-the art adaptive nonlinear control method. In the most challenging scenario (in- feasible trajectories with wind and drag plate), our method significantly outperforms the adaptive MPC approach with 15% less error and 1/4th of the computation time.

---

2 Problem Statement and Related Work
2.1 Problem Statement
In this paper, we let i' denote the derivative of continuous variable æ regarding time. We consider the following quadrotor dynamics:
p
mu mg + Reafx + d jw X
(1a)
R RS(W),
(Ib)
where p, v.gs E F2 are position, velocity, and gravity vectors in the world frame, R E SO(3) is the attitude rotation matrix, w E R is the angular velocity in the body frame, m, J are mass and inertia matrix, E3 = [0; 0; 1], and 5(-) R + so(3) maps a vector to its skew-symmetric matrix form. Moreover, the time-variant translational disturbance, which includes parameter mismatch (e-g., mass error) and environmental perturbation (e.g., wind perturbation) [9, 10, 11, 12]. The control input is the total thrust fr and the torque in the body frame. For quadrotors, there is linear invertible actuation matrix between [/>; r and four motor speeds.
We let 2 denote the temporal discretization of 3-D trajectory tracking problem with the desired trajectory p pg. ... ,p4, with average tracking error as the performance metric: + Ei= llp: - p We do not have any assumptions on the desired trajectory p². In particular, pd not necessarily differentiable or smooth.
time step E Z+. In this work, we focus on the
-pp
2.2 Differential Flatness
The differential flatness property of quadrotors allows efficient generation of control inputs to follow smooth trajectories [1, 5]. Differential flatness has been extended to account for unknown linear dis- turbances [3], learned nonlinear disturbances [13], and also to deal with the singularities associated with pitching and rolling past 90 degrees [14]. While
methods can show impressive performance for smooth and aggressive trajectories, they struggle with nondifferentiable trajectories or trajectories that require reasoning about actuation constraints.
2.3 Model Predictive Control (MPC)
MPC is a widely used optimal control approach that online optimizes control inputs over a finite time horizon, considering system dynamics and constraints [15, 16).
Model Predictive Path Integral Control (MPPI) [4, 17] is sampling- baapee MPC incorporating path integral control formulation and stochastic sampling. Unlike deterministic optimization, MPPI em- ploys a stochastic optimization approach where control sequences are sampled from a distribution. These samples are then evaluated based on cost function, and the distribution is iteratively updated to improve control performance. Recently MPPI has been applied quadrotor control [18, 19].
Gradient-bard based nonlinear MPC techniques have been widely used for vonary-winged-baet based flying robots or drones. Hanover al. [12] and Sun al. [5] have shown good performance of nonlinear MPC in agile trajectory tracking of drones and adaptation to external perturbations. Moreover, these techniques are being used for vision-basee based agile maneuvers of drones [20, 7].
However, for either sampling-based or gradient-based MPC, the control performance heavily relies on the optimality of the optimizer for the underlying nonconvex problems. Generally speaking, MPC-based approaches require much more computing than -ittrnntial--aatess---ee Moreover, MPC's robustness and adaptability for infeasible trajectories remain unclear since exist- ing works consider smooth trajectory tracking. In this paper, we implemented MPPT [4] and La augmented MPPI [18] for our baselines.
methods [5].
2.4 Adaptive Control and Disturbance Estimation
Adaptive controllers aim to improve control performance through online estimation of unknown system parameters in closed-loop. For quadrotors, adaptive controllers typically estimate a three- dimensional force disturbance d [21, 10, 22, 23, 18). Most recently, L1 adaptive control for quadro- tors [11] has been shown to improve trajectory tracking performance in the presence of complex and time- varying disturbances such sloshing payloads and mismatched propellers. Recently, deep- based adaptive flight controllers have also emerged [10, 24, 25).
3

---

rel trajectory pi: .P..".
feedforward encoder
feedforward embedding Ag
R
state
feedback error body frame
low-level controller
L adaptive control
disturbance estimation d,
nominal model
desired fs and w
policy net
Figure 2: Algorithm Overview. Blue, yellow, and green blocks represent feedforward, feedback, and adaptation modules respectively. In training the policy has access to the true disturbance d whereas in real we use Li adaptive control to get the disturbance estimation d closed-loop.
Learning dynamical models is a common technique to improve quadrotor trajectory tracking per- formance [9, 26, 27, 28] and can provide more accurate disturbance estimates than purely reactive adaptive control, due the model of the disturbance over the state and control space. In this work, we use the disturbance estimation from & adaptive control, but note that our method can lever- age any disturbance estimation or model learning techniques.
In particular, Rapid Motor Adaptation (RMA) is supervised learning-based approach that aims to predict environmental parameters using history of stat-acction pairs, which are then inputted the controller [29]. This approach has been shown to work for real legged-robots, but we find that it can be susceptible to domain shift during sim2real transfer on drones.
2.5 Reinforcement Learning for Quadrotor Control
Reinforcement learning for quadrotor stabilization is studied in [6, 30, 24). Molchanov et al. [30] uses domain randomization show policy transfer between multiple quadrotors. Kaufmann et al. [31] compares three different policy formulations for quadrotor trajectory tracking and finds that outputting body thrust and body rates outperforms outputting desired linear velocities and individual rotor thrusts. [31] only focuses on feasible trajectories while in this work, we aim to track infeasible trajectories as accurately as possible. Simulation-baal learning with imitation learning to an expert MPC controller is used to generate acrobatic maneuvers in [7]. In this work, we focus on trajectories and environments for which obtaining an accurate expert even in simulation difficult or expensive and thus use reinforcement learning learn the controller.
Methods
3.1 Algorithm Overview
A high-level overview of DATT is given in Fig. 2. Using model-free RL, DATT learns a neural net- work quadrotor controller æ capable of tracking arbitrary reference trajectories, including infeasible trajectories, while being able adapt to various environmental disturbances, even those unseen during training. We condition our policy on learned feedforward embedding h, which encodes the desired reference trajectory, in the body frame, over fixed time horizon, well the force disturbance d in Eq. (1).
The state D, consists of the position p, the velocity v, and the orientation R, represented as quaternion q. We convert p, the body frame and input them to Our policy controller outputs which includes the desired total thrust fx,des and the desired body rates Waes- In summary, controller functions as follows:
h; = (R, (p: P()) R. (p: - p(+H))
(2a)
(R, pt. R, qt. h R (p: p:), dd)
(2b)

---

We define the expected reward for our policy conditioned on the reference trajectory as follows:
J (TTPP()HH) B(
Σ r(z,, ur
a ((:++)
(3a)
raa,, uel MelPer+H) = lpp. Pfll 0.5/| + 0.1
(3b)
de denotes the yaw of the drone. The reward function optimizes for accurate position and yaw tracking, with a small velocity regularization penalty. π and Φ are jointly optimized with respect to using the Proximal Policy Optimization (PPO) algorithm [32].
3.2 Arbitrary Trajectory Tracking
Classical controllers, such as differential-flatnes. controllers, rely on higher-order position deriva- tives of the reference trajectory for accurate tracking (velocity, acceleration, jerk, and snap), which are needed for incorporating future information about the reference, i.e., feedforward control. How- ever, arbitrary trajectories can have undefined higher order derivatives, and exact tracking may not be feasible. With RI., a controller be learned to optimally track arbitrary reference trajec- tory, given just the desired future positions p. Thus, we input just the desired positions, in the body-frame, into a feedforward encoder Φ, which learns the feedforward embedding that contains the information of the desired future reference positions. For simplicity, we assume the desired yaw for all trajectories is zero. The reference positions are provided evenly spaced from the current time t to the feedfoward horizon t + H, and are transformed into the body frame.
3.3 Adaptation to Disturbance
During training in simulation, we add a random time-vare time-narying force perturbation d the environ- ment. We use L1 adaptive control [11, 33] to estimate d, which directly passed into our policy network during both training and inference. Li adaptive control first builds closed-loop estimator to compute the difference between the predicted and true disturbance, and then uses a low pass filter to update the prediction. The adaptation law is given by:
è 9 + ft//f//m m + d/m + A, v) A.dt
(4a)
d dnew = - (e A. -1) A, (0-
v)
(4b)
d - low pass filter(d, (d, ddew)
(4c)
where A, is a Hurwitz matrix, di is the discretization step length and Ó is the velocity prediction. Generally speaking, (4a) is a velocity predictor using the estimated disturbance d, and (4b) and (4c) update and filter d. Compared to other sim-to-real techniques such as domain randomization [30]
and student-teacher adaptation [24]. the adaptive-contee wwweeeeooooooo-bssseeeee disturbance adaptation method in DATT tends be more reactive and robust, thanks the closed-loop nature and provable stability and convergence of C1 adaptive control.
We note that DATT provides general framework for adaptive control. Other methods to estimate d, for example RMA, can easily be used instead, but we found them be less robust than L₁ adaptive control. We compare against RMA baseline in our experiments.
Experiments
4.1 Simulation and Training
Training is done in a custom quadrotor simulator that implements (I) using on-manifold integration, with body thrust and angular velocity as the inputs the system. In order convert the desired body thrust fs and body rate waes output from the controller to the actual thrust and body rate for the drone in simulation, we use first-order time delay model:
W. Wi-1 ke wi-1)
(5a)
fΣ,1 fE.-1 k(FE,dcs -E,t-1
(5b)
We & fixed value of 0.4, which we found worked well on the real drone. In practice the algorithm generalizes well to a large range of k when training fixed k. Our simulator effectively runs at 50 Hz, with dt 0.02 for each simulation step.
5

---

We train across a series of xy-planar smooth and infeasible reference trajectories. The smooth trajectories are randomized degree-five polynomials and series of degree-five polynomials chained together. The infeasible trajectories are we refer to as zigzag trajectories, which are trajectories that linearly connect a series of random waypoints, and have either zero or undefined acceleration. The average speed of the infeasible trajectories is approximately 2 m/s. /s. See Appendix for more details on the reference trajectories.
At the start of each episode, we apply a force perturbation with randomized direction and strength in the range of |-3.5 m/ss 3.5 m/s) representing translational disturbances. We then model time varying disturbance Brownian motion; each time step, update d <- d + e, with (0, Edt). We chose Σ 0.01I. This is meant to model potentially complex time and state- dependent disturbances during inference time, while having few modeling parameters we wish to demonstrate zero-shot generalization to complex target domains without prior knowledge. We run each episode for a total of 500 steps, corresponding to 10 seconds. By default, we set to 0.6s with 10 feedforward reference terms. In Appendix A, we show ablation results for various different horizons.
R
We also note that stable training and best performance require fixing an initial trajectory for the first 2.5M steps of training (see Appendix A for more details). Only after that initial time period do we begin randomizing the trajectory. We train the policy using PPO for a total of 20M steps. Training takes slightly over hours on an NVIDIA 3080 GPU.
4.2 Hardware Setup and the Low-level Attitude Rate Controller
We conduct hardware experiments with the Bitcraze Crazyflie 2. equipped with the longer 20 mm motors from the thrust upgrade bundle for more agility, The quadrotor tested weighs 40 g and has thrust-to- thrust-to-weight ratio of slightly under 2.
Position and velocity state estimation feedback provided by the OptiTrack motion capture system 50 Hz an offboard computer that runs the controller. The Crazyflie quadrotor provides orien- tation estimates via a 2.4 GHz radio and control commands are sent to the quadrotor over the same radio at 50 Hz. Communication with the drone is handled using the Crazyswarm API [34]. Body rate commands Waes received by the drone are converted to torque commands T using a custom low- level Pl attitude rate controller on the firmware: T = - (y Waes) - Ky f(w wess). Finally, this torque command and the desired total thrust fxides from the RL policy are converted to motor thrusts using the invertible actuation matrix.
4.3 Baselines
We compare our reinforcement learning approach against two nonlinear baselines: differential flatness-based feedback control and sampling-basee Model Predictive Control (MPC) [4]. We also compare using L1 adaptive control, which we propose, against RMA.
Nonlinear Tracking Controller and L1 Adaptive Control The differential flatness-based con- troller baseline consists of a PID position controller, which computes desired acceleration vector, and a tilt-prioritized nonlinear attitude controller, which computes the body thrust /x and desired body angular velocity Waes-
p²)) ) a -g g d m, ab KKP(p- -p") -KDUU-u 0²)) (')-Ki / 100-p')
(6a)
2th
-Re3, sr. aT,
(6b)
|an]
Waes =
K R 2tb x = + UMz thm = - Kym (vb θ vier)
(6c)
where d is the disturbance estimation. For the nonlinear baseline, we set d = 0, and for C1 adaptive control [11] we use (4) to compute d in real time [11]. For our experiments, we set Kp = diag(16 6 6)., Kr diag([1.5 1.5 1.5)), Ko - diag([4 4]), KR diag diag([120 120 0)). and Kyaw = 13.75. PID gains were empirically tuned the hardware platform track both smooth and infeasible trajectories while minimizing crashes.
Nonlinear MPC and Adaptive Nonlinear MPC We use Model Predictive Path Integral (MPPI)
[4] control our second nonlinear baseline. MPPI is sampling based nonlinear optimal control

---

external wind
X(ml
Crazyflie 2.1
-0.5
swing drag plate
-1.0
-1.5
0
2
6
8
10
Time (s)
Reference
With Adeotation
Without Adaptation
Figure 3: Left: Crazyflie 2.1 with a swinging cardboard drag plate in an unsteady wind field. Right: Comparison between our methods with and without adaptation with the drag plate on a zigzag trajectory. With wind added, adaptation needed, otherwise the drone crashes.
technique that computes the optimal control sequence w.r.t. known dynamics model and specified cost function. In our implementation, we use (1) (d 0) as the dynamics model with the body thrust /s and angular velocity w the control input. The cost function is the sum of the position error norms along A 40 horizon steps. We use 8192 samples, dt 0.02, and temperature of 0.05 for the softmax. For adaptive MPC, similar to prior works [18, 12], we augment the standard MPPI with the disturbance estimation d from L1 adaptive control, which we refer to as LPCM.
RMA We compare against RMA for adaptive control baseline. Instead of using Lı estimate d. we train adaptation neural network v that predicts d from history of state-action pairs using the RMA method (denoted DATT-RMA), similar to prior works [29]. We first train our policy æ in sim using PPO as usual. but conditioned on the ground truth d. To train v we then roll out π with predicted by a randomly initialized va for 500 timesteps. & is then trained with supervised learning in order to minimize the loss ||d - dll. We repeat this process for 10000 iterations, when the loss converges. Our adaptation network v takes as input the previous seen 50 state-action pairs, and the architecture consists of 3 1D convolutional layers with 64 channels and kernel size of for each, followed by fully connected layers of size 32 and ReLU activations.
4.4 Arbitrary Trajectory Tracking
We first evaluate the trajectory tracking performance of DATT compared to the baselines in the absence of disturbances. We test on both infeasible zigzag trajectories and smooth polynomial trajectories. Each controller is run 2 times on the same bank of 10 random zigzag trajectories and 10 random polynomials. Results are shown in Table For completeness, we also compare with the tracking performance of adaptive controllers in the absence of any disturbances. We also compare method to version without adaptation, meaning that enforce d = 0.
Arbitrary trajectory tracking without external disturbances
Method
Smooth trajectory
Infeasible trajectory
Inference time (ms)
Nonlinear tracking control
0.098 + 0.012
crash crash 0.183 0.027
0.21
Li adaptive control
0.091 + 0.009
0.93
MPC
0.104 0.009
12.62
LPMMPC
0.088 + 0.010
0.181 0.031
13.10
DATT (w/ å = 0)
0.054 0.013
0.089 0.026
2.41
DATT
0.049 0.017
0.083 0.023
3.17
Table 1: Tracking error (in m) of DATT vs. baselines, without any environmental disturbances (no
wind or plate). crash indicates crash for all ten trajectory seeds.
We see that DATT achieves the most accurate tracking, with a fraction of the compute cost of MPC. With our current gains, the nonlinear and L1 adaptive control baselines are unable to track the infeasible trajectory. With reduced controller gains, it is possible these controllers would not crash when tracking the infeasible trajectories, but doing so would greatly decrease their performance for smooth trajectories.

---

4.5 Adaptation Performance in Unknown Wind Fields with Drag Plate
To evaluate the ability of DATT to compensate for unknown disturbances, we test the Crazyflie in a high wind scenario with three fans and an attached soft cardboard plate hanging below the vehicle body. Figure 3 shows this experimental setup. We note that this setup differs significantly from simulation the placement of the fans and the soft cardboard plate creates highly dynamic and state dependent force disturbances, as well as torque disturbances, yet in simulation we model only the force disturbance as a simple random walk. However, our policy is able to generalize well zero-shot to this domain, as shown in Table 2.
Arbitrary trajectory tracking with external disturbances
Smooth traj. w/ plate
Smooth traj.
Infeasible traj. w/ plate
Infeasible traj. w/ plate & wind
Method
w/ plate & wind
L1 adaptive control
0,163 + 0.013
0.184 + 0.020
crash
crash 0.243 + 0.026
LI-MPC
0.121 + 0.010
0.181 + 0.04
0.216 + 0.028
DATT (w/ d = 0)
0.091 + 0.040
0.118 0.054
0.143 + 0.031
crash 0.193 + 0.075
DATT-RMA
0.091 + 0.049
0.115 + 0.071
0.164 + 0.051
DATT
0.063 + 0.052
0.095 + 0.053
0.122 + 0.041
0.161 + 0.056
Tracking error (in m) of DATT baselines, with attached plate and/or wind. Results are effectively for zero-shot generalization, as we do not model plate, torque disturbances, or exact force disturbances in simulation.
Table 2:
In Table 2, we see that the baseline nonlinear adaptive controller is unable to track infeasible trajec- tories, similar to the experiment without adaptation. Our method with adaptation enabled is able to track all the trajectories tested, with the lowest tracking error. We also verify that using & adaptive control results in better performance than using RMA. We note that this is due to large sim2real gap with the adaptation network for RMA, which we discuss in the Appendix. Figure 3 shows the difference in tracking performance between our method using adaptive control and our method without, on an example zigzag trajectory with a drag plate. We see that our approach of integrating L1 adaptive control with our policy controller is effective in correcting the error introduced by the presence of the turbulent wind field and plate. Our method performs better than LMPM without any knowledge of the target domain, and with a fraction of the compute cost. Figures 5 and 6 in the Appendix visualizes the tracking performance of DATT vs. -MPPCC on infeasible and smooth trajectory, respectively.
5 Limitations and Future Work
Our choice of hardware presents some inherent limitations. The relatively low thrust-to- ratio of the Crazyflie (less than 2) means that we are unable fly very agile or aggressive trajectories on the real drone perform complex maneuvers such drone flip mid-trajectory. For this reason, we focused on zy-planar trajectories in this paper, and did not vary the direction. However, our method provides the framework for performing accurate tracking for any trajectory, as we note we are able to perform much larger range of agile maneuvers in simulation, including flips.
Our simulator is only an approximation of the true dynamics. For example, model the lower- level angular velocity controller with a simplified first-order time delay model, which limits sim2real generalization for very agile tasks. Furthermore, our force disturbance model is highly simplified in sim, which only approximates the highly time- and state-dependent force and torque disturbances the drone can encounter in reality. However, we show that we can already achieve good zero-shot generalization to highly dynamic environment and challenging tasks.
We also note that our training process has fairly high variance and can be sensitive to the hyperpa- rameters of the PPO algorithm, typical of RL. As seen in Appendix A, we use few tricks for stable learning, including fixing the reference trajectory for the first 2.5M training steps. Future work is needed to understand the role of these architectural and training features and help inform the best algorithm design and training setup.
8

---

Acknowledgments
We would like to acknowledge the Robot Learning Lab at the University of Washington for pro- viding the resources for this paper. We would also like to thank the reviewers for their helpful and insightful comments.
References
[1] D. Mellinger and V. Kumar. Minimum snap trajectory generation and control for quadrotors. In 2011 IEEE International Conference on Robotics and Automation (ICRA), pages 2520-2525. IEEE, 2011. URL http:ttttp/eeee http:://ieeeeppore.e.e. ieee org/ htgg//llstaacccooocmmee////oo
[2] T. Lee, M. Leok, and N. H. McClamroch. Geometric tracking control of quadrotor uav on se
(3). In 49th IEEE conference on decision and control (CDC), pages 5420-5425. IEEE, 2010.
[3] M. Faessler, A. Franchi, and D. Scaramuzza. Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories. IEEE Robotics and Automation Letters, 3(2):620-626, Apr. 2018. ISSN 2377-3766, 2377-3774. doi:10.1 1009/ 02402. arXiv: 1712.02402.
LRA.2017.2776335 URL
[4] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Interna- lional Conference on Robotics and Automation (ICRA), pages 1714-1721, IEEE, 2017.
[5] Sun, A. Romero, Foehn, E. Kaufmann, and Scaramuzza. comparative study non- linear mpe and differeeniill-Rannss--bbee control for quadrotor agile flight. IEEE Transactions on Robotics, 38(6)3357--377, 2022. doi: 10. 109/TT0
[6] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter. Control Quadrotor with Reinforcement Learning. IEEE Robotics and Automation Letters, 2(4):2096-2103, Oct. 2017. ISSN 2377- 3766, 2377-3774. doi:10.1 109/LRA.2017.27205511 URL http::/ http://rxivv.oog/a////////// 05110.
[es].
[7] E. Kaufmann, A. Loquercio, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Deep Drone Acrobatics. In Robotics: Science and Systems XVI. Robotics: Science and Systems Foundation, July 2020. ISBN 978-0-9923747-6-1. http://nn...gggrssr//.../e otg/rss16/p040. pdf.
URL
[8] B. Kiumarsi, K. G. Vamvoudakis, H. Modares, and F. L. Lewis. Optimal and autonomous control using reinforcement learning: A survey. IEEE transactions on neural networks and learning systems, 29(6):2042-262, 2017.
[9] G. Shi, X. Shi, M. O'Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung. Neural Lander: Stable Drone Landing Control using Learned Dynamics. 2019 In- ternational Conference on Robotics and Automation (ICRA), pages 9784-9790, May 2019. doi: 100.109//CAA22219999953333 1811 1811.08027.
URL http://axxi..oog//////////// 08027. arXiv:
[10] M. O' O'Connel,, G. Shi, X. Shi, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66); cabm6597, 2022.
[11] Z. Wu. S. Cheng, P. Zhao, A. Gahlawat, K. A. Ackerman, A. Lakshmanan. C. Yang, J. Yu, and N. Hovakimyan. Li quad: L1 adaptive augmentation of geometric control for agile quadrotors with performance guarantees. arXiv preprint arXiv:2302.07208 2023.
[12] D. Hanover, P. Foehn, S. Sun, E. Kaufmann, and D. Scaramuzza. Performance, precision, and payloads: Adaptive nonlinear mpe for quadrotors. IEEE Robotics and Automation Letters. (2):690-697, 2022. doi: 10. 10011999

---

[13] A. Spitzer and N. Michael. Inverting Learned Dynamics Models for Aggressive Multirotor Control. In Robotics: Science and Systems XV. Robotics: Science and Systems Foundation, June 2019. ISBN 978-0-9923747--5-44 doi: 10560...55/7
2t1.///S....1........ URL http://www.
roboticsproceddings http:///r...rrrrra/.p.......... pdf. arXiv: 1905.13441
[14] B. Morrell, M. Rigter, G. Merewether, R. Reid, R. Thakker, T. Tzanetos, V. Rajur, and
G. Chamitoff. Differential Flatness Transformations for Aggressive Quadrotor Flight. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5204-5210, Bris- bane, QLD, May 2018. IEEE, ISBN 978-1-5386-308155. doi: 109///1A.2008..... 2.1886688...1...46888.. seee. URL https:://eeepppppl wtt.:p/orr........re//eeeeeeeeee
[15] E. F. Camacho and C. B. Alba. Model predictive control. Springer science & business media, 2013.
[16] C. Yu, G. Shi, S.-J. Chung, Y. Yue, and A. Wierman. The power of predictions in online control. Advances in Neural Information Processing Systems, 33:1994-200 2004, 2020.
[17] G. Williams, Drews, B. Goldfain, M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1440-1440. IEEE, 2016.
[18] Pravitra, A. Ackerman, Cao, N. Hovakimyan, and A. Theodorou. Li-adaptive mppi architecture for robust and agile control of multirotors. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7661-7666, 2020, doi:10.1109/ 100245573.200.994111
[19] K. Lee, J. Gibson, and E. A. Theodorou. Aggressive perception-aware navigation using deep optical flow dynamics and pixelmpe. IEEE Robotics and Automation Letters, 1(2):1207-1214, 2020. doi: 10. (099000A.2220.9999999
[20] Y. Zhang, W. Wang, P. Huang, and Z. Jiang. Monocular vision- sense and avoid of uav using nonlinear model predictive control. Robotica, 37(9):1582-159)) 2019. doi:10.1017 5026357471900015
[21] B. Michini and J. How. LI Adaptive Control for Indoor Autonomous Vehicles: Design Pro- cess and Flight Testing. In Proceeding of AIAA Guidance, Navigation, and Control Con- ference, pages 5754-5768, 2009. URL https://arc. aiaa:://aaaaaaaaaaaaaaaaaaaaaaa. aiaa. org/doi/pdf///oooooooooogggggg 2514//6. 2009-5754.
[22] C. D. McKinnon and A. P. Schoellig. Unscented external force and torque estimation for quadrotors. In 20/6 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 565 5651-5657, Daejeon, South Korea, Oct. 2016. IEEE. ISBN 978-1-5090-3762- ieee. wrgeeeeo///doomoott.o
9. 100.0..0000//0..00..... 7759831/.
URL
[23] E. Tal and S. Karaman. Accurate Tracking of Aggressive Quadrotor Trajectories using Incre- mental Nonlinear Dynamic Inversion and Differential Flatness. In 20/8 IEEE Conference on Decision and Control (CDC), pages 4282-4288, Miami Beach, FL, Dec. 2018. IEEE, ISBN 978-1 1-1-1-6-19195555 2tt.00.110//CDC.2118880///
URL https://arxiv. org/
abs/1809. 04048.
ISSN: 0743-1546.
[24] D. Zhang, A. Loquercio, X. Wu, A. Kumar, Malik, and M. W. Mueller. Learning single near-hover position controller for vastly different quadcopters. arXiv preprint arXiv:2209.0 09232, 2022.
[25] C. K. Verginis, Z. Xu, and U. Topcu. Non-parametric neuro-adaptive coordination of multi- agent systems. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, page 1747-1749, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 450392113.
10

---

[26] G. Torrente, E. Kaufmann, P. Foehn, and D. Scaramuzza. Data-Driven MPC for Quadrotors. JEEE Robotics and Automation Letters, 2021. ISSN 2377-3766, 2377-3774. doi:10.1109) arXiv: 2102.05773.
LRA.2021.3061307 URL
[27] A. Spitzer and N. Michael. Feedback Linearization for Quadrotors with Learned Accel- eration Error Model. In 2021 IEEE International Conference on Robotics and Automa- lion (ICRA), pages 6042-6048, May
000 URL
https://ieseexplore ieee. org/document/95550
ISSN: 2577-087X.
[28] G. Shi, W. Hönig, X. Shi, Y. Yuc, and S.-J. Chung. Neural-swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. IEEE Transactions on Robotics, 38())1063-1079, 2021.
[29] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots, July 2021. URL htttp://arree org/ ab1/200/ 04034. arXivv2107.04034 [es].
[30] A. Molchanov, Chen, W. Hönig, A. Preiss, N. Ayanian, and G. S. Sukhatme. Sim- to- to-(Multi)-Real Transfer of Low-Level Robust Control Policies Multiple Quadrotors. 1rXiv:1903.04622 [cs], Apr. 2019. URL http://arxiv.org/ aaaaaaas//110 04628. arXiv: 1903.04628.
[31] E. Kaufmann, Bauersfeld, and D. Scaramuzza. A Benchmark Comparison Learned Con- trol Policies for Agile Quadrotor Flight, Feb. 2022. URL http://arxx..rg//abbttt 10796. arXiv:2202.10796 [cs].
[32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, 0bbs/1707.063477 2017. URL
06347.
[33] N. Hovakimyan and C. Cao. CI Adaptive Control Theory: Guaranteed Robustness with Fast Adaptation. Society for Industrial and Applied Mathematics, 2010.
[34] Preiss, W. Honig, G. S. Sukhatme, and N. Ayanian. Crazyswarm: A large nano-
quadcopter swarm. In 2017 IEEE International Conference Robotics and Automation (ICRA), pages
2017. N0::10.119//CRRA..17779999
[35] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines:: Reliable reinforcement learning implementations, Journal of Machine Learning Research, 22 (268):1-8, 2021. URL http://jmlr. org/ http:://mml...ooppppppss//////// wttpp/pppr//////2000/p22 1364. html.

---

A
Ablations
Ablation
Tracking error (sim) (m)
No body frame
failed
No fixed intial reference
0.437 = 0.08
No feedback term
0.077 0.011
Feedforward horizon 1 (H 0.02s)
failed 0.240 + 0.008
Feedforward horizon (H = 0.3s)
Feedforward horizon 10 (H 0.6s) (used in main experiments)
0.055 = 0.007
Feedforward horizon 15 (H = 0.9s)
0.073 - 0.010 0.101 + 0.018
Feedforward horizon 20 (H 1.2s)
Base policy (no ablation)
0.046
Table 3: Tracking error (in m), in simulation, of various ablations after 15M training steps. Failed indicates the drone diverges from the reference trajectory. Tracking error is with respect to infeasible zigzag trajectories. The ablations done without adaptation, and with disturbances in the environment. 5 runs were attempted for each ablation.
We test various ablations of our primary method, with results shown in Table 3. In particular, test
No body frame:
With our training setup, we found that transforming all state inputs (except for the orientation) into the body frame was necessary for accurate trajectory tracking. This ablation tests method, but with the position p. velocity and reference positions in the world frame instead of the body frame.
No fixed initial reference This ablation removes the initial 2.5M training steps where we do not randomize the reference trajectory. We see that PPO converges to a much worse tracking performance. We note that the choice of the initial fixed reference does have much impact on the variance of training, only the existence of the fixed reference.
No feedback term We remove the feedback term R (p: pt) from our controller in- puts. This term might appear redundant with the reference trajectory, but we find explicitly conditioning on the feedback eπror consistently results in slightly more accurate tracking.
Feedforward horizon We test varying sizes of our feedforward horizon. In Table 3, Feed- forward horizon N refers to passing in future reference positions. described Sec- tion 3.2, we linearly space the N reference positions across time from to + H.
Base policy For comparison, we list the tracking error in sim of the main policy that use in our experiments section.
Adaptive Control in Simulation As seen in Table in simulation, using RMA the adaptive control strategy actually yields slightly better performance than L1 adaptive control. However, the real drone, as we report in Table 2, RMA performs significantly worse than Li adaptive control, indicating a significant sim2real gap. This is likely because the adaptation network in DATT-RMA highly susceptible to the domain shift in state-action pair inputs on the real drone, while the closed-loop nature LI guarantees fast disturbance estimation for any state-action pairs.
Method
Tracking error (sim) (m)
DATT (w/ disturbances)
0.062 + 0.011
DATT-RMA (w/ disturbances)
0.055 + 0.009
Table 4: Tracking error (in m), in simulation, of standard DATT (using L1 adaptive control) and DATT-RMA with random force disturbances
12

---

B Training Details and Network Architecture
Training done with the PPO implementation in the Stable Baselines3 library [35]. All PPO pa- rameters are left default.
The feedforward encoder architecture consists of 1-D convolution layers with ReLU activations that project the reference positions into 32-dim representation for input the main policy. Each 1-D convolution has 16 filters with kernel size of 3. The main policy network is 3-layer MLP with 64 neurons per layer and ReLU activations, and the value network shares this structure.
C
Reference Trajectory Details
M
0.2
0.5
x ((m)
0.0
0.0
-0.
-0.5
1.0
0.4
0.5
(m)
0.2
0.0
0.0
-0.5
-0.2
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Time (s)
Time (s)
Example of a random zigzag trajectory (infeasible). Right: Example of a random chained polynomial trajectory (smooth).
Figure 4: Left:
C.1 Smooth Trajectory
For smooth trajectories, we include mix of degree 5 polynomials and chained polynomials. Poly- nomials start at x 0 and y = O, and return to the origin after 10s, corresponding to our episode length. They are randomly generated by randomly selecting initial and end conditions. Chained polynomials are a series of random polynomials. We generate these trajectories by randomly se- lecting "nodes" at x = and y 0 at random times between s and 10s, and fitting degree polynomials between each node, ensuring that first, second, and third order derivatives are continu- ous at each node. Note that these trajectories are not guaranteed to be feasible, although in practice they easy to track as they are highly smooth.
C.2 Infeasible Trajectory
We use a class of what we refer to zigzag trajectories. We generate these trajectories by randomly selecting time intervals between 0.5 and 1.5 seconds, randomly generating waypoints after each time interval, and linearly connecting each waypoint. The waypoints can vary from -1 m to 1 m in both the x and y directions. By training on these zigzags, we are able to generalize well to wide variety trajectories, including polygons and stars as seen in Figure which are similar random zigzags.
C.3 Additional Figures of Results
We show additional figures from our results from Table 2. Figure 7 shows the values of the predicted d over time environment with wind versus one without wind. Figure 5 and Figure 6 show our tracking performance against LPCMPC for smooth and infeasible trajectory, respectively.
13

---

0.0
(m)
-0.55
(m)
-0.5
-1.0
0.8
(m)
0.7
0.6
10
14
Time (s)
Reference
DATT
L1-MPC
Figure 5: Performance of DATT against MPCM on a smooth trajectory with both wind and a plate attached.
(m)
-1
-0.5
[m)
-1.0
-1.5
0.8
(m))
0.6
8
10
12
14
Time (s)
Reference
DATT
L1-MPC
Figure 6: Performance of DATT against LP-MPC on an infeasible trajectory with both wind and plate attached.
14

---

10
(m/ 5
5.0
7.5
10.0
12.5
15.0
2.5
Ym//s 22
Any
-2.5
5.0
7.5
10.0
12.5
15.0
time (s)
without wind
with wind
Figure 7: Predicted d terms on two infeasible trajectories, one with wind, one without wind but with an air drag plate.
15


Reviews:
[{"review_id": "o5pMR8t5peE", "reviewer": "Reviewer_Bvch", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper is about precision trajectory tracking for quadrotors. It presents a learning-based approach that can track challenging trajectories in the presence of large disturbances. After RL training in simulation, the approach transfers to the real world. The experimental results are impressive. ", "summary_of_recommendation": "This a very nice paper and a pleasure to read. The algorithm is well explained, the results are convincing, and the demonstration on hardware is excellent. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I have no questions for rebuttal and recommend the paper be published as submitted.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work is really nice and clearly presented. No apparent weaknesses.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "HnWURbi4A-s", "reviewer": "Reviewer_Tsi5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an algorithm for controlling a quadrotor to track arbitrary trajectories. Deep RL is used along with disturbance adaptation in real-time.  With respect to previous works, the algorithm can track infeasible trajectories (in the sense of non-differentiable ones) with better accuracy. Extensive simulation and experimental results are presented. ", "summary_of_recommendation": "I believe there is theoretical novelty in the paper and good experimental demonstrations. However, the paper needs to be revised, mostly on its presentation in the sense of clarification of the algorithm.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The recent work 'Non-parametric neuro-adaptive control' should be referenced and a comparison (in the related work section) should be made.  \n- The disturbance d is time-varying and as far as I understand the training uses a constant one. Have you checked training with a time-varying one? Do you expect it would yield better results?\n- It is not clear what the embedding h represents. What does 'encodes the desired reference trajectory' mean? It is stated that \n'Thus, we input just the desired positions into a feed-forward encoder φ, which learns the feedforward embedding that contains the information of the desired future reference positions' but in (2a), the inputs are the errors. Is it a feedback-control term learned with a neural network? \n- Where does (4) come from? I understand the authors refer to an arxiv paper, but it would be very beneficial to briefly explain what the intuition of (4) and how it is derived.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Combining off-line learning and online adaptation (in a feedforward-feedback sense) seems a very promising technique for the control of dynamical systems.   \nThe concept is not entirely new, a recent work introduced a very similar concept with feedforward-feedback adaptive control: \nVerginis et al, 'Non-parametric neuro-adaptive control', ECC 2023, https://ieeexplore.ieee.org/document/10178288\nand an association/comparison with it should be made. \n\nThe paper is generally well-written, and the algorithm is verified extensively in simulation and experiments. \nI believe it mostly needs clarification in Section 3 regarding the proposed methodology, which is not completely clear.  \nMore specific comments follow below. \n ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8zHMADUAIuw", "reviewer": "Reviewer_cDww", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a learning-based controller for quadrotors that is robust to external disturbances such as unsteady wind fields. The learned controller is trained via model-free reinforcement learning in simulation, where it is exposed to random force disturbances while being tasked to track random trajectories. After training the controller for 2e7 environment steps, the controller is deployed zero-shot on a physical quadrotor. \n\nThe presented method is compared against a differential flatness-based controller and a sampling-based model predictive controller on the task of trajectory tracking with and without external wind disturbances. The proposed learning-based controller outperforms both baselines.", "summary_of_recommendation": "My recommendation is based on the good presentation of the paper, the convincing set of baselines that the presented approach is evaluated against, and the reported performance of the method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In the simulation, the proposed method is conditioned on ground-truth translational disturbance, while in the real world, the translational disturbance is estimated by an L1 adaptive controller. Is there a reason to not run the same L1 adaptive controller in simulation as well? Would doing so potentially reduce the sim2real gap?\n\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n=========\n- The paper is clearly written, reads well, and is easy to follow.\n- The experiments are performed on a physical robot.\n- The paper is very clear and transparent about the limitations of the presented work. \n\n\nConstructive Criticism\n==================\n- L16: UAV = *unmanned* aerial vehicle, not uninhabited aerial vehicle\n- In the simulation, the proposed method is conditioned on ground-truth translational disturbance, while in the real world, the translational disturbance is estimated by an L1 adaptive controller. Is there a reason to not run the same L1 adaptive controller in simulation as well? Would doing so potentially reduce the sim2real gap?\n- The proposed controller is able to reject external disturbances based on the information contained in the observed positional tracking error and the estimated force disturbance generated from the L1 adaptive controller. Previous work (ref [28] in the paper) proposed to use a history of state-action pair observations instead to achieve a similar effect. Would the proposed controller potentially benefit from such a history-based observation as well?\n- Minor in Figure 1: always add a grid to every plot, it improves the reading experience.\n- Minor: how does the performance of MPPI change with varying sample numbers?\n- Minor thought to think about, does not require to be addressed: how much sense does it make to evaluate tracking performance on infeasible trajectories? Isn't such a metric ill-posed in the first place? Evaluating tracking performance on infeasible trajectories becomes extremely sensitive to controller tuning.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "50ZWxrb8ok", "reviewer": "Reviewer_fQDU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a control architecture for a quadrotor to track arbitrary (possibly non-feasible) trajectories subject to uncertainties using reinforcement learning. Two components are learned: one is a feedforward embedding h, which takes the inputs of the position error to future planned positions on the trajectory and projects it to a latent space; the other one is the feedback policy that takes in the state information, latent-space represented terms from the embedding h, and disturbance and outputs the control actions. Both the two components are trained via reinforcement learning in simulations. The online deployment uses the uncertainty estimation obtained via the L1 adaptive control. Experimental validations are provided to demonstrate the performance of the proposed architecture.", "summary_of_recommendation": "The recommendation is made based on the comments listed above. Please provide extra experimental results and supply more details to the questions above.\n\nupdated on Aug. 18: the reviewer's questions have been addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. It's not clear why the position and velocity have to be converted to the body frame as inputs to the policy /pi. Why cannot those in the inertial frame work here?\n\n2. It seems a limitation where the training is conducted with constant force perturbation. Why not use a time-varying disturbance? Wouldn't the policy perform better when you train it with more realistic data?\n\n3. What about generalizing to different quadrotor platforms? The experiments are only demonstrated on a crazyflie quadrotor. \n\n4. In Section 5, the statement \"although our method also improves the performance in the z direction (Fig. 1)\" seems to contradict the results shown in Figs. 5 and 6, where the tracking on the z-axis is not as good as the L1-MPC. Please either fix the statement or provide explanations.\n\nupdated on Aug. 18: my concerns have been addressed by the authors. I have upgraded my score to strongly accept.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Below are my reviews of this paper\n\n0. The paper is written well. Many questions while reading the paper are answered in the supplementary material. The experimental validations look good. \n\n1. This is an interesting paper that tries to build the latent space based on the position error to the desired future positions, which should carry useful information on whether the desired positions to follow next will be feasible. The idea of training with simulated (known) disturbance and deploying an online estimator of the disturbance for the actual system is not new: the Rapid Motor Adaptation (RMA) mechanism (reference [26]) has proposed a similar idea, and it worked well for quadrupedal and quadrotors. The authors should know that RMA has been applied to quadrotors [R1] and must compare the difference in methodology and experimental verifications.\n\n2. The authors only compare the proposed method's performance with conventional non-learning controllers. However, given that neural networks (NNs) contain way more parameters than conventionally designed controllers, such a comparison can be an apple-to-orange comparison, which for sure the NN-parameterized controller will beat the conventional ones given the former's larger capabilities with more parameters. This is another reason why a comparison to the RMA [R1] should be conducted.\n\n3. When referring to the L1 adaptive control literature, the authors may want to cite the guaranteed bounded disturbance error using the state predictor with the piecewise constant adaptation law, e.g., proposition 4 in [11] and lemma 3 in [R2].\n\n4. In the experiments, it's fundamental to add unknown weights as a benchmark comparison (unknown mass induces unknown forces when the quadrotor tries to generate forces in the horizontal directions). Please consider adding this comparison.\n\n5. In Figs. 5 and 6 in the appendix, the authors didn't show the tracking performance of the baseline L1 adaptive control method (only L1-MPC was shown), which was mentioned in Table 2. Please add the corresponding results.\n\nReference\n[R1] Zhang, Dingqi, Antonio Loquercio, Xiangyu Wu, Ashish Kumar, Jitendra Malik, and Mark W. Mueller. \"Learning a Single Near-hover Position Controller for Vastly Different Quadcopters.\" In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1263-1269. IEEE, 2023. \n[R2] Zhao, Pan, Yanbing Mao, Chuyuan Tao, Naira Hovakimyan, and Xiaofeng Wang. \"Adaptive robust quadratic programs using control Lyapunov and barrier functions.\" In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3353-3358. IEEE, 2020.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary of Paper
The paper combines ideas from control and RL to come up with quadrotor controllers that are robust to external disturbances. The controllers are trained in simulation and extensively evaluated on a real quadrotor.

## Summary of Reviews
The reviewers found the paper to be well written and interesting. They had a few questions about the details of the method and requested stronger baselines.

## Influence of Rebuttal
With the extensive new results and the clarifications all concerns of the reviewers have been addressed.

## ToDos for Final Version
Please incorporate all the material from the rebuttal in the final version and also see comment by Reviewer fQDU on \sigma.
Decision: Accept (Oral)