Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight

Authors: jiaxu  xing
angel  romero
leonard  bauersfeld
davide  scaramuzza
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:39:32

Abstract: Learning visuomotor policies for agile quadrotor flight presents significant difficulties, primarily from inefficient policy exploration caused by high-dimensional visual inputs and the need for precise and low-latency control.
To address these challenges, we propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) in the task of vision-based autonomous drone racing.
While RL provides a framework for learning high-performance controllers through trial and error, it faces challenges with sample efficiency and computational demands due to the high dimensionality of visual inputs.
Conversely, IL efficiently learns from visual expert demonstrations, but it remains limited by the expert's performance and state distribution.
To overcome these limitations, our policy learning framework integrates the strengths of both approaches.
Our framework contains three phases: training a teacher policy using RL with privileged state information, distilling it into a student policy via IL, and adaptive fine-tuning via RL.
Testing in both simulated and real-world scenarios shows our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance, successfully navigating a quadrotor through a race course using only visual information.

Main paper: 
 introduction: 
 Visuomotor policy learning enables robots to perform complex tasks by directly mapping visual information into action. This technique has been successfully demonstrated in various robotic systems to learn complex behaviors such as visual navigation [1, 2] , dexterous manipulation [3, 4, 5] , and agile maneuvers [6, 7, 8] . The ability to learn directly from visual data allows machines to interpret visual observations and translate them directly into corresponding motor actions, akin to the human skill of hand-eye coordination. However, learning from only visual inputs introduces a range of distinct challenges. The intrinsic high dimensionality of visual input makes the policy exploration and learning process more inefficient than using low-dimensional input, such as robot states.In agile quadrotor flight, these challenges are more pronounced due to the platform's agility, inherent instability, and reliance on low-level commands like collective thrust and body rates, which necessitate precise low-latency closed-loop control. Previous works [8, 1] have demonstrated the capability of piloting quadrotors at high speeds using visual inputs for acrobatics and obstacle avoidance. These systems always rely on high-frequency proprioceptive data from an Inertial Measurement Unit (IMU) for accurate state estimation. However, despite these advancements, the goal of training and deploying an autonomous drone directly using RGB images has yet to be achieved. This limitation is particularly relevant in scenarios such as first-person-view (FPV) drone racing, where pilots achieve high-performance flight near the physical limits of the platform using only visual information from 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.Figure 1 : Long exposure image of real-world flights shows a blue trajectory for our approach and a red one for the imitation policy. Training on the same number of samples, our approach yields a tighter trajectory, resulting in faster lap times and demonstrating superior performance and robustness. a 60 Hz monocular camera. Addressing this gap, our work specifically focuses on autonomous quadrotor racing, employing purely visual inputs without relying on metric state estimation or IMU data. Two primary learning methods have emerged in tackling this challenge: Reinforcement Learning (RL) and Imitation Learning (IL). RL is gaining traction as a general framework for designing complex controllers that are difficult to handcraft using classical methods [9, 7, 10] . This method involves learning through millions of trial-and-error interactions within a simulated environment. Despite the potential of RL, it often requires collecting extensive data samples to learn effectively, which can be computationally demanding. As a result, enhancing the sample efficiency of RL algorithms has become a critical focus. Thus, the ability to explore and learn efficiently in vision-based RL environments from scratch is essential, presenting a key challenge that our research seeks to overcome.Conversely, IL has been successfully demonstrated on different mobile platforms to perform end-toend vision-based learning [8, 1, 11, 5] . IL synthesizes policies using either expert demonstrations or a privileged policy in a supervised fashion. The optimization objective is to clone an expert rather than to interact with the environment and collect rewards as in RL. Due to this simplification, IL typically requires fewer samples and, consequently, has been validated for real-world robot learning problems [1, 8, 5, 11, 12, 13] . However, IL faces several challenges, including the significant issue of covariate shift. This term describes the discrepancies between training data (demonstrations for IL) and real-world scenarios, which can lead to degraded performance as the learned policy may not generalize well to new situations. Additionally, the effectiveness of an IL-derived policy is inherently limited to the quality of the expert demonstrations it is based on; it cannot surpass the performance of the demonstrations used for training.The RL policy for state-of-the-art autonomous drone racing [7] , which outperformed worldchampion pilots, still relies on explicit state estimates, including position, velocity, and orientation. Our work, however, focuses on learning visuomotor policies that map visual information directly to control commands without explicit state estimation. Achieving this would bring autonomous flying machines closer to how human pilots navigate. However, this ambition was unattained in the realm of drone racing due to one fundamental challenge: sample inefficiency.Contributions By leveraging the complementary advantages of IL and RL, we propose a framework that trains a policy capable of navigating through a sequence of gates using solely gate corners or RGB images. Through experiments in both simulation and real-world environments, we demonstrate that our approach, given the same sample budget, outperforms existing IL methods in robustness and performance and succeeds where RL from scratch fails. While most state-of-the-art visuomotor policy learning for mobile robotics adopts the teacher-student IL framework [8, 1, 6, 14] , we extend this framework by bootstrapping the IL policy for adaptive RL fine-tuning to enhance the performance. Although we validate our method using vision-based drone racing, our approach does not rely on task-specific adaptations that might limit its applicability to other robotic platforms or tasks. 
related work: 
 Vision-Based Robot Learning Deep visuomotor policies directly map actions from visual inputs, such as RGB images [4] or depth images [1, 15] . In contrast to conventional methods, end-toend approaches often operate without the need of environmental mapping [16, 17] , precise state estimation [18, 19] , or motion planning [18, 20] . Existing works have demonstrated end-to-end vision-based policies primarily through either RL or IL [8, 21, 1, 22, 23] . In RL, the optimization objective is shaped by task-specific reward designs. The policy gains robustness and generalizability by learning from both positive and negative samples through interactions with training environments [24] . Despite having a task-centric objective, vision-based RL has predominantly found application in simulations, such as fixed-based manipulation tasks [25, 21] , or video games [26] , due to its sample inefficiency. Therefore, efficient approaches for vision-based RL in mobile robots are still actively being explored. In contrast, in most IL approaches, the optimization objective is a difference measure in behavior between the learned policy and the expert demonstrations [27, 28] . The simplification of learning the task solely from expert demonstrations dramatically reduces the required samples for effective training. Thanks to its sample efficiency, various visuomotor policy learning methods have been demonstrated on mobile robots [8, 1, 6, 11, 5] . Due to the supervised fashion of policy training, the performance of IL policies is capped by the teacher policies and suffers degradation for out-of-distribution observations [28] , particularly in task settings where actions are only partially observable from visual inputs.RL Finetuning from Expert Demonstrations For various robot learning applications, one central question is how expert demonstrations can be used to develop high-performance policies. The predominant method in existing research begins with behavior cloning (BC) and refines the policy through RL to enhance state exploration and sample efficiency, as seen in tasks like pole balancing and dexterous manipulation [29, 30, 31, 32, 33] . A major challenge in online fine-tuning with RL is the "stability-plasticity dilemma", where networks catastrophically forget old behaviors while continuing learning [34] . This problem is especially acute for RL settings, as exploration generates out-of-distribution (OOD) data, leading to suboptimal learning or even completely unlearning the previous skills. Using our approach, which adaptively updates the policy based on performance, we demonstrated that catastrophic forgetting is greatly reduced. 
methodology: 
 The drone racing task can be formulated as an optimization problem where the objective is to minimize the time required to navigate through a predefined sequence of gates [35] , as illustrated in Fig. 4 . The drone perceives the environment solely through a single RGB camera, and the learned policy network utilizes egocentric vision input o p to output Collective Thrust and Bodyrates control L P G (θ) CNN CNN TCN MLP • • • • • • Command ateacher astudent LIL = ∥ateacher -astudent∥ 2 It It-N+1 State Input It It-N+1 • • • • • • CNN CNN TCN Command L P G (θ)Figure 2 : We demonstrate visuomotor policy learning in three different stages. In stage I, we train a state-based teacher policy using RL. In stage II, we use IL to learn a student distillation policy using visual inputs. In stage III, we bootstrap the actor using the student policy to fine-tune the policy through vision-based RL.command [c, ω x , ω y , ω z ], where c represents collective thrust, and ω denotes the body rates [36] .As shown in Fig. 2 , our approach consists of three phases: (I) initial training of a teacher policy using state information, (II) distillation into a student policy via IL to transfer knowledge and create an efficient baseline model, and (III) fine-tuning the student policy through a novel performanceconstrained adaptive RL approach to enhance the policy's performance and robustness.The teacher policy π teacher processes state observations s = p, R, v, ω, i, d , where p ∈ R 3 denotes the drone's position, R ∈ R 6 is a vector comprising the first two columns of R WB [37] , v ∈ R 3 and ω ∈ R 3 denote the linear and angular velocity of the drone, and d ∈ R 3 represents the position of the next gate center relative to the current drone position. The training of the teacher policy employs a model-free reinforcement learning approach using Proximal Policy Gradient (PPO) [24] . The RL policy training rewards are adjusted based on [38] . The reward at time t, denoted as r t , is defined as the sum of various componentsEQUATIONwhere r prog t encourages progress towards the next gate to be passed [39] , r perc t encodes perception awareness by adjusting the quadrotor's attitude such that the optical axis of its camera points towards the next gate's center, r act t penalizes action changes from the last time step, r br t penalizes bodyrates and consequently reduces motion blur, r pass t is a binary reward that is active when the robot successfully passes the next gate, r crash t is a binary penalty that is only active when a collision happens, which also ends the episode. For the detailed reward formulation, we refer the readers to the Appendix.Phase II: Imitation Learning using Visual Input The goal of this phase is to distill a student policy π student from the expert π teacher . The vision-based student policy takes a sequence (history length H timesteps) of perceptual observations [o t-H+1 , . . . , o t ] as input. We use a Temporal Convolutional Network (TCN) [40] to encode the series of vision embeddings from I or corners C. The output features from the two TCNs are concatenated and fed into a two-layer MLP, which outputs the actions. The supervision loss is formulated as L A (action error), which is the mean square error between the outputs of the teacher policy and the student policy To acquire an imitation learning policy, the most common methods are (i) Behavior Cloning (BC) or (ii) DAgger [27] . In the case of BC, the state-based teacher policy is executed for a fixed number of steps, generating a dataset that encompasses corresponding perceptual observations and action outputs. In contrast, DAgger involves iteratively training the student policy by executing the learned student policy with a gradually increasing probability over time. The on-policy training fashion of DAgger results in a broader distribution of demonstrations from the student policy. This approach results in improved performance compared to traditional BC, contributing to more effective learning.EQUATIONPhase III: Performance-Adaptive Online Fine-Tuning State-of-the-art visuomotor policy learning for mobile robotics typically employs a teacher-student IL framework; however, integrating online RL fine-tuning could further enhance policy performance and robustness. The major challenge of online policy fine-tuning is catastrophic forgetting during exploration, where the state distribution shift during exploration causes the pre-trained policy to generalize poorly. To address this, we propose an algorithm that conditions exploration and network updates on the policy's performance, as shown in Algorithm 1. In our algorithm, we initiate the fine-tuning process using the pre-trained student policy as the starting actor in RL. However, a straightforward plug-and-play approach may not yield optimal results due to two key challenges: (i) the critic function requires interactions to adapt the pre-trained actor, necessitating a "warm-up" process. (ii) During the initial training phase, policy update steps should be kept relatively adaptive based on the learning progress to prevent catastrophic forgetting resulting from sudden, large updates. In our approach, if the policy improvement is small, the update rate of the critic is automatically increased. Once the policy achieves high-reward action sequences, the policy update rate also increases. By linking the learning rates and the clip range to the policy rollout performance, we eliminate the need for heuristic tuning of learning rates to mitigate catastrophic forgetting. We believe this approach is easily generalizable to other platforms as it does not require task-specific information.In our approach, the exploration and learning rates should dynamically depend on the agents' performance rather than being solely determined by the number of iterations.Algorithm 1 Adaptive Fine-tuning and Proximal Policy Gradient Update LRπ ← min (LRπ + max(α -1, 0) • cV , LRπmax) 13: LRV ← max (LRV -max(α -1, 0) • cπ, LRVmin) 14: ϵ ← min (ϵ + max(α -1, 0) • cϵ, ϵmax) 15:end if 16:Update policy by maximizing PPO objective 17:Lπ(ϕ) = E min π ϕ (a|s)π old (a|s) Â, clip π ϕ (a|s)π old (a|s) , 1ϵ, 1 + ϵ Â 18:ϕ ← ϕ + LRπ∇ ϕ Lπ(ϕ) 19: end for 20: Output: Policy parameters Asymmetric Critic For the third phase vision-based RL, we employ an asymmetric actor-critic setup, as illustrated in Fig. 3 (b) . This approach involves augmenting the critic function inputs with privileged information, such as the robot state s. Incorporating this privileged state information enables more precise learning of the critic function, thereby enhancing the efficiency and performance of actor training in partially observable settings [21] . 
training setup: 
 We validate our approach on three different race tracks, as shown in Fig. 4 , namely the "SplitS", "Figure 8" , and "Kidney" tracks. During evaluation, each policy is rolled out until it completes 10 laps or for 3000 simulation time steps (50 seconds). We evaluate each policy 100 times, starting from positions uniformly sampled within a 1m cubic box centered on the nominal starting position of the racing track. Each method is repeated 5 times with different random seeds. The evaluation uses a realistic simulation based on the BEM model for aerodynamic effects [41] .We use three evaluation metrics: success rate (SR), mean-gate-passing-error (MGE), and lap time (LT). SR is the ratio of completed laps to total trials. MGE measures the distance between the drone's position and the gate center when passing through, here the inner gate size used for experiments is 1.5 m. LT indicates the duration to complete a full race track, flying through all gates. Further details on training configurations and our hardware setup are available in the Appendix. Policy Input To evaluate the performance of our approach, we conduct IL experiments with two categories of visual input: (i) implicitly learned from ResNet [42] . Here we employ a pre-trained ResNet50 [42] network on ImageNet [43] to produce embeddings of length 128. Hence, the input of the TCN network corresponds to the history of the embeddings, which is H × 128. (ii) racing gate corners projected in the camera pixel frame. Here we utilize the normalized pixel coordinates of all gate corners projected in the camera view. We model gate corners based on the gate detection system presented in [7] . To simulate real-world scenarios, we include domain randomization such as gate scales, pixel position noise (10 pixels in both (u, v) in a 1280 × 760 image frame), and a 10% random missing probability for each corner at each frame to mimic the detection failures of the gate detector. After projection, we sort the gate corner pixels by their u coordinates, and we model corners out of the image view as (-1, -1). Detailed simulation visualizations for both inputs are available in the Appendix.Performance comparison to baseline approaches One inherent limitation of the student-teacher IL framework is to infer reasonable actions from partial information. This limitation arises because the student policy is trained only on the explicit actions of the expert, without understanding the underlying context that the expert may infer from unobservable cues. As a result, the student policy can struggle in scenarios where critical information is missing, leading to suboptimal actions and reduced overall performance. An example of this is the SplitS Track, where frames often lack visible corners (detailed in Appendix). Hence, to benchmark the learned policies' performance, we conduct a detailed analysis of our approach to the existing baselines using various time horizons for the policy H ∈ [2, 4, 8, 16, 32, 64] .Table 1 : Policy performance evaluation averaged by 6 different history lengths using both implicitly learned representations, specifically a common ResNet50 [42] for RGB images, and the task-specific gate corners positions across three different racing tracks. Ours is the only approach that is able to learn to perform the task. Right: Using a fixed sample budget, we varied the IL policy percentage for our approach on the vision-based policy for SplitS track, achieving a local maximum in deployment reward with 60% IL pretraining and 40% RL fine-tuning.We included 4 baseline methods, namely BC and IL, RL from scratch, and a fine-tuning baseline, PIRLNav [22] , where BC policy is fine-tuned using RL without adaptivity. We average the evaluation metrics, SR, MGE, and LT among various H. The results are shown in Table . 1. Across all configurations and racing tracks, our approach consistently exhibits the best performance in all the metrics compared to all baselines, under the same 10M data sample budget. In our approach, since our third stage takes advantage of an asymmetric critic containing privileged state information, the resulting policy will possess more task understanding. For the detailed metrics for the individual history length, we refer the readers to the Appendix.To demonstrate the effectiveness of our visuomotor policy learning approach, we ablate the training performance of our approach with different baseline RL configurations: (i) RL from scratch using the asymmetric critic with privileged states, and (ii) Vanilla fine-tuning, where we initialize the parameters of the actor network using the same pre-trained DAgger policy perform same RL training setup as used in stage I. This setup is distinct from the baseline PIRLNav where a BC policy is used for initialization. For (i) we train the RL policy using RGB images with 10M samples and our approach and baseline (ii) we use 5M data samples for pretraining and 5M data samples for fine-tuning. For a fair comparison, we selected the policy with H = 32 for all the methods, which balances good policy performance with the network's capability for real-time, low-latency control of our quadrotor. The results are illustrated in Fig. 5 left. Firstly, it is noteworthy that the direct RL from corners or pixels achieves a 0% success rate in all three tracks. This once again underscores the difficulty of RL exploration in high-dimensional time series without bootstrapping. The policy initially showed improved learning performance for the vanilla bootstrapping baseline but deteriorated with increasing timesteps. This highlights the importance of the adaptive component in our third phase for maintaining policy performance. Robutstness to unknown disturbances In the lowing experiments, we conduct two trials to verify whether our approach enhances the policy's robustness to unknown disturbances. To simulate realworld uncertainties, we conducted two experiments: i) random frame blackouts to mimic sensor failures like communication loss, and ii) random positional jumps during flight to simulate disturbances such as strong winds. The results are demonstrated in Table. 2, it is clear that our approach outperforms the baseline DAgger approach in terms of robustness to unknown disturbance.Towards the Limit of End-to-End Policy. To explore how far we can push or approach achieving an end-to-end policy that maximizes the potential of the physical platform, we conducted an experiment aimed at pushing the speed to its limit. We applied our approach to train on the SplitS track, based on the setup described in [7] , where the policy outperformed several human champions. In [7] , the policy achieved a median lap time of 5.52 seconds, while the best human pilots achieved 5.76 seconds in the real world. The main difference from our previous experiments is that we increased the maximum thrust limit. We tested our approach using the aforementioned corner observation and validated it through 100 runs as part of our evaluation. The policy's performance resulted in an average lap time of 5.82 seconds, with the best policy achieving a lap time of 5.54 seconds. This indicates that an end-to-end policy's extreme performance can match that of human world champions. Table 3 : Real-world performance comparison between DAgger and our fine-tuned approach for two types of input representations.Figure 8 SplitS Kidney MGE [m] LT [s] MGE [m] LT [s] MGE [m] LT [To demonstrate policy improvements, we validated our policy in real-world scenarios using Hardware-in-the-Loop (HIL) simulations, aided by a VICON motion capture system for perceptual inputs. We compared our approach against a DAgger policy trained using the same number of 10M data samples. The results are detailed in Table 3 , with supplementary videos providing visual footage. Our approach consistently achieved faster lap times and smaller gate errors in the real-world setting, confirming the effective real-world transfer of our vision-based quadrotor enhancements. Further analysis of the policies' real-world trajectories, depicted in Fig. 1 and Fig. 6 , shows that fine-tuning results in tighter trajectories and higher peak velocities. Notably, in the challenging SplitS maneuver, the RL-fine-tuned policy executed tighter turns and longer straights, optimizing overall speed. These improvements illustrate that RL fine-tuning enables the discovery of maneuvers that enhance speed and performance, surpassing the capabilities of imitation learning alone. 
limitations and discussions: 
 In this work, we introduced a novel approach by fusing the strengths of Reinforcement Learning (RL) and Imitation Learning (IL) for vision-based agile quadrotor flight, specifically focusing on autonomous drone racing. We demonstrate for the first time a visuomotor policy capable of navigating through a sequence of gates using solely gate corners or RGB images. One limitation is that our current setup is tested in the controlled lab settings, it will likely fail in an in-the-wild setup. Despite demonstrating superior robustness compared to existing baselines, we believe the perception module in our framework is to improve to handle more out-of-distribution cases. In this work, we utilized a pre-trained ResNet to ensure a fair performance assessment. For future work, we aim to integrate a customized vision encoder that leverages data from diverse simulation settings, modalities, and extensive real-world environments.The quadrotor is assumed to be a 6 degree-of-freedom rigid body of mass m and diagonal moment of inertia matrix J = diag(J x , J y , J z ). Furthermore, the rotational speeds of the four propellers Ω i are modeled as a first-order system with a time constant k mot where the commanded motor speeds Ω cmd are the input. World W and Body B frames are defined with an orthonormal basis i.e. {x W , y W , z W }. The frame B is located at the center of mass of the quadrotor. The state space is thus 17-dimensional and its dynamics can be written as:EQUATIONwhere g W = [0, 0, -9.81 m/s 2 ] ⊺ denotes earth's gravity, f prop , τ prop are the collective force and the torque produced by the propellers, and f drag is a linear drag term. The quantities are calculated as follows:EQUATIONEQUATIONwhere r P,i is the location of propeller i expressed in the body frame , f i , τ i are the forces and torques generated by the i-th propeller, and (k vx , k vy , k vz ) [44, 45] are linear drag coefficients.A.2 Reward Formulations for RL Trainings.The reward components are formulated as follows:EQUATIONHere d Gate (t) denotes the distance from the robot's center of mass to the center of the next gate to pass, δ cam is the angle between the camera's optical axis and the direction towards the center of the next gate. a represents the control command, and ω the bodyrate. λ 1 , λ 2 , λ 3 , λ 4 , c 1 , c 2 are different positive hyperparameters.In our experiments, we employ identical hyperparameters for both state-based teacher training and vision-based RL fine-tuning to ensure a fair comparison. These parameters are determined based on iterations, shown in Table . 4 in both simulation and real-world experiments, aiming to achieve optimal and smooth performance for the state-based policy.For state-based teacher training, we employ a policy network consisting of a two-layer MLP, each layer containing 256 neurons, with a final layer outputting a 4-dimensional vector using a tanh activation function. In imitation learning, a 3-layer Temporal Convolutional Network (TCN) is utilized to encode the 32 timestamps of perceptual inputs. The length of the temporal embedding is 128, followed by another two-layer MLP to output the control command. For imitation learning, we employ a batch size of 512, and convergence typically occurs after collecting 5M data samples over approximately 100 epochs. We incorporate a linear decay in the learning rate, starting at 1e-3 and decreasing to 1e-5 at 50 epochs, remaining unchanged for the remainder of the training process.A.4 Hardware Configurations.We deploy our approach in the real world using a high-performance racing drone with a maximum thrust-to-weight ratio (TWR) of 5.78. However, for our experiments, we have limited the TWR to 2.7. We use a modification of the Agilicious platform [46] for the real-world deployment. We have replaced the onboard computer with an RF receiver, which is connected directly to the flight controller 1 and takes care of parsing the collective thrust and bodyrate commands from the computer. Additionally, we also mount an ultra-low latency camera feed sender, which sends the live video stream to the base computer. This configuration is similar to the one used by professional drone racing pilots.A.5 Ablation study on Asymmetric Critic Formulation In stage III of our approach, the visuomotor policy undergoes fine-tuning using an asymmetric critic setup. In this experiment, we ablate how the critic configurations, as demonstrated in Fig. 3 , can impact policy performance. As depicted in Fig. 7 , RL finetuning with an asymmetric critic function achieves the highest reward within the same sample budget. At the same time, as shown previously, including privileged knowledge in the training process can also lead to better performance when handling partial observations.A.6 Performance w/ Diff. History LengthIn Table . 5 we detailed the results of our approach against the best baseline methods DAgger on varying different history lengths. It is evident that by incorporating more historical information, the student could achieve a higher success rate. More importantly, in all of these cases, our approach achieves both better performance and success rate. The enhanced performance of our RL-based approach in partially observable situations can be attributed to our asymmetric actor-critic setup, where additional state information is provided for value estimations during environment interactions. This setup significantly mitigates the challenges of partial observability, thereby improving the robustness and effectiveness of the learning process. Table 6 : Ablation study on history length of the policy observations using raw pixels. We could clearly find out by using more history observations, that the policy improvement will get improved. Notably, our approach consistently outperforms baseline methods across all history lengths ger to imitate the slow policy, after which we apply our approach to fine-tune the policy with the full thrust of 11.3N per motor. The detailed training curve is visualized in 9. We demonstrate that our policy training achieves improved sample efficiency, even for the state-based approach. The quantitative results, shown in 6, clearly indicate that our approach greatly improves policy performance, achieving lap times within a difference of 0.05s to that in [9] , where they outperformed human champions.https://www.betaflight.com 
 
 Appendix:  


Decision: Strengths: The paper is very well written, easy to understand and well motivated. 

Weaknesses: Additional comparisons to relevant baselines and a better substantiation for both sample-efficiency novelty and mitigation of catastrophic forgetting are needed.

-----

The effort the authors put into the careful rebuttal is commendable and in the revised paper the strengths clearly outweigh the weaknesses.

Reviews: review: title: title: The paper makes advancements towards true end-to-end agile robot control.
paper_summary: paper_summary: The paper presents how to combine reinforcement and imitation learning to distill a learning-based control policy to autonomously complete a drone racing track solely from vision inputs.
main_review: main_review: The paper is very well written, easy to follow, and supported by good/clear figures and tables. (The only suggestions I would have in terms of presentation is to explain all acronyms used in the tables also in their captions and fix the fact that some of the paragraph titles have title case, some have sentence case.) The experiment results section is, in a few instances, a bit less clear and I added a few questions for clarification below. The methodology is sufficiently clearly explained and multiple options/ablations (with regard to, race tracks, visual encoding and the imitation learning strategy) are explored. The paper is indeed very relevant to robotics and autonomy and, in a broader sense, could also serve as a starting point to better understand how humans implicitly understand and perform complex physical tasks almost only relying on vision and few other propioception/auditive cues. In this sense, the main thing that is missing from the paper is a direct comparison with a human and/or the other learning and state-based approaches that have achieved super-human performance in this task.
questions: questions: - "we validated our policy in real-world scenarios using Hardware-in-the-Loop (HIL) simulations, aided by a VICON motion capture system to generate perceptual inputs through rendering or gate corner projections." this sentence is ambiguous, how does the mocap system aid the rendering?
- "It is important to note that both i) and ii) are not domain randomization during the training" meaning 
- why imitate only from a state-based RL policy and not also from expert-collected data?
limitations: limitations: 3
review_summary: review_summary: The paper is a pleasant read and an interesting stepping stone in autonomous aerial robotics.
title: title: Review
paper_summary: paper_summary: This paper presents a method for optimizing vision-based flight controllers via reinforcement learning with imitation learning as an initialization.
main_review: main_review: The paper is clearly well written with high-quality images illustrating the results and proposed method. The problem setting is definitely original and the results are very well done. The experimental results seems very exciting and demonstrate what was done to be effective. In terms of strength, the paper does provide good motivation for the work and the missing components: sample-efficiency and mitigating catastrophic forgetting.

The main weakness of the work is that results do not necessarily support the sample-efficiency claim nor does the approach have any certainty that catastrophic forgetting is mitigated at all. Though there are results in the supplementary, it seems it is just DAgger with an ablation study on how much data it useful and a low-learning rate to prevent catastrophic forgetting. Specifically, at the start of page 5, the claim that the policy step should be kept small, while it may work for this specific problem, seems arbitrary and difficult to reproduce. What exact values are good learning rates versus bad and what can be learned from the appropriate learning rate?
questions: questions: First, the paper should address the following on sample-efficiency novelty claim: 

How is the approach improving sample-efficiency? 
Is there a specific gain in certain training components that do enhance sample-efficiency more than other components?

Second, regarding mitigation of catastrophic forgetting claims: 

At what step-size does catastrophic forgetting not occur? 
Are there any specific guarantees or rule of thumb based on empirical analysis that can be used or followed?

Third, regarding results: 
In table 2, what happens at 10 % disturbance?
Does the system perform just as good or does performance degrade significantly? 
Additional results on catastrophic forgetting would improve the paper's reproducibility.
limitations: limitations: 3
review_summary: review_summary: While the results and platform are exciting, the contributions currently stated are not strongly supported and come across as application of existing methods. More work should be done to highlight the contributions and support the claims.
title: title: Simple and powerful technique but missing baselines
paper_summary: paper_summary: The authors introduce a method for training a visuomotor policy by a three-stage training procedure: RL with privileged state information, IL, fine-tuning. The effectiveness of their approach is demonstrated in simulation and with a real-world quadrotor.
main_review: main_review: **Quality:** The approach is sensible, well-motivated and demonstrates strong results over baselines. However, I do feel as if there are some obvious shortcomings in the results and analysis: for baselines, the RL baseline is from scratch, but another obvious approach would be to mix the privileged state information used by the authors approach, with the image inputs. Since this is not the first paper to use privileged information, it would've been nice to see results against other existing methods. In terms of analysis, it would be insightful to understand the importance of the fine-tuning stage (is this the Dagger baseline? it is unclear).

**Clarity:** The paper is mostly well-written. There are some elements of the experimental results that could be more clear--for example it is unclear how the Dagger baseline is performed.

**Originality & Significance:** While the method itself offers mild novelty over existing approaches, there is originality and significance in the system design, particularly for the area of quadrotors.
questions: questions: - Include a RL baseline which uses privileged state information. 
- Can you explain the Dagger baseline in more detail? How does this differ from your approach without the fine-tuning stage?
limitations: limitations: 3
review_summary: review_summary: Straightforward method with convincing results in a real-world system.