FlightBench: Benchmarking Learning-based Methods for Ego-vision-based Quadrotors Navigation

## Abstract

Abstract Ego-vision-based navigation in cluttered environments is crucial for mobile systems, particularly agile quadrotors. While learning-based methods have shown promise recently, head-to-head comparisons with cutting-edge optimization-based approaches are scarce, leaving open the question of where and to what extent they truly excel. In this paper, we introduce FlightBench, the first comprehensive benchmark that implements various learning-based methods for ego-vision-based navigation and evaluates them against mainstream optimization-based baselines using a broad set of performance metrics. Additionally, we develop a suite of criteria to assess scenario difficulty and design test cases that span different levels of difficulty based on these criteria. Our results show that while learning-based methods excel in high-speed flight and faster inference, they struggle with challenging scenarios like sharp corners or view occlusion. Analytical experiments validate the correlation between our difficulty criteria and flight performance. We hope this benchmark and these criteria will drive future advancements in learning-based navigation for ego-vision quadrotors.
The source code and documentation is available at https://github.com/thu-uav/FlightBench .

## 1 Introduction

Ego-vision-based navigation in cluttered environments is a fundamental capability for mobile systems and has been widely investigated (Anwar & Raychowdhury, ; Chi et al., ; Wang et al., ; Xiao et al., ; Stachowicz et al., ). It involves navigating a robot to a goal position without colliding with any obstacles in its environment, using equipped ego-vision cameras (Agarwal et al., ). Quadrotors, known for their agility and dynamism (Verbeke & Schutter, ; Loquercio et al., ), present unique challenges in achieving fast and safe flight. Traditionally, hierarchical methods address this problem by decoupling it into subtasks such as mapping, planning, and control (Xiao et al., ), optimizing the trajectory to avoid collisions. In contrast, recent works (Loquercio et al., ; Song et al., ; Kaufmann et al., ) have demonstrated that learning-based methods can unleash the full dynamic potential of agile platforms. These methods employ neural networks to generate a sequence of waypoints (Loquercio et al., ) or motion commands (Song et al., ; Kaufmann et al., ), utilizing state estimation and ego-vision input. Unlike the high computational costs associated with sequentially executed subtasks, this end-to-end manner significantly reduces processing latency and enhances agility (Loquercio et al., ).

Despite the promising results of learning-based navigation methods, the lack of head-to-head comparisons with state-of-the-art optimization-based methods makes it unclear in which areas they truly outperform and to what degree. Traditional methods are often evaluated using customized scenarios and sensor configurations (Ren et al., ; Zhou et al., ; Song et al., ), which complicates reproducibility and hinders fair comparisons. Moreover, the absence of a quantifiable approach for scenario difficulty further obscures the analysis of the strengths and weaknesses of current methods.

In this paper, we introduce FlightBench, a comprehensive benchmark that evaluates methods for ego-vision-based quadrotor navigation.
We initially developed a suite of learning-based methods, encompassing both ego-vision and privileged ones for an in-depth comparative analysis. Additionally, we incorporated several representative optimization-based benchmarks for further head-to-head evaluation.
Moreover, we established three criteria to measure the difficulty of scenarios, thereby creating a diverse array of tests that span a spectrum of difficulties.
Finally, we compared these methods across a wide range of performance metrics to gain a deeper understanding of their specific attributes.
Our experiments indicate that learning-based methods demonstrate superior performance in high-speed flight scenarios and generally offer quicker inference times. However, they encounter difficulties in handling complex situations such as sharp turns or occluded views. In contrast, our findings show that traditional optimization-based methods maintain a competitive edge. They perform well in challenging conditions, not just in success rate and flight quality, but also in computation time, particularly when they are meticulously designed. Our analytical experiments validate the effectiveness of the proposed criteria and emphasize the importance of latency randomization for learning-based methods.
In summary, our key contributions include:

- 1.
The development of FlightBench, the first unified open-source benchmark that facilitates the head-to-head comparison of learning-based and optimization-based methods on ego-vision-based quadrotor navigation under various 3D scenarios.
- 2.
The proposition of tailored task difficulty and performance metrics, aiming to enable a thorough evaluation and in-depth analysis of specific attributes for different methods.
- 3.
Detailed experimental analyses that demonstrate the comparative strengths and weaknesses of learning-based versus optimization-based methods, particularly in difficult scenarios.

## 2 Related Work

##### Planning methods for navigation.

Classical navigation algorithms typically use search or sampling to explore the configuration or state space and generate a free path (Kamon & Rivlin, ; Rajko & LaValle, ; Penicka & Scaramuzza, ). With optimization, a multi-objective optimization problem is often formulated to determine the optimal trajectory (Paull et al., ; Ye et al., ). This is commonly done using gradients from local maps, such as the Artificial Potential Field (APF) (Zhu et al., ; Sfeir et al., ) and the Euclidean Signed Distance Field (ESDF) (Zhou et al., ).
On the other side, the development of deep learning enables algorithms to perform navigating directly from sensory inputs such as images or lidar (Xiao et al., ). The policies are trained by imitating expert demonstrations (Schilling et al., ) or through exploration under specific rewards (Liu et al., ). Learning-based algorithms have been applied to various mobile systems, such as quadrupedal robots (Agarwal et al., ), wheeled vehicles (Chaplot et al., ; Stachowicz et al., ), and quadrotors (Kaufmann et al., ; Xing et al., ). In this work, we examine representative methods for ego-vision-based navigation on quadrotors, including two learning-based approaches and three optimization-based methods, providing a comprehensive comparison between these categories.

**Table 1: A comparison of FlightBench to other open-source benchmarks for navigation.**
| Benchmark | 3D Scenarios | Classical<br>Methods | Learning<br>Methods | Sensory<br>Input |
| --- | --- | --- | --- | --- |
| MRBP 1.0 (Wen et al., ) | ✗ | ✓ | ✗ | LiDAR |
| Bench-MR (Heiden et al., ) | ✗ | ✓ | ✗ | - |
| PathBench (Toma et al., ) | ✓ | ✓ | ✓ | - |
| Gibson Bench (Xia et al., ) | ✗ | ✗ | ✓ | Vision |
| OMPLBench (Moll et al., ) | ✓ | ✓ | ✗ | - |
| RLNav (Xu et al., ) | ✗ | ✓ | ✓ | LiDAR |
| Plannie (Rocha & Vivaldini, ) | ✓ | ✓ | ✓ | - |
| FlightBench (Ours) | ✓ | ✓ | ✓ | Vision |

##### Benchmarks for navigation.

Several benchmarks exist for non-sensory-input navigation algorithms, such as OMPL (Moll et al., ), Bench-MR (Heiden et al., ), PathBench (Toma et al., ), and Plannie (Rocha & Vivaldini, ). OMPL and Bench-MR primarily focus on sampling-based methods, while PathBench evaluates graph-based and learning-based methods. Plannie offers sampling-based, heuristic, and learning-based methods for quadrotors. However, these algorithms do not utilize input from onboard sensors as well.
For methods with sensory inputs, most benchmarks are primarily designed for 2D scenarios. MRBP1.0 (Wen et al., ) and RLNav (Xu et al., ) evaluate planning methods using laser-scanning data for navigation around columns and cubes. GibsonBench (Xia et al., ) features a mobile agent equipped with a camera, navigating in interactive environments. As outlined in Tab. 1, there’s a notable lack of a benchmark with 3D scenarios and ego-vision inputs to assess and compare both classical and learning-based navigation algorithms, a gap that FlightBench aims to fill.

## 3 FlightBench

Figure: Figure 1: An overview of the FlightBench. FlightBench consists of three main components: (1) Tasks, featuring three scenarios categorized into eight difficulty levels. (2) Baselines, the core benchmarking platform supporting five ego-vision-based methods and two privileged methods. (3) Evaluation Metrics, offering a thorough suite of performance assessment metrics.
Refer to caption: extracted/5892626/figs/overview.png

In this section, we detail the components of FlightBench. An overview of our benchmark is depicted in Fig. 1.
In FlightBench, to design a set of Tasks with distinguishable characteristics for assessment, we propose three criteria, named task difficulty metrics, and develop eight tests across three scenarios based on these metrics. We integrate various representative Baselines to examine the strengths and features of both learning-based and optimization-based navigation methods. Furthermore, we establish a comprehensive set of performance Evaluation Metrics to facilitate quantitative comparisons. The next subsections will provide an in-depth look at the Tasks, Baselines, and Evaluation Metrics.

### 3.1 Tasks

#### 3.1.1 Task Difficulty Metrics

Figure: (a) Traversability Obstruction
Refer to caption: extracted/5892626/figs/clip_TO.png

Each task difficulty metric quantifies the challenge of a test configuration from a specific perspective. In quadrotor navigation, a test is configured by the obstacles-laden scenario, start point, and end point.
We utilize the topological guide path $\mathcal{T}$, which comprises interconnected individual waypoints (Penicka & Scaramuzza, ), to establish the quantitative assessment. In FlightBench, we propose three main task difficulty metrics: Traversability Obstruction (TO), View Occlusion (VO), and Angle Over Length (AOL).

##### Traversability Obstruction.

Traversability Obstruction (TO) measures the flight difficulty due to limited traversable space caused by obstacles.
We use a sampling-based approach (Ren et al., ) to construct sphere-shaped flight corridors $\{B_{0},\dots,B_{N_{T}}\}$, where $N_{T}$ is the number of spheres representing traversable space along path $\mathcal{T}$. Fig. 2(a) illustrates the primary notations for computing these corridors.
The next sampling center $\mathbf{p}_{\text{hi}}$ is chosen from existing spheres $\{B_{0},\dots,B_{i-1}\}$ along $\mathcal{T}$. We sample $K$ candidate centers from a 3D Gaussian distribution $\mathcal{D}$ around $\mathbf{p}_{\text{hi}}$. Each candidate sphere $B_{\text{cand}}$ is defined by its center $\mathbf{p}_{\text{cand}}$ and radius $r_{\text{cand}}=||\mathbf{p_{\text{cand}}}-\mathbf{n_{\text{cand}}}||_{2}-r_{d}$, where $\mathbf{n_{\text{cand}}}$ is the nearest obstacle and $r_{d}$ is the drone radius.
For each $B_{\text{cand}}$, we compute $S_{\text{cand}}$:

$$ $S_{\text{cand}}=k_{1}V_{\text{cand}}+k_{2}V_{\text{inter}}-k_{3}(\mathbf{d} \cdot\mathbf{z})-k_{4}||\mathbf{d}-(\mathbf{d}\cdot\mathbf{z})\mathbf{z}||_{2}$ (1) $$

where $k_{1},k_{2},k_{3},k_{4}\in\mathbb{R}^{+}$, $V_{\text{cand}}$ is the volume of $B_{\text{cand}}$, $V_{\text{inter}}$ is the overlap with $B_{i-1}$, $\mathbf{d}=\mathbf{p}_{\text{cand}}-\mathbf{p}_{\text{hi}}$, and $\mathbf{z}$ is the unit vector along $\mathbf{p}_{\text{hi}}-\mathbf{p}_{i}$. The sphere with the highest $S_{\text{cand}}$ is selected as the next sphere. This process repeats until path $\mathcal{T}$ is fully covered.
Occlusion challenges mainly occur in narrow spaces, so sphere radii $\{r_{1},\dots,r_{N_{T}}\}$ are sorted in ascending order. The traversability obstruction metric $\mathbb{T}$ is defined in Eq. 2, where $R$ represents the sensing range.

$$ $\mathbb{T}=\frac{1}{N_{T}}\sum_{i=1}^{\lfloor N_{T}/2\rfloor}\frac{R}{r_{i}}.$ (2) $$

##### View Occlusion.

In ego-vision-based navigation tasks, a narrow field of view (FOV) can limit the drone’s perception (Tordesillas & How, ; Chen et al., ), posing a challenge to the perception capabilities of various methods (Gao et al., ). We use the term view occlusion (VO) to describe the extent to which obstacles block the FOV in a given scenario. The more obstructed the view, the higher the view occlusion.
As shown in Fig. 2(b), we sample drone position $\left\{\mathbf{q_{i}}\right\}$ and FOV unit vector $\left\{\mathbf{v_{i}}\right\}$ along $\mathcal{T}$ with $i\in\left\{1,\ \cdots,\ N_{V}\right\}$. For each sampled pair $\left\{\mathbf{q_{i}},\mathbf{v_{i}}\right\}$, we divide FOV into $M$ parts and calculate the distance $s_{ij}$ between the nearest obstacle point and drone position $q_{i}$ in each part $j$. The view occlusion $\mathbb{V}$ can be represented as Eq. 3, where $m_{j}$ is a series of weights, which gives higher weight to obstacles closer to the center of the view.

$$ $\mathbb{V}=\frac{1}{N_{V}}\sum_{i=1}^{N_{V}}\sum_{j=1}^{M}m_{j}\frac{R}{s_{ij}}.$ (3) $$

##### Angle Over Length.

For a given scenario, frequent and violent turns in traversable paths pose challenges for the agility of planning. Inspired by Heiden et al. (), we employ the concept of Angle Over Length (AOL) denoted as $\mathbb{A}$ to quantify the sharpness of a path. The AOL $\mathbb{A}$ is defined by Eq. 4, where $N_{AOL}$ signifies the number of angles depicted in Fig. 2(c), $\theta_{i}$ represents the $i$-th angle within the topological path $\mathcal{T}$, and $L$ stands for the length of $\mathcal{T}$.

$$ $\mathbb{A}=\frac{1}{L}\sum_{i=1}^{N_{AOL}}\left(\exp{\left(\frac{\theta_{i}}{ \pi/6}\right)}-1\right).$ (4) $$

#### 3.1.2 Scenarios and Tests

**Table 2: Task difficulty score of each test case.**
| Scenarios | Test Cases | TO | VO | AOL |
| --- | --- | --- | --- | --- |
| Forest | 1 | 0.76 | 0.30 | 7.64$\times$10^-4 |
| 2 | 0.92 | 0.44 | 1.62$\times$10^-3 |  |
| 3 | 0.90 | 0.60 | 5.68$\times$10^-3 |  |
| Maze | 1 | 1.42 | 0.51 | 1.36$\times$10^-3 |
| 2 | 1.51 | 1.01 | 0.010 |  |
| 3 | 1.54 | 1.39 | 0.61 |  |
| MW | 1 | 1.81 | 0.55 | 0.08 |
| 2 | 1.58 | 1.13 | 0.94 |  |

As illustrated in Fig. 1, our benchmark incorporates specific tests based on three scenarios: Forest, Maze, and Multi-Waypoint. These scenarios were chosen for their representativeness and frequent use in evaluating the performance of quadrotor navigation methods. (Ren et al., ; Kaufmann et al., ). Within these scenarios, we developed eight tests, each characterized by varying levels of difficulty. The task difficulty scores for each test are detailed in Tab. 2.

The Forest scenario serves as the most common benchmark for quadrotor
navigation. We differentiate task difficulty based on obstacle density and establish three tests, following the settings used by Loquercio et al. ().
In the Forest scenario, TO and VO metrics increase with higher tree density. AOL is particularly low due to sparsely spanned obstacles, making this scenario suitable for high-speed flights (Loquercio et al., ; Ren et al., ).

The Maze scenario consists of walls and boxes, creating consecutive sharp turns and narrow gaps. Quadrotors must navigate these confined spaces while maintaining flight stability and perception awareness (Park et al., ). We devise three tests with varying lengths and turn complexities for Maze, resulting in discriminating difficulty levels for VO and AOL.

The Multi-Waypoint (MW) scenario involves flying through multiple waypoints at different heights sequentially (Song et al., ). This scenario also includes boxes and walls as obstacles. We have created two tests with different waypoint configurations. The MW scenario is relatively challenging, featuring the highest TO in test 1 and the highest AOL in test 2.

### 3.2 Baselines

Here, we introduce the representative methods for ego-vision-based navigation evaluated in FlightBench, covering two learning-based methods and three optimization-based methods, as well as two privileged methods that leverage access to environmental information. The characteristics of each method are detailed in Tab. 3. For links to the open-source code, key parameters, and implementation details, please refer to Appendix A.

**Table 3: Characteristics of the navigation methods for quadrotors. "RL" denotes reinforcement learning. "IL" represents imitation learning. "GM" and "EM" refer to Grid Mapping and ESDF Mapping, respectively. The control level indicates the part of the control stack used by the baseline.**
|  | Method Type | Priv. | Decision | Mapping | Planning | Control Stack |
| --- | --- | --- | --- | --- | --- | --- |
| Info. | Horizon | Traj. | Waypoint | Motion Cmd |  |  |
| SBMT | Samp.-based | ✓ | Global | Planning Module | MPC |  |
| LMT | RL | ✓ | Local | Policy Network |  |  |
| Fast-Planner | Opti.-based | ✗ | Global | GM+EM | Planning Module | MPC |
| EGO-Planner | Opti.-based | ✗ | Local | GM | Planning Module | MPC |
| TGK-Planner | Opti.-based | ✗ | Global | GM | Planning Module | MPC |
| Agile | IL | ✗ | Local | Policy Network | MPC |  |
| LPA | RL+IL | ✗ | Local | Policy Network |  |  |

##### Learning-based Methods.

Utilizing techniques such as imitation learning (IL) and reinforcement learning (RL), learning-based methods train neural networks for end-to-end planning, bypassing the time-consuming mapping process. Agile (Loquercio et al., ) employs DAgger (Ross et al., ) to imitate an expert generating collision-free trajectories using Metropolis-Hastings sampling and outputs mid-level waypoints. LPA (Song et al., ) combines IL and RL, starting with training a teacher policy using LMT (Penicka et al., ), then distilling this expertise into a ego-vision-based student. Both teacher and student policies generate executable motion commands, specifically collective thrust and body rates (CTBR).

##### Optimization-based Methods.

Optimization-based methods typically include an online mapping module followed by planning and control modules, often referred to as planners. The three optimization-based methods described below generate B-spline trajectories (see Tab. 3). The control stack samples mid-level waypoints from the trajectory function and uses a low-level Model Predictive Control (MPC) controller to convert these waypoints into motion commands (Fig. 3).
Among these baselines, Fast-Planner (Zhou et al., ) constructs both occupancy grid and Euclidean Signed Distance Field (ESDF) maps, whereas TGK-Planner (Ye et al., ) and EGO-Planner (Zhou et al., ) only require an occupancy grid map. In the planning stage, EGO-Planner (Zhou et al., ) focuses on trajectory sections with new obstacles, acting as a local planner, while the other two use a global search front-end and an optimization back-end for long-horizon planning.

Figure: Figure 3: A generic processing pipeline for hierarchical navigation systems on quadrotors.
Refer to caption: x1.png

3. We follow Song et al. () to reproduce the work with out open source code and the key hyperparameters are listed in Table 8

##### Privileged Methods.

SBMT (Penicka & Scaramuzza, ) is a sampling-based method that uses global ESDF maps to generate a collision-free trajectory of dense points, which is then tracked by an MPC controller (Falanga et al., ). Its follow-up, LMT (Penicka et al., ), employs RL to train an end-to-end policy for minimum-time flight. This policy uses the quadrotor’s full states and the next collision-free point (CFP) to produce motion commands, with the CFP determined by finding the farthest collision-free point on a reference trajectory (Penicka & Scaramuzza, ).

### 3.3 Evaluation Metrics

We represent quadrotor states as a tuple $\left(\mathbf{x}(t),\mathbf{v}(t),\mathbf{a}(t),\mathbf{j}(t)\right)$, where $t$ denotes time, $\mathbf{x}(t)$ denotes the position, and $\mathbf{v}(t)=\dot{\mathbf{x}}(t)$, $\mathbf{a}(t)=\dot{\mathbf{v}}(t)$, $\mathbf{j}(t)=\dot{\mathbf{a}}(t)$ are the velocity, acceleration, and jerk in the world frame, respectively. $T$ denotes the time taken to fly from the starting point to the end point.

First, we integrate three widely used metrics (Ren et al., ; Loquercio et al., ) into FlightBench. Success rate measures if the quadrotor reaches the goal within a 1.5 m radius without crashing. Average speed, defined as $\frac{1}{T}\int_{0}^{T}||\mathbf{v}(t)||_{2}\,\mathrm{d}t$, reflects the achieved agility. Computation time evaluates real-time performance as the sum of processing times for mapping, planning, and control. Additionally, we introduce average acceleration and average jerk (Zhou et al., ; ), defined as $\bar{\mathbf{a}}=\frac{1}{L}\int_{0}^{T}||\mathbf{a}(t)||_{2}^{2}\,\mathrm{d}t$ and $\bar{\mathbf{j}}=\frac{1}{L}\int_{0}^{T}||\mathbf{j}(t)||_{2}^{2}\,\mathrm{d}t$, respectively. Average acceleration indicates energy consumption, while average jerk measures flight smoothness (Wang et al., ). These metrics, though not commonly used in evaluations, are crucial for assessing practicability and safety in real-world applications.
However, average acceleration and jerk only capture the dynamic characteristics of a flight. For instance, higher flight speeds along the same trajectory result in greater average acceleration and jerk. To assess the static quality of trajectories, we propose average curvature, inspired by Heiden et al. (). Curvature is calculated as $\kappa(t)=\frac{|\mathbf{v}(t)\times\mathbf{a}(t)|}{|\mathbf{v}(t)|^{3}}$, and average curvature is defined as $\bar{\kappa}=\frac{1}{L}\int_{0}^{T}\kappa(t)\mathbf{v}(t)\,\mathrm{d}t$.

Together, these six metrics provide a comprehensive comparison of learning-based algorithms against optimization-based methods for ego-vision-based quadrotor navigation. Our extensive experiments will demonstrate that while learning-based methods excel in certain metrics, they also have shortcomings in others.

## 4 Experiments

Upon FlightBench, which provides a variety of metrics and scenarios to evaluate the performance of ego-vision-based navigation methods for quadrotors, we carried out extensive experiments to study the following research questions:

- •
What are the main advantages and limitations of learning-based methods compared to optimization-based methods?
- •
How does the navigation performance vary across different scenario settings?
- •
How much does the system latency introduced by the practical evaluation environment affect performance?

### 4.1 Setup

For simulating quadrotors, we use Flightmare (Song et al., ) with Gazebo (Koenig & Howard, ) as its dynamic engine. To mimic real-world conditions, we develop a simulated quadrotor model equipped with an IMU sensor and a depth camera, calibrated with real flight data. The physical characteristics of the quadrotor and sensors are detailed in Appendix B. To simulate real-world communication delays, all data transmission in the simulation uses ROS (Quigley et al., ).
All simulations are conducted on a desktop PC with an Intel Core i9-11900K processor and an Nvidia 3090 GPU. To evaluate real-time reaction performance on embedded platforms with limited computational resources, we also measure computation time on the Nvidia Jetson Orin NX module. Each evaluation metric is averaged over ten independent runs.

### 4.2 Benchmarking Flight Performance

**Table 4: Performance evaluation of different methods under tests with the highest AOL within each scenario. The highest performing values for each metric are highlighted in bold, with the second highest underlined.**
| Scen. | Metric | Privileged | Optimization-based | Learning-based |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SBMT | LMT | TGK | Fast | EGO | Agile | LPA |  |  |
| Forest | Success Rate $\uparrow$ | 0.80 | 1.00 | 0.90 | 0.90 | 1.00 | 0.90 | 1.00 |
| Avg. Spd. (ms^-1) $\uparrow$ | 15.25 | 11.84 | 2.30 | 2.47 | 2.49 | 3.058 | 8.96 |  |
| Avg. Curv. (m^-1) $\downarrow$ | 0.06 | 0.07 | 0.08 | 0.06 | 0.08 | 0.37 | 0.08 |  |
| Avg. Acc. (ms^-3) $\downarrow$ | 28.39 | 10.29 | 0.25 | 0.19 | 0.83 | 4.93 | 9.96 |  |
| Avg. Jerk (ms^-5) $\downarrow$ | 4.27$\times$10^3 | 8.14$\times$10^3 | 1.03 | 3.97 | 58.39 | 937.02 | 1.14$\times$10^4 |  |
| Maze | Success Rate $\uparrow$ | 0.60 | 0.9 | 0.50 | 0.60 | 0.20 | 0.50 | 0.30 |
| Avg. Spd. (ms^-1) $\uparrow$ | 8.73 | 9.62 | 1.85 | 1.99 | 2.19 | 3.00 | 8.35 |  |
| Avg. Curv. (m^-1) $\downarrow$ | 0.31 | 0.13 | 0.17 | 0.23 | 0.33 | 0.68 | 0.21 |  |
| Avg. Acc. (ms^-3) $\downarrow$ | 60.73 | 26.26 | 0.50 | 0.79 | 1.91 | 15.45 | 37.30 |  |
| Avg. Jerk (ms^-5) $\downarrow$ | 6.60$\times$10^3 | 4.64$\times$10^3 | 6.74 | 9.62 | 80.54 | 2.15$\times$10^3 | 4.64$\times$10^3 |  |
| MW | Success Rate $\uparrow$ | 0.70 | 0.90 | 0.40 | 0.80 | 0.50 | 0.60 | 0.50 |
| Avg. Spd. (ms^-1) $\uparrow$ | 5.59 | 6.88 | 1.48 | 1.73 | 2.13 | 3.05 | 6.72 |  |
| Avg. Curv. (m^-1) $\downarrow$ | 0.47 | 0.30 | 0.46 | 0.32 | 0.62 | 0.67 | 0.26 |  |
| Avg. Acc. (ms^-3) $\downarrow$ | 80.95 | 31.23 | 1.07 | 0.97 | 5.06 | 16.86 | 36.77 |  |
| Avg. Jerk (ms^-5) $\downarrow$ | 9.76$\times$10^3 | 1.66$\times$10^4 | 25.52 | 22.72 | 155.83 | 2.07$\times$10^3 | 6.19$\times$10^3 |  |

#### 4.2.1 Flight Quality

To systematically assess the strengths and weaknesses of various methods on ego-vision navigation, we conduct evaluations across all tests within three distinct scenarios. Tab. 4 displays the results for tests with the highest AOL in each scenario. The comprehensive results for all tests are included in Sec. C.1 due to space limitations.
We evaluate these methods using our proposed evaluation metrics, and computation time will be addressed separately in a subsequent discussion.
In these experiments, we standardize the expected maximum speed at $3m/s$ for a fair comparison. Exceptions are SBMT, LMT, and LPA, whose flight speeds cannot be manually controlled.

As shown in Tab. 4, the privileged methods, with a global awareness of obstacles, set the upper bound for motion performance in terms of average speed and success rate. In contrast, the success rate of ego-vision methods in the Maze and MW scenarios is generally below $0.6$, indicating that our benchmark remains challenging for ego-vision methods, especially at the perception level.

Learning-based methods, known for their aggressive maneuvering, tend to fly less smoothly and consume more energy. They also experience more crashes in areas with large corners, as seen in the Maze and MW scenarios. When performing a large-angle turn, an aggressive policy is more likely to cause the quadrotor to lose balance and crash. Optimization-based methods are still competitive or even superior to current learning-based approaches, particularly in terms of minimizing energy costs. By contrasting the more effective Fast-Planner with the more severely impaired TGK-Planner and EGO-Planner, we find that global trajectory smoothing and enhancing the speed of replanning are crucial for improving success rates in complex scenarios. Detailed analyses on the failure cases can be found in the Sec. C.2.

###### Remark .

Learning-based methods tend to execute aggressive and fluctuating maneuvers, yet they struggle with instability in scenarios with high challenges related to VO and AOL.

###### Remark .

#### 4.2.2 Impact of Flight Speed

Figure: Figure 4: Success rate with different flight speeds.
Refer to caption: extracted/5892626/figs/success-speed.jpg

As discussed above, learning-based methods, which tend to fly more aggressively with greater fluctuation, underperform in scenarios requiring large turning angles. This section examines the performance of various methods in the expansive Forest scenario as flight speeds vary. We conduct experiments under test 2 of the Forest scenario, which records the highest TO values. As shown in Fig. 4, we evaluate the success rate of each method at different average flight speeds, excluding three methods where speed is integral to the planning process and non-adjustable.

Learning-based methods exhibit agile avoidance and operate closer to dynamic limits due to their straightforward end-to-end architecture. In contrast, optimization-based methods struggle in high-speed flights, as their hierarchical architecture and latency between sequential modules can cause the quadrotor to overshoot obstacles before a new path is planned. A detailed analysis of the computation time for the pipeline is discussed in Sec. 4.2.3. Additionally, privileged methods show higher success rates and faster flights, highlighting the substantial potential for improvement in current ego-vision-based methods.

###### Remark .

Learning-based methods consistently surpass optimization-based baselines for high-speed flight, yet they fall significantly short of the optimal solution.

**Table 5: Computation time of different baselines. $T_{\text{map}},\ T_{\text{plan}},\ T_{\text{ctrl}},\ T_{\text{tot}}$ stands for mapping time, planning time, control time, and total time, respectively.**
|  | SBMT | LMT | TGK | Fast | EGO | Agile | LPA |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Desktop | $T_{\text{tot}}$ (ms) | 3.189$\times$10^5 | 2.773 | 11.960 | 8.196 | 3.470 | 5.573 | 1.395 |
| $T_{\text{map}}$ (ms) | - | 1.607 | 3.964 | 7.038 | 2.956 | 0.338 | 0.399 |  |
| $T_{\text{plan}}$ (ms) | 2.589$\times$10^5 | 1.167 | 7.994 | 1.155 | 0.510 | 5.115 | 0.995 |  |
| $T_{\text{ctrl}}$ (ms) | - | - | 0.002 | 0.003 | 0.003 | 0.119 | - |  |
| Onboard | $T_{\text{tot}}$ (ms) | - | 13.213 | 37.646 | 39.177 | 24.946 | 27.458 | 12.293 |
| $T_{\text{map}}$ (ms) | - | 2.768 | 27.420 | 36.853 | 24.020 | 1.175 | 4.313 |  |
| $T_{\text{plan}}$ (ms) | - | 10.445 | 10.211 | 2.310 | 0.910 | 26.283 | 7.980 |  |
| $T_{\text{ctrl}}$ (ms) | - | - | 0.015 | 0.014 | 0.016 | - | - |  |

###### Remark .

#### 4.2.3 Computation Time Analyses

To evaluate the computation time of various planning methods, we measured the time consumed at different stages on both desktop and onboard platforms. The results are presented in Tab. 5, with the computation time broken down into mapping, planning, and control stages as shown in Fig. 3. For learning-based methods, the mapping time includes converting images to tensors and pre-processing quadrotor states. Notably, SBMT optimizes the entire trajectory before execution, resulting in the highest computation time.

The computation time for learning-based methods is significantly influenced by neural network design. Agile exhibits longer planning time compared to LPA and LMT due to its utilization of a MobileNet-V3-small (Howard et al., ) encoder for processing depth images. Conversely, LPA and LMT employ simpler CNN and MLP structures, resulting in shorter inference time.
Regarding optimization-based methods, the mapping stage often consumes the most time, particularly on the onboard platform. Fast-Planner experiences the longest mapping duration due to the necessity of constructing an ESDF map. EGO-Planner demonstrates superior planning time, even outperforming some end-to-end learning-based methods, owing to its local replanning approach. Nonetheless, this may pose challenges in scenarios with high VO and AOL.
As discussed in Sec. 4.2.2, for algorithms requiring real-time onboard computation, computation time significantly influences the upper limit of flight speed. Algorithms with more lightweight architectures can operate at a higher replan frequency, enhancing their ability to react to sudden obstacles.

###### Remark .

Learning-based methods can achieve faster planning with compact neural network designs, while a well-crafted optimization-based methods can be equally competitive.

###### Remark .

### 4.3 Analyses on Effectiveness of Different Metrics

To demonstrate how various task difficulties influence different aspects of flight performance, we calculate the correlation coefficients between six performance metrics and three difficulty metrics for each method across multiple tests. The value at the intersection of the horizontal and vertical axes represents the absolute value of the correlation coefficient between the two metrics. A higher value indicates a stronger correlation.
Fig. 5 presents the average correlations for privileged and ego-vision-based methods, separately evaluating the impacts on agility and partial observation. Refer to Sec. C.3 for calculation details and the specific correlation coefficients for each method.

The results for privileged methods shown in Fig. 5(a) indicate that AOL and TO have a significant impact on the baseline’s motion performance. The correlation coefficients between AOL and average curvature, velocity and acceleration are all above 0.9, indicating that AOL describes the sharpness of the trajectory well. More specifically, high AOL results in high curvature and acceleration of the flight trajectory, as well as lower average speed. TO, indicating task narrowness, is a crucial determinant of flight success rates. In contrast to privileged methods where global information is accessible, as shown in Fig. 5(b), ego-vision-based methods primarily struggle with partial perception. Field-of-view occlusions and turns challenge real-time environmental awareness, making VO and AOL highly correlated with success rates.

###### Remark .

High VO and AOL significantly challenge learning-based methods, as these factors heavily impact the ego-vision-based method’s ability to handle partial observations and sudden reactions.

Figure: (a) Correlation heatmap for privileged methods
Refer to caption: extracted/5892626/figs/heat-privilege.png

###### Remark .

### 4.4 Impact of Latency on Learning-based Methods

**Table 6: Performance of RL-based methods under test 2 of Multi-Waypoint scenario.**
|  | Train w/ latency | Train w/o latency |  |  |
| --- | --- | --- | --- | --- |
|  | Success Rate $\uparrow$ | Progress | Success Rate $\uparrow$ | Progress |
| LMT | 0.9 | 0.96 | 0.3 | 0.57 |
| LPA | 0.5 | 0.73 | 0.0 | 0.32 |

Beyond the algorithmic factors previously discussed, learning-based methods face considerable challenges transitioning from training simulations to real-world applications. Latency significantly impacts sim-to-real transfer, particularly when simplified robot dynamics are used to enhance high-throughput RL training. We assess the influence of latency by testing learning-based methods in a ROS-based environment, where ROS, as an asynchronous system, introduces approximately $45ms$ of delay in node communication.

Tab. 6 details the performance of RL-based methods under the most challenging test, i.e., test 2 of the Multi-Waypoint scenario, with the highest VO, AOL, and the second-highest TO. To further evaluate the baseline’s performance at low success rates, we introduce *Progress*, a metric ranging from 0 to 1 that reflects the proportion of the trajectory completed before a collision occurs. The “train w/ latency” column displays results from training with simulated and randomized latencies between $25ms$ to $50ms$, whereas the “train w/o latency” column serves as a control group.
As shown in Tab. 6, “train w latency” significantly improves both success rate and progress by more than $50\%$ for two methods, emphasizing the importance of incorporating randomized latency in more realistic simulation environments and real-world deployments.

###### Remark .

Integrating latency randomization into the training of RL-based methods is essential to enhance real-world applicability.

###### Remark .

## 5 Conclusion

We present FlightBench, an open-source benchmark for comparing learning-based and optimization-based methods in ego-vision quadrotor navigation. It includes three optimization-based and two learning-based methods, along with two privileged baselines for thorough comparison. To assess the difficulty of different test configurations, we defined three criteria, resulting in diverse test scenarios. Comprehensive experiments were conducted using these baselines and tests, evaluated by carefully designed performance metrics.

Our results show that while learning-based methods excel in high-speed flight and inference speed, they need improvement in handling complexity and latency robustness. Optimization-based methods remain competitive, producing smooth, flyable trajectories. Analysis of correlations between metrics shows that the proposed task difficulty metrics effectively capture challenges in agility and partial perception. We aim for FlightBench to drive future progress in learning-based navigation for ego-vision quadrotors.

## 6 Limitations and Future Work

FlightBench currently focuses on ego-vision-based quadrotor navigation in static, simulated environments. In the future, we plan to extend its scope to dynamic scenarios and multi-sensor integration, incorporating navigation methods tailored for dynamic environments. This expansion will include comprehensive analysis and a user-friendly hardware platform for real-world validation. Additionally, we will explore sim-to-real transfer techniques to enhance the deployment of navigation methods, with a focus on learning-based approaches.

## Appendix A Implementation Details

In this section, we detail the implementation specifics of baselines in FlightBench, focusing particularly on methods without open-source code.

### A.1 Optimization-based Methods

Fast-Planner (Zhou et al., )(^1^11[https://github.com/HKUST-Aerial-Robotics/Fast-Planner](https://github.com/HKUST-Aerial-Robotics/Fast-Planner)), EGO-Planner (Zhou et al., )(^2^22[https://github.com/ZJU-FAST-Lab/ego-planner](https://github.com/ZJU-FAST-Lab/ego-planner)), and TGK-Planner (Ye et al., )(^3^33[https://github.com/ZJU-FAST-Lab/TGK-Planner](https://github.com/ZJU-FAST-Lab/TGK-Planner)) have all released open-source code. We integrate their open-source code into FlightBench and apply the same set of parameters for evaluation, as detailed in Tab. 7.

**Table 7: Key parameters of the optimization-based methods.**
|  | Parameter | Value | Parameter | Value |
| --- | --- | --- | --- | --- |
| All | Max. Vel. | 3.0 ms^-1 | Max. Acc. | 6.0 ms^-2 |
| Obstacle Inflation | 0.09 | Depth Filter Tolerance | 0.15m |  |
| Map Resolution | 0.1m |  |  |  |
| Fast-Planner<br>EGO-Planner | Max. Jerk | 4.0 ms^-3 | Planning Horizon | 6.5 m |
| TGK-Planner | krrt/rho | 0.13 m | Replan Time | 0.005 s |

### A.2 Learning-based Methods

Agile (Loquercio et al., )(^4^44[https://github.com/uzh-rpg/agile_autonomy](https://github.com/uzh-rpg/agile_autonomy)) is an open-source learning-based baseline. For each scenario, we finetune the policy from an open-source checkpoint using 100 rollouts before evaluation.

LPA (Song et al., ) has not provided open-source code. Therefore, we reproduce the two stage training process based on their paper. The RL training stage involves adding a perception-aware reward to LMT (Penicka et al., ) method, which will be introduced in Sec. A.3.
At the IL stage, DAgger (Ross et al., ) is employed to distill the teacher’s experience into an ego-vision student. All our experiments on LPA and LMT use the same set of hyperparameters, as listed in Tab. 8.

### A.3 Privileged Methods

SBMT (Penicka & Scaramuzza, )(^5^55[https://github.com/uzh-rpg/sb_min_time_quadrotor_planning](https://github.com/uzh-rpg/sb_min_time_quadrotor_planning)) is an open-source sampling-based trajectory generator. Retaining the parameters in their paper, we use SBMT package to generate topological guide path to calculate the task difficulty metrics, and employ PAMPC (Falanga et al., ) to track the generated offline trajectories.

We reproduce LMT (Penicka et al., ) from scratch based on the original paper, implementing the observation, action, reward function, and training techniques described in the paper. PPO (Yu et al., ) is used as the backbone algorithm, and its hyperparameters are listed in Tab. 8.

**Table 8: Key hyperparameters used in RL and IL.**
|  | Parameter | Value | Parameter | Value |
| --- | --- | --- | --- | --- |
| RL (Yu et al., ) | Actor Lr | 5e-4 | Critic Lr | 5e-4 |
| PPO Epoch | 10 | Batch Size | 51200 |  |
| Max Grad. Norm. | 8.0 | Clip Ratio | 0.2 |  |
|  | Entropy Coefficient | 0.01 |  |  |
| IL (Ross et al., ) | Lr | 2e-4 | Training Interval | 20 |
| Training Epoch | 6 | Max Episode | 2000 |  |

## Appendix B Onboard Parameters

The overall system is implemented using ROS 1 Noetic Ninjemys. As mentioned in the main paper, we identify a quadrotor equipped with a depth camera and an IMU from real flight data. The physical characteristics of the quadrotor and its sensors are listed in Tab. 9.

**Table 9: Parameters of the quadrotor and the sensors. $\omega_{xy}$ and $\omega_{z}$ refer to the angular velocity of roll, pitch, and yaw in the body frame of the quadrotor. “SRT” refers to single rotor thrust and “FOV” denotes the field of view.**
|  | Parameter | Value | Parameter | Value |
| --- | --- | --- | --- | --- |
| Quadrotor | Mass | 1.0 kg | Max $\omega_{xy}$ | 8.0 rad/s |
| Moment of Inertia | [5.9, 6.0, 9.8] g m^2 | Max. $\omega_{z}$ | 3.0 rad/s |  |
| Arm Length | 0.125 m | Max SRT | 0.1 N |  |
| Torque Constant | 0.0178 m | Min. SRT | 5.0 N |  |
| Sensors | Depth Range | 4.0 m | Depth FOV | 90${}^{\circ}\times$ 75^∘ |
| Depth Frame Rate | 30 Hz | IMU Rate | 100 Hz |  |

## Appendix C Additional Results

### C.1 Benchmarking Performance

The main paper analyzes the performance of the baselines only on the most challenging tests in each scenario due to space limitations. The full evaluation results are provided in Tab. 10, Tab. 11, and Tab. 12, represented in the form of “mean(std)”.

Figure: Table 10: Performance evaluation of different navigation methods in Forest scenario.

Figure: Table 11: Performance evaluation of different navigation methods in Maze scenario.

Figure: Table 12: Performance evaluation of different navigation methods in Multi-Waypoint scenario.

The results indicate that optimization-based methods excel in energy efficiency and trajectory smoothness. In contrast, learning-based approaches tend to adopt more aggressive maneuvers. Although this aggressiveness grants learning-based methods greater agility, it also raises the risk of losing balance in sharp turns.

### C.2 Failure Cases

As discussed in Sec. 4.2, our benchmark remains challenging for ego-vision planning methods. In this section, we specifically examine the most demanding tests within the Maze and Multi-Waypoint scenarios to explore how scenarios with high VO and AOL cause failures. Video illustrations of failure cases are provided in the supplementary material.

As shown in Fig. 6(a), Test 3 in the Maze scenario has the highest VO among all tests. Before the quadrotor reaches waypoint 1, its field of view is obstructed by wall (A), making walls (B) and the target space (C) invisible. The sudden appearance of wall (B) often leads to collisions. Additionally, occlusions caused by walls (A) and (C) increase the likelihood of navigating to a local optimum, preventing effective planning towards the target space (C).

Fig. 6(b) illustrates a typical Multi-Waypoint scenario characterized by high VO, TO, and AOL. In this scenario, the quadrotor makes a sharp turn at waypoint 2 while navigating through the waypoints sequentially. The nearest obstacle, column (D), poses a significant challenge due to the need for sudden reactions. Additionally, wall (E), situated close to column (D), often leads to crashes for baselines with limited real-time replanning capabilities.

Figure: (a) Maze Scenario
Refer to caption: extracted/5892626/figs/fail1.png

### C.3 Analyses on Effectiveness of Different Metrics

##### Correlation Calculation Method.

As the value of two metrics to be calculated for the correlation coefficient are denoted as $\{x_{i}\},\ \{y_{i}\}$, respectively. The correlation coefficient between $\{x_{i}\}$ and $\{y_{i}\}$ defines as

$$ $\text{Corr}_{x,y}=\frac{\sum_{i}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i} (x_{i}-\bar{x})^{2}\sum(y_{i}-\bar{y})^{2}}},$ (5) $$

where $\bar{x},\ \bar{y}$ are the average values of $\{x_{i}\}$ and $\{y_{i}\}$.

##### Results.

Fig. 7 shows the correlation coefficients between six performance metrics and three difficulty metrics for each method across multiple scenarios. As analyzed in Sec. 4.3, TO and AOL significantly impact the motion performance of two privileged methods (Fig. 7(a) and Fig. 7(b)). In scenarios with high TO and AOL, the baselines tend to fly slower, consume more energy, and exhibit less smoothness. Ego-vision methods are notably influenced by partial perception, making VO a crucial factor. Consequently, high VO greatly decreases the success rate of ego-vision methods, much more so than it does for privileged methods.

When comparing the computation times of different methods, we observe that the time required by learning-based methods primarily depends on the network architecture and is minimally influenced by the scenario. Conversely, the computation times for optimization- and sampling-based methods are affected by both AOL and TO. Scenarios with higher TO and AOL demonstrate increased planning complexity, resulting in longer computation times.

Figure: (a) Correlations for SBMT (Penicka & Scaramuzza, )
Refer to caption: extracted/5892626/figs/heat-sb.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Apart from the new metrics, how is this work different from Plannie? If I understand correctly, support for real environments is not included.

Otherwise, more metrics could be added (such as battery consumption) and more environmental factors (such as wind) could be also benchmarked for a better real world modelling.
soundness: 3
strengths: ### Novel benchmark
- unified open-source benchmark that enables direct comparison between learning-based and optimization-based methods for UAV navigation
- 3 new quantitative metrics for measuring scenario difficulty
- evaluation across multiple scenarios with varying difficulty levels
### Paper
- well written and illustrated, only minor errors falling through
summary: The authors propose FlightBench - a comprehensive benchmark for evaluating ego-vision-based drone navigation methods, comparing learning-based approaches with traditional optimization-based methods. 7 baseline methods are evaluated (2 learning-based, 3 optimization-based, 2 privileged).

The benchmark introduces three key metrics for assessing scenario difficulty: Traversability Obstruction (TO), View Occlusion (VO), and Angle Over Length (AOL). 

The test scenarios comprise three categories (Forest, Maze, Multi-Waypoint) with varying difficulty levels.
weaknesses: ### Scope / contribution
- engineering work, the paper appears to be a mix of experiments with existing methods; 3 task difficulty metrics are not enough for ICLR
- out of scope, more suitable for a robotics conference such as ICRA
### Benchmark issues
- limited number of learning-based methods evaluated
- limited to simulated environments

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Pl. refer to weaknesses.
soundness: 3
strengths: 1. The paper is generally well-written, and well motivates the need for a 3D scene based benchmark for ego-vision agile navigation, which is currently unaddressed by other benchmarks.  
2. The 3 proposed task-difficulty metrics, can comprehensively capture and quantify the challenge of a particular scene with obstacles. In addition, the proposed scenarios are diverse in terms of the task difficulty metrics and capture the different operating conditions often faced. 
3. A number of SOTA baselines have been used in the benchmark, covering both learning based, and optimization based methods, as well as methods that leverage additional environmental information. This helps the authors to analyze and remark on various factors in flight performance. Furthermore, the supplementary qualitatively discusses failure cases and their correlation with difficulty metrics.
summary: Summary
The paper introduces a benchmark, named FlightBench for ego-vision based quad rotor navigation methods. In particular, the paper first introduces the paucity of a 3d scene based benchmark to compare learning-based navigation methods against classical optimization methods. To do so, the paper has 3 main contributions:

1. 8 new tasks categorized into 3 scenarios - forests, mazes and multi-waypoints scenarios with varying levels of difficulty, all of which are simulated with Gazebo, Flightmare and ROS.
2. A total of 7 baseline methods categorized into ego-vision based (learning and optimization), and privileged methods.
3. 3 new task difficulty metrics - traversability obstruction, view occlusion and angle over length; to quantify challenges faced during agile navigation.

In addition to comparing learning based methods vs optimization based methods, the paper aims to also analyze navigation performance across different difficulty settings, and the effect of system latency, and provides conclusions on flight quality, flight speed, compute cost, latency and the effectiveness of metrics.
weaknesses: 1. While the benchmark introduces a variety of scenes, it is still quite limited - as a dataset benchmark, I would have expected more scenarios.  
2. Furthermore, the authors do not discuss and analyze the number of scenarios compared to previous baselines. It would be good to have this in the paper. The authors should also discuss the diversity of previous baselines in terms of their proposed task difficulty metrics. 
3. Gazebo and ROS-Noetic is used as the simulation platform, which often does not provide realistic scene quality, which could be important for ego-vision based learning. This is also visible through the lack of realistic scene quality. I am curious why the authors did not choose a newer platform like Isaac-Sim and ROS-2 given that they support the new de-facto standards?

Minor comments: This paper seems to be more suitable for a robotics conference/venue than ICLR.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please consider providing your feedback to the comments raised in the weaknesses section.
soundness: 3
strengths: - The motivation to bring together different methods (learnable and not) for vision-based UAV navigation and compare them head-to-head, addresses an important lack of standardization in the field, that can drive further progress.
- The selected testing environments are quite representative of a wide range of different flying scenarios. 
- The evaluation section offers valuable insights on the pros and cons of both categories of UAV navigation solutions, which is very informative and can drive further research and development in the field.

- Overall the paper is well-written and easy to follow, and features useful illustrations. A careful proof reading is needed to correct some sparingly appearing syntax and grammatical errors.
summary: This submission introduces FlightBench, a benchmark for ego-vision based UAV navigation in simulated environments. The proposed methodology brings together implementations of various traditional (optimisation-based) and learnable solutions from the literature, as well as oracle (environment-aware) baselines, and quantitatively compares them in 3 different environments with varying challenges. Several metrics capturing different aspects of navigation performance are employed (including success rate, speed, latency and other aspects such as energy and dynamic behavior, through proxies), along with a set of metrics to capture task difficulty based on the simulator environment. The manuscript concludes with several insights arising from this comparison, indicating areas for further improvement in the adopted baselines.
weaknesses: - The majority of metrics and methods considered in the proposed baseline have been proposed above, or widely used in the community. This may limit the novelty of the proposed approach. However, in my opinion its contribution remains unaffected, as it still provides a comprehensive suite for uniformly evaluating different navigation approaches. 
- The proposed benchmark remains rather targeted on static environments and sensing modalities. Driven by the easy extendibility of the proposed framework due to the use of simulated environments, incorporating more sensory cues (e.g. LiDAR, depth, IMU, event camera etc) and more scenarios (e.g. navigation in dynamic environments, drone racing etc) will create a more robust arena and further facilitate the attempt of this work towards standardization in the experiments of the field. 
- Relying solely on simulated data, several important aspects of the navigation performance of different methods cannot be accurate evaluated. These include (most crucially) robustness to noisy input data, or lack of scale of training data for deployment in real-world scenarios. At least emulating such cases in the simulated setting will make the obtained results for the proposed benchmark more convincing and representative. 


Minor comments:
- Can you define the bounds, or provide qualitative example-score pairs, for the task difficulty metrics defined in Section 3.1 to make the interpretation of Table 2 more intuitive?
- Although the proposed benchmark is more comprehensive, related work could benefit from a broader discussion of other real-world UAV, that can also act as benchmarks for more specific tasks such as ego-motion estimation or drone racing. e.g. :

Delmerico, J., Cieslewski, T., Rebecq, H., Faessler, M. and Scaramuzza, D., 2019, May. Are we ready for autonomous drone racing? the UZH-FPV drone racing dataset. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 6713-6719). IEEE.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. This benchmark has capability of 3D scenarios, classical methods and learning methods for planning while allowing sensory inputs in form of vision.
2. Three different scenarios with eight difficulty level has been presented.
3. Performance on different computing platforms have been shown.
summary: This paper introduces a comprehensive benchmark called FlightBench which implements and compares several learning-based methods for ego-vision-based navigation in order to compare them against optimization-based baselines. The paper also develops several assessment metrics, e.g. Traversability Obstruction, View Occlusion, and Angle Over Length.
weaknesses: 1. The overall paper's core contribution is unclear, i.e. whether it is on the technological side, creating virtual scenarios, etc.
2. It seems this paper has proposed three simulation scenarios with difficulty levels based on different evaluation metrics. However, I don't see any novelty or crucial research contribution from the ICLR perspective. In other words, there are no theoretical or experimental contributions. The paper appears as a system paper where multiple things are simply combined together.
3. Overall, I find this paper a sort of comparison between different planning methods and how they behave in the benchmark. However, It would be interesting to see how this benchmark is advantageous compared to the existing ones.