Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

## Abstract

Abstract Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest.
Most research has focused on ground-based agents, while UAV-based VLN remains relatively underexplored.
Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navigation tasks between ground and aerial environments.
To address these disparities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology.
To enable realistic UAV trajectory simulation in VLN tasks, we propose the OpenUAV platform, which features diverse environments, realistic flight control, and extensive algorithmic support.
We further construct a target-oriented VLN dataset consisting of approximately 12k trajectories on this platform, serving as the first dataset specifically designed for realistic UAV VLN tasks.
To tackle the challenges posed by complex aerial environments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help, which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks.
We also propose a UAV navigation LLM that, given multi-view images, task descriptions, and assistant instructions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajectory generation.
The evaluation results of our method significantly outperform the baseline models, while there remains a considerable gap between our results and those achieved by human operators, underscoring the challenge presented by the UAV-Need-Help task.
The project homepage can be accessed at https://prince687028.github.io/OpenUAV .

## 1 Introduction

Constructing embodied agents capable of understanding human commands remains a long-term objective in the field of artificial intelligence. Among these (Qi et al., ; Ku et al., ; Shridhar et al., ; Shen et al., ), visual-language navigation (VLN)—navigating to a target location based on language instructions and visual information—has garnered significant research interest. Current research in VLN focuses primarily on ground-based agents (Krantz et al., ; Blukis et al., ), while UAV-based VLN has received comparatively less attention. This area presents a wealth of application scenarios, and due to the notable differences in action space and observations between UAVs and ground-based agents, it represents a valuable area for research.

Figure: Figure 1: We propose a realistic UAV simulation platform and a novel UAV-Need-Help benchmark. The OpenUAV platform focuses on realistic UAV VLN tasks, integrating diverse environmental components, realistic flight simulations, and algorithmic support. The UAV-Need-Help benchmark introduces an assistant-guided UAV object search task, where the UAV navigates to a target object using object descriptions, environmental information, and guidance from assistants.
Refer to caption: x1.png

Recent UAV VLN benchmarks (Liu et al., ; Fan et al., ; Lee et al., ) typically adopt ground-based VLN settings, relying on a fixed set of discrete actions.
However, we argue that a significant gap exists between ground-based movement and UAV flight characteristics, along with the distinct operational environments, making a direct transfer of ground-based approaches to UAVs insufficient for precise aerial navigation.
The primary differences and challenges are manifested in two aspects:
1) *Mismatch in agent movement dynamics.*
Ground-based agents (Anderson et al., ; Yan et al., ; Shah & Levine, ) typically move along a horizontal plane, which makes it straightforward to plan navigation using discrete actions such as horizontal movement and rotation. In contrast, UAVs can operate freely in three-dimensional airspace.
As shown in Fig. 1, traditional methods attempt to define UAV movement using fixed action sets, which incorporate basic directions such as up, down, and horizontal movements. These approaches oversimplify UAV control and fail to capture realistic flight dynamics, as UAVs often perform maneuvers like pitching up, diving, and rolling simultaneously to achieve spatial movement. UAV trajectories are inherently continuous and difficult to decompose into discrete actions, leading to unrealistic navigation when such simplifications are applied.
2) *Disparity in navigation task complexity*.
UAVs frequently operate in diverse outdoor, open environments, where navigation paths are often long and complex.
Unlike ground-based agents (Anderson et al., ) that navigate using well-defined descriptions of target objects, UAVs must deal with obstructed views and shifting perspectives due to their high maneuverability. Therefore, relying solely on target descriptions is inadequate for precise localization and navigation in such complex, dynamic scenarios.
To address these disparities and associated challenges, we propose a UAV simulation platform OpenUAV, a target-oriented VLN dataset, a novel benchmark UAV-Need-Help, along with a UAV navigation LLM to create a more realistic framework for UAV vision-language navigation which supports continuous trajectories and complex scenarios.

To narrow the gap to realistic UAV VLN tasks, a crucial advancement lies in replacing fixed action sequences with continuous flight trajectories.
To facilitate this, we introduce OpenUAV, a platform
offering realistic environments, authentic flight simulation, and extensive algorithmic support.
As shown in Fig. 1, we utilize UE4’s realistic rendering capabilities, integrate 22 scenarios and 89 objects, and provide APIs for object placement and scenario configuration.
We integrate AirSim plugin
to translate trajectory sequences into continuous paths with realistic flight dynamics and support continuous flight control using real flight signals via remote controllers or designed APIs.
Building on the features above, we develop a parallel realistic trajectory collection framework and a closed-loop simulation framework, providing comprehensive algorithmic support for UAV tasks.

To mitigate the scarcity of realistic UAV trajectory data for VLN tasks,
We leverage the unique features of the OpenUAV platform to construct a target-oriented VLN dataset, which is the first realistic UAV VLN dataset to incorporate 6 degrees of freedom (DoF) motion, accurately capturing the complex flight dynamics of UAVs.
Human annotators conducted continuous flights on the OpenUAV platform, regularly recording UAV states during annotation and asynchronously collecting additional sensor data to obtain navigation trajectories. We utilize GPT-4 to generate target descriptions and follow up with manual quality checks to generate high-quality navigation instructions, which result in a total of approximately 12k trajectory-instruction pairs.

The high maneuverability of UAVs and the complex aerial environments pose significant challenges for UAV object search task relying solely on descriptive instructions. To support UAV to accomplish such tasks, we enhance UAV navigation capabilities by providing additional information and accordingly propose the UAV-Need-Help benchmark. As shown in Fig. 1, we establish an assistant that provides action guidance to UAVs in specific scenarios and categorize the assistant into three types based on the varying levels of guidance. The UAV performs the task based on the initial target description, environmental information, and instructions from the assistant.

To tackle the challenges presented by the UAV-Need-Help benchmark and enhance the efficiency of UAVs in object search tasks, we propose a UAV navigation LLM.
We leverage MLLM’s understanding and decision-making capabilities to produce hierarchical outputs for long-distance and fine-grained trajectories. Additionally, we propose a backtracking sampling-based data aggregation strategy to enhance environmental understanding and obstacle avoidance capabilities. Comprehensive closed-loop evaluations across various settings validate the effectiveness of our approach.

We hope that the proposed platform, benchmark, and methodology can promote research and development in VLN tasks based on continuous and realistic UAV trajectories, thereby facilitating the transfer of UAV VLN systems to real-world applications.

## 2 Related Work

Table: Table 1: Comparison of existing VLN datasets based on the DoF of navigation action space and dataset scale. Our UAV dataset incorporates 6 DoF trajectories for realistic VLN tasks. $N_{traj}$ and $N_{vocab}$ represent the number of collected trajectories and vocabulary size, respectively.

Embodied Simulator. Simulation platforms are crucial for intelligent systems research. For ground-based simulations, platforms like Habitat (Puig et al., ), Matterport3D (Chang et al., ) and Google Street View (Anguelov et al., ) have been pivotal simulated realistic indoor and outdoor scenes, significantly propelled research by improving data collection and algorithm evaluation in complex scenarios.
However, UAV simulation platforms
are still in their early stages of development. While some existing platforms (Lee et al., ; Liu et al., ; Fan et al., ), provide a foundation for UAV navigation, they still exhibit limitations in terms of realism, scalability, and various scenarios.
The xView platform (Lam et al., ) only offers remote sensing satellite images which are unsuitable for low-altitude navigation tasks.
Recently, some studies (Liu et al., ; Ma et al., ; Haley et al., ; Gill et al., ; Madaan et al., ; Alvey et al., ) have combined UE with AirSim to develop high-fidelity imaging platforms.
However, these platforms either suffer from low rendering precision or lack algorithmic support for continuous trajectory VLN tasks.
In contrast, our platform offers realistic environments, authentic flight simulation, and extensive algorithmic support, providing a foundation for realistic UAV VLN research.

Vision-Language Navigation Datasets. UAV VLN is an emerging task recently formed from the expansion of outdoor VLN tasks. Table 1 presents a comparison between representative VLN datasets.
Beginning with indoor navigation, R2R dataset (Anderson et al., )
establishes a foundation for grounded VLN. Inspiring works (Vasudevan et al., ; Zhu et al., ; Chen et al., ; Misra et al., ) such as TOUCHDOWN, LANI have employed verbal navigation instructions to address the challenges of outdoor long-range navigation.
Moving beyond ground-level navigation,
AVDN (Fan et al., ) collects aerial navigation trajectories with human dialogues. AerialVLN (Liu et al., ) and CityNav (Lee et al., ) propose VLN for UAVs and collect aerial trajectory datasets of discrete 4-DOF UAV action.
In previous datasets, UAV trajectories were collected by simply modifying the UAV’s position, which significantly deviates from real UAV flights.
Based on our platform, we collect a target-oriented realistic UAV VLN dataset with continuous 6-DOF trajectories to enhance the realism of UAV navigation tasks.

LLM-based Navigation Agent.
The application of LLM-based methodologies has led to significant advancements in developing navigation agents. LM-Nav (Shah et al., ) propose a goal-conditioned robotic navigation method that enables robust generalization to real-world outdoor environments without language-annotated data.
LMDrive (Shao et al., ) further progressed the field by introducing a language-guided, end-to-end autonomous driving framework that integrates multi-modal sensor data, facilitating effective human-robot interaction in challenging urban scenarios.
Some studies (Vemprala et al., ; Zhong et al., ) have utilized the zero-shot capabilities of LLMs to generate UAV navigation code, while (Lee et al., ) combined LLMs for coordinate optimization of drone target points, validating the basic capabilities of LLMs in drone navigation tasks.
To further expand the application scenarios for drones, we proposed a UAV navigation LLM composed of hierarchical trajectory generation models in solving the object search task.

## 3 OpenUAV Simulation Platform

As illustrated in Fig. 1, the OpenUAV simulation platform is a fully open-source platform devoted to realistic UAV VLN tasks, integrating three modules: environment composition, flight simulation, and algorithmic support to achieve comprehensive functionality.

### 3.1 Environment Composition

Diverse Scenario Resources. OpenUAV offers a wide range of scenarios and achieves high-fidelity visual effects through the advanced graphical rendering capabilities of UE4. We integrate high-quality scenarios from online repositories and urban scenarios from the CARLA simulator (Dosovitskiy et al., ), resulting in the OpenUAV platform featuring 22 distinct scenarios, including urban, rural, and natural landscapes, as detailed in Appendix A.
The platform can also simulate dynamic environments, such as vegetation swaying and variations in lighting. This further enhances the platform’s realism, reducing the gap when transitioning to real-world environments.

Tailored Object Assets. OpenUAV platform features a wide variety of standalone object assets, including humans, vehicles, animals, road signs, tables and other items suited for urban and natural environments.
Users can place objects in scenarios using different methods based on task requirements.
In addition to UE4’s built-in scenario editor, we support the automatic placement method at runtime.
The method is implemented through the OpenUAV API, where we have pre-defined feasible regions within each scenario and classified various object categories. Annotators can utilize the object placement API to select the object type and place it in appropriate and suitable areas.

### 3.2 Flight Simulation

Realistic UAV Flight Control. OpenUAV platform integrates the AirSim plugin to achieve realistic UAV flight simulation, enabling more precise flight control.
Our platform can utilize the flight control API to achieve physics-based UAV maneuvers with a 6 DoF trajectory representation.
The pose of the UAV at each time is represented as $P=\{x,y,z,\theta,\phi,\psi\}$, where $(x,y,z)$ represent the position coordinates, and $(\theta,\phi,\psi)$ denote the pitch, roll, and yaw angles respectively. The pose can be obtained at any time, allowing highly accurate simulation of UAV movement.

Configurable UAV Sensors. OpenUAV platform supports the simulation of various sensor payloads during UAV flight, including IMU, RGB and depth cameras, LiDARs and GPS.
The platform initially configures the UAV with RGB and depth cameras covering front, rear, left, right, and downward views, along with a LiDAR sensor set up with a 360-degree horizontal field of view.
Users can add necessary payloads and adjust detailed sensor configurations based on specific requirements, such as modifying image resolution to optimize response time.

Human Control Interface. Our platform offers multiple methods for humans to control UAVs, enabling continuous motion.
Firstly, we integrate with PX4 to support remote controller operation. Additionally, the OpenUAV platform offers two operation APIs for UAVs: manual and position modes. The manual mode operates similarly to remote controller control, while the position mode provides a more intuitive and straightforward way to operate.

### 3.3 Algorithmic Support

Data Collection Tools. OpenUAV platform has contained a data collection framework to tackle the challenge of limited UAV training data. Collecting data from multiple sensors requires a certain amount of time, which can affect the continuity of annotation. Therefore, we have implemented an asynchronous collection method
that initially gathers UAV attitude information at equal time intervals, followed by the data collection of sensors in the background.

Closed-loop Simulation. The platform provides an extended interface for the UAV navigation model, allowing for flexible integration of model outputs to control drone flight and real-time feedback of environmental information to the model.
Moreover, we have implemented a dataset aggregation (DAgger) method to enhance model training and proposed a backtracking policy. When the UAV encounters a collision, it is reverted to its previous position and uses teacher action to perform the next step, allowing it to recover from the collision and produce longer navigation trajectories.

Parallelization. OpenUAV platform employs an environment parallelization strategy, allowing multiple simulated environments to run concurrently to enhance the efficiency of data collection and closed-loop evaluation. With 8 NVIDIA A100 GPUs, the simulation of a single UAV achieves a performance boost of 16 times, reaching frame rates of between 160 and 1600 fps, depend on the amount of data captured by UAV payloads.

## 4 Target-Oriented Realistic UAV VLN Dataset

We construct a target-oriented realistic UAV VLN dataset using the proposed OpenUAV platform, which is the first dataset accurately capturing complex flight dynamics for UAV VLN tasks. As shown in Fig. 2 (a), we introduce this dataset in two aspects: data collection and data analysis.

### 4.1 Data Collection

Figure: Figure 2: Overview of our dataset construction and statistical analysis. (a) Data collection pipeline for generating high-quality target descriptions and realistic UAV trajectories. (b) - (e) Statistical analysis of the dataset, covering trajectory lengths, task distances, object categories, and dataset splits. In (e), UM and UO represent Unseen Map and Unseen Object, respectively.
Refer to caption: x2.png

Description Collection. In the UAV object search task, the target descriptions consist of three key components: 1) Target direction, indicating the relative position between the target and UAV’s initial pose; 2) Object description, detailing the visual features of the target; and 3) Environmental information, describing the surrounding spatial context.
During the description collection process, objects can be placed in the feasible regions of the scene using the OpenUAV object placement API. Object descriptions and environment information are then obtained by positioning a camera above the object to capture images from five different views (front, back, left, right, and down), followed by generating textual descriptions using GPT-4 (Achiam et al., ), with the prompts detailed in Appendix B. Human experts review and refine the generated content to maintain data quality, removing any inaccuracies or hallucinations.

Asynchronous Trajectory Collection. Human common sense in target searching provides valuable guidance for UAVs to learn effective search strategies. During the trajectory collection process, human experts utilize the human control interface of the OpenUAV platform to manually control the UAV according to the given instructions to search for the target. To minimize operational discontinuity caused by sensor data storage latency, only UAV states are recorded at fixed time intervals during the flight. After completing the trajectory collection, sensor data is acquired through the parallel data collection framework of OpenUAV. The trajectory recording terminates when the UAV approaches within 5 meters of the target, and any trajectory is discarded if a collision is detected. Ultimately, a total of 12,149 valid trajectories are obtained.

### 4.2 Data Analysis

Trajectory Analysis. As illustrated in Fig. 2 (b), the UAV-Need-Help dataset contains a total of 12,149 trajectories, where trajectories shorter than 250 meters are classified as easy, and those exceeding 250 meters are classified as hard. The diversity in trajectory lengths ensures the challenge and complexity of the tasks. Fig. 2 (c) illustrates that the distance to the targets ranges from 50 meters to 400 meters, representing the spatial scale of the environment.

Description Analysis. The most frequently occurring descriptions are illustrated in Fig. 2 (a), including building, tree, and car. These descriptions provide contextual information for UAVs, enhancing their capability to estimate object locations through visual cues and thereby accurately find the target. Fig. 2 (d) shows the target object set consists of 89 distinct categories for search tasks, encompassing vehicles, humans, animals and other objects.

Dataset Split. As shown in Fig. 2 (e), to comprehensively evaluate the model’s performance on both seen and unseen environments, and to analyze its ability to generalize to new maps and objects, we divide the dataset into four subsets: Train, Test Seen, Test Unseen Map, and Test Unseen Object. Each test subset is further divided into easy and hard categories, consistent with the criteria mentioned above. Specifically, the data distribution for each subset is as follows:

- •
Train - 9152 trajectories with 76 objects across 20
scenes as the training set.
- •
Test Seen - 1410 trajectories generated using objects and scenes seen in the training set.
- •
Test Unseen Map - 958 trajectories with 2 scenes unseen in the training set.
- •
Test Unseen Object - 629 trajectories with 13 objects unseen in the training set.

## 5 UAV-Need-Help Benchmark

### 5.1 Task Formulation

We propose an assistant-guided UAV object search task named UAV-Need-Help, where the UAV navigates to the target object following the target description, environment information and guidance from the assistant. Fig. 1 provides an illustration of the UAV-Need-Help task.

Formally, in each episode, the UAV starts at an initial position with posture $P_{0}$ and receives a target description $I$ that specifies the target direction, object features, and its surrounding environment. At each time step, the UAV obtains its state $S$ (position, posture, velocity), along with RGB images $R$ and depth images $D$ from five perspectives: front, left, right, rear, and below. An assistant monitors its status and provides additional instructions $I^{\prime}$ to suggest flight strategies when needed. In the closed-loop simulation, the UAV navigation model predicts a 6 DoF trajectory sequence based on $\{I,S,R,D\}$. Using the OpenUAV platform’s flight API, the UAV navigates to each predicted position with a 6 DoF attitude $\{x^{\prime},y^{\prime},z^{\prime},\theta^{\prime},\phi^{\prime},\psi^{\prime}\}$ while adhering to its flight dynamics and updating its observations. The task is successful if the UAV lands within a 20-meter radius of the target.

### 5.2 Assistant Mechanism

As previously mentioned, the inherent complexity and dynamic nature of aerial environments, with obstructed views and shifting perspectives, make basic target descriptions insufficient for UAV object searching. Thus, we introduce assistants to aid UAVs in this task, defining three distinct tiers based on the level of guidance provided, as shown in Fig. 3 (a).

- •
*L1* assistant provides high-frequency guidance closely aligned with the ground truth (GT) trajectory. It continuously calculates the trajectory point in the oracle path that is closest to the UAV’s current position, then identifies the appropriate actions based on the UAV’s orientation and the direction toward the nearest GT point. These actions may include cruising, turning, or landing. This setting ensures that the UAV stays on the correct path at all times.
- •
*L2* assistant intervenes when the UAV encounters difficulties, providing low-frequency corrections to steer it back toward the ground truth trajectory. It activates only when a collision risk is identified through depth maps or when the UAV deviates too far from the GT path, offering corrective actions to guide the UAV back to the desired trajectory.
- •
*L3* assistant only provides obstacle avoidance assistance when the UAV is in a hazardous scenario. It determines the distance between the UAV’s current position and obstacles using depth maps, and when proximity to an obstacle is detected, it issues avoidance commands to prevent collisions.

## 6 UAV Navigation LLM

Figure: Figure 3: Overview of the assistant mechanism and UAV Navigation LLM framework. (a) Three different assistant settings for providing varying levels of guidance. (b) UAV Navigation LLM framework: The instructions, multi-view images, and a learnable query are encoded into the MLLM, where the query extracts features to predict a long-distance target pose. This pose is then refined using front-view inputs by a trajectory completion model to generate fine-grained trajectories.
Refer to caption: x3.png

UAV navigation LLM is a multimodal LLM capable of handling various input types, including images and texts. We first tokenize the multi-model input, where vision tokens are aligned with the language space.
These tokens are then concatenated and fed into the LLM, with Vicuna-7B serving as the base model. Utilizing the features obtained by LLM, a hierarchical trajectory decoder generates both the UAV’s next target pose $P_{target}$ and refined pose sequence $\{P_{traj}^{0},P_{traj}^{1}...P_{traj}^{N}\}$, where $N$ represents the length of this trajectory. The overview architecture is shown in Fig. 3 (b). Additionally, we have designed a backtracking sampling-based data aggregation to expand the dataset, thereby enhancing models’ obstacle avoidance capabilities in complex scenarios.

### 6.1 Hierarchical Trajectory Generation

Multimodal Tokenization.
Given
the task description $I_{task}$, assistant instructions $I_{instr}$,
we tokenize $I_{instr}$ and $I_{task}$ using a pre-trained language tokenizer (Li et al., ) to obtain corresponding tokens.
For the multi-view images, we utilize a EVA-CLIP (Sun et al., ) and a Q-former structure to extract the visual features. Each image is converted into a set of tokens, consisting of 1 context token that captures global features and 16 content tokens that represent local details through grid pooling.
Finally, we concatenate the three types of tokens: $T_{img}$ for image tokens, $T_{task}$ for task description tokens, and $T_{instr}$ for instruction tokens, to construct the multi-modal input token sequence, denoted as $T_{input}=<T_{img},T_{task},T_{instr}>$.

Hierarchical Trajectory Decoder.
The decoder is the core component that translates the input into the UAV’s next step target pose. To enable complex trajectory planning within dynamic environments, we employ a hierarchical structure with two levels: a high-level MLLM-based trajectory decoder and a fine-grained path decoder.

The MLLM-based trajectory decoder utilizes a special learnable trajectory token $T_{wp}$ as input to the LLM, which helps to extract trajectory-specific features $Q_{wp}$. These features are fed into a multilayer perceptron (MLP) to decode the target pose $P_{target}$, allowing the model to plan UAV trajectories based on environmental data and detailed language instructions.

The fine-grained path decoder is introduced to generate the trajectory details and enhance navigation efficiency. It takes the front-view visual token $T_{img}^{f}$ and combines it with the processed pose features derived from the MLLM-based decoder. The concatenated features are passed through an MLP to produce fine-grained trajectories $P_{traj}$. The overall process is defined as follows:

$$ $\displaystyle P_{traj}$ $\displaystyle=\textsc{MLP}\left(T_{img}^{f},f_{target}\left(\textsc{MLLM}(<T_{ input},Q_{wp}>)\right)\right),$ $$

where $f_{target}$ represents the alignment process of target poses from the MLLM output with the dimensions of visual tokens. To further improve UAV landing accuracy near the target, a Grounding DINO model (Liu et al., ) is utilized for detection. Once the target object is identified in multi-view images, the navigator initiates the landing process.

### 6.2 Backtracking Sampling-based Data Aggregation

We implement a DAgger (Ross et al., ) module applied to continuous and realistic UAV trajectories. When collecting data using DAgger in closed-loop simulation, this module samples from both the predicted trajectories generated by the model and the ground-truth trajectories provided by a teacher model to create paths for VLN tasks.
Considering that collisions often occur when simulating UAV flight, it becomes challenging to collect complete trajectories or those containing sufficient obstacle avoidance knowledge.
Thus, we propose a backtracking sampling mechanism, where, if the UAV takes an action based on the model’s output that results in a collision, it is reverted to its state from two frames earlier, restoring its pose, velocity, and other attributes.
It will then follow the trajectory given by the teacher model, helping the UAV to avoid collisions and stay on track. In this way, we can obtain more successful obstacle-avoidance trajectories, thereby enhancing the navigation capability of the model.

## 7 Experiments

### 7.1 Experimental Setup

Evaluation Metrics. We adapt evaluation metrics commonly used in VLN (Anderson et al., ; Krantz et al., ), including success rate (SR), oracle success rate (OSR), success weighted by path length (SPL), and navigation error (NE).
SR measures the percentage of tasks where the UAV successfully reaches the target.
OSR measures whether the UAV reaches any location along the optimal trajectory, even if it doesn’t exactly reach the final destination.
SPL evaluates both the success rate and the efficiency of the path taken, rewarding shorter, more optimal paths.
NE calculates the average distance between the UAV’s final position and the target.

Comparison Baselines. We compare the following baselines against our method on the UAV-Need-Help task.
1) Random: The UAV randomly selects trajectory poses without any structured planning or guidance. This method is employed to illustrate the extent of the solution space.
2) Fixed Action: The UAV maps the assistant’s instructions into predefined fixed actions. For example, cruise means moving forward by 5 meters, while turn left results in a 30-degree turn followed by a 5-meter forward movement.
3) Cross-Modal Attention (CMA): The CMA model (Anderson et al., ) is commonly used in grounded VLN tasks and employs a bi-directional LSTM to simultaneously process image inputs and instruction comprehension. To adapt to our task setting, we modify its recurrent predictor to output a set of trajectories instead of traditional navigation actions.

### 7.2 Quantitative Result

Table: Table 2: Results on Test Seen Set across different assistant levels. DA refers to a model trained using backtracking sampling-based data aggregation.

Comparison with Baselines. Table 2 shows that our method outperforms across all metrics at different difficulty levels on the test seen set. At various assistant levels, the SR metric improves by an average of 5% over the CMA model. This demonstrates that our MLLM-based hierarchical trajectory generation approach enhances scene understanding and produces more accurate and adaptable pose sequences, ultimately improving decision-making and overall performance.

Table: Table 3: Generalization capabilities across different test sets with L1 assistant, where UO and UM represent the Test Unseen Object Set and the Test Unseen Map Set, respectively.

The classical baseline model CMA faces several challenges due to limitations in model size and task complexity and performs poorly on multiple metrics.
The Random and Fixed methods struggle to complete the tasks, demonstrating that achieving the target search tasks without understanding instructions and visual information is extremely difficult.
We further evaluate our model trained with backtracking sampling-based data aggregation, which shows an improvement in SR compared to the original model, indicating this method has enhanced the UAV’s navigation capability.

Additionally, we evaluate the performance of humans operating UAVs with guidance from L1 assistant.
As shown in Table 2, humans achieve a high success rate but sometimes take longer paths. This suggests that while humans make correct decisions, they may choose more cautious or exploratory routes, leading to slightly less efficient path planning.

Comparison under Different Assistant Level. Under the continuous guidance of the L1 assistant, our method achieves the highest success rate.
As the assistant level increases, the UAV agent needs to rely more on its navigation planning capabilities to complete tasks. The results show that both our method and the CMA method experience a decline in success rate and other metrics at higher levels of assistance, demonstrating that long-term VLN for UAVs is extremely challenging.

Figure: Figure 4: Visualization of object search results of our method. First two rows demonstrate our UAV successfully follows the instruction. Notably, the third to fifth images depict the drone executing a turning maneuver, resulting in a change in the drone’s perspective. The third row illustrates a failed example, depicting a collision with trees in a forest scenario.
Refer to caption: x4.png

Generalization on Unseen Cases. Table 3 shows the performance on unseen datasets, highlighting our method’s superior zero-shot capability and its adaptability to new scenarios.
Our method demonstrates a slightly higher success rate in the unseen object test set than in the seen set. This phenomenon can be attributed to the inherent generalization capabilities of the Grounding DINO module (Liu et al., ) used in our approach and the relatively simpler nature of the data. We have still partitioned this dataset for future research aimed at models that do not require external detectors.
In the unseen scene test set, results of the fixed action method reflect the complexity of the environment through its low success rate. Our method shows a noticeable decline in the unseen scene test set, yet still outperforms other methods.

Figure: Table 4: Performance scalability with varying amounts of training data.

Performance Scalability. Table 7.2 reports the model’s performance across various quantities of training data with L1 assistant. The results show that as the data volume increases, the model performs better, indicating that a larger and more diverse dataset could enhance the model’s understanding and decision-making capabilities.

### 7.3 Qualitative Result

Fig. 4 presents two examples evaluated in OpenUAV platform.
The first two rows show the UAV successfully following the instructions, maneuvering through the buildings, and ultimately locating a yellow vehicle. During this process, the UAV experienced camera view shifts due to its attitude changes, highlighting the realistic fidelity of our platform. In contrast, the third row demonstrates a collision caused by insufficient altitude while navigating through a forested area, showcasing the challenges posed by complex environments.
Additional results can be found in Appendix D.

## 8 Conclusion

We address the challenges of realistic UAV VLN from three aspects: platform, benchmark, and methodology. To achieve this, we develop the OpenUAV platform, which provides realistic environments, flight simulation, and comprehensive algorithmic support. We also construct a target-oriented realistic UAV VLN dataset and propose the UAV-Need-Help benchmark, which provides assistance to guide UAVs through complex VLN scenarios. Additionally, we propose a UAV navigation LLM along with a backtracking sampling-based data augmentation strategy, which together effectively enhance the performance of realistic trajectory-based VLN tasks. Our contributions establish a unified framework for realistic UAV VLN research, making significant strides toward bridging the gap between simulation and real-world UAV navigation applications.
Furthermore, there are two promising directions for future research in realistic UAV VLN tasks. The first is to enhance the autonomous navigation capabilities of UAVs, enabling them to operate effectively in complex environments with minimal guidance. The second is to improve the transferability from UAV simulation to real-world deployment, facilitating the application of UAVs in real-world scenarios.

## Appendix A Platform Scenarios

We collected a total of 22 high-fidelity scenarios. In addition to those shown in Fig. 5, we have also included 11 urban scenes transferred from the CARLA simulator.

Figure: Figure 5: Platform Scenarios Overview: We present a selection of scenarios, respectively depicting city, town, forest, and desert environments.
Refer to caption: x5.png

## Appendix B Prompt Engineering in Description Collection

As shown in Fig. 6, we use the following prompt to collect object descriptions (Section  4.1).
The prompt integrates information from multiple camera views to provide a comprehensive representation of the target object’s visual characteristics. The generated instruction incorporates detailed descriptions of the surrounding environment’s colors and shapes to facilitate accurate visual identification. Furthermore, it specifies the spatial relationship between the target and nearby objects, enabling precise localization of the target within its context.

Figure: Figure 6: The instruction includes the creation of target direction, object descriptions, and environmental information. The target direction is determined based on the coordinates of the UAV’s start and end positions, while the object descriptions and environmental information are generated by GPT-4 using the provided prompts, incorporating the top, front, rear, left, and right views and the object’s name.
Refer to caption: x6.png

## Appendix C Experimental Details

Implement Detail. We include the implementation details necessary for reproducing results. In general, we adopted a training strategy similar to that described in Li et al. (). During the training of the MLLM, we freeze most of the model’s parameters and only compute gradients on the visual projector, trajectory prediction head, and LoRA layers. These trainable parameters make up just 4% of the total model parameters and significantly reduce computational costs while maintaining the model’s core capabilities.
We supervise the predicted 3D angles using cosine similarity loss and apply L1 loss between the predicted waypoints and the ground truth. Similarly, for the trajectory completion model, we also freeze the parameters of the vision encoder. The MLP trajectory prediction layer is trained using L1 loss between the predicted trajectory and the labelled trajectory.

Figure: Table 5: Ablation on model structure.

The MLLM model is trained on 8 NVIDIA A100 GPUs with a batch size of 128 for 2 epochs, while the fine-grained model and CMA model are trained with a batch size of 128 for 10 epochs.
We use Adam optimizer with a one-cycle learning rate decay schedule to train all models, where we set the maximum learning rate to 5e-4.

Ablation Study. To validate the effectiveness of each part, we
summarize experiments results in Table C, where $L_{cos}$ indicates the cosine loss of poses, $Q_{wp}$ indicates the learnable query.
The ablation of model architecture highlights the critical role of LoRA. The exclusion of LoRA resulted in an 8% decrease in OSR and 4% in SR.
Removing the cosine loss and the learnable query also leads to a decline in performance, but the impact is relatively smaller. The complete model performs best across all metrics, validating the effectiveness of each component in enhancing model performance.

## Appendix D Additional Qualitative Results

Here, we provide more visualization results of our method in closed-loop evaluation.

Figure: Figure 7: More examples of the UAV object search task using our method illustrate longer trajectories and improved navigation performance across various scenarios.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: 1. The paper address realistic flight dynamics in UAV navigation instead of using oversimplified discrete actions.
2. Provides a comprehensive open-source simulation platform that combines realistic environments, flight physics, and algorithmic support.
3. Creates the first large-scale UAV navigation dataset (12k trajectories) that captures true 6-DOF flight dynamics.
4. Introduces a novel assistance mechanism with three levels of guidance, making the navigation task more adaptable and learnable.
5. Proposes a practical hierarchical architecture combining language models with trajectory planning for complex navigation.
6. Includes thorough experimental validation across different difficulty levels and scenarios, with clear comparisons to human performance.
7. Shows strong generalization ability on unseen objects, though with expected performance drops on unseen environments.
summary: This paper makes three key contributions to advance realistic UAV vision-language navigation. First, they introduce OpenUAV, a simulation platform that properly models realistic drone flight dynamics with 6 degrees of freedom, rather than using simplified discrete movements like previous work. Second, they create what appears to be the first UAV navigation dataset (about 12k trajectories) that incorporates realistic flight physics. Third, they propose a novel "UAV-Need-Help" benchmark where an AI assistant provides varying levels of guidance, along with a hierarchical navigation model using large language models.
The approach significantly outperforms baselines in experiments, though still falls well short of human performance. What makes this work valuable is how it addresses fundamental limitations in current UAV navigation research by developing tools and datasets that better match reality, particularly through the inclusion of realistic flight dynamics. While substantial work remains before real-world deployment would be feasible, this paper lays important groundwork and provides valuable resources for future research in autonomous drone navigation.
weaknesses: 1. The dataset size (12k trajectories) is relatively small compared to other vision-language datasets, which might limit model learning.
Lacks detailed comparison with real drone flight data to validate how well their simulation matches actual flight dynamics.
2. No discussion of computational requirements or real-time performance, which is crucial for practical drone navigation.
3. Success rate metrics (around 16-17%) are still quite low even with the highest level of assistance.
4. Missing ablation studies on different environmental conditions and their impact on navigation performance.
5. No discussion of safety mechanisms or fallback strategies when the navigation system fails.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Real-world Deployment:
How well do the models and methods developed on the OpenUAV platform perform in real-world UAV navigation tasks? Has there been any experimental validation involving physical UAVs? Although the simulation platform is quite realistic, it remains unclear to what extent the models trained on this platform can generalize to real-world UAV tasks. A comparison with experiments using real UAVs is lacking.

2. Autonomy and Reliance on Assistance:
The UAV-Need-Help benchmark seems overly reliant on the assistance mechanism, which could reduce the autonomy of the UAV. It is unclear how the UAV Navigation LLM would perform without such assistance. Can the UAV autonomously recover from navigation failures in the absence of external help, particularly when faced with obstacles?

3. Efficiency and Real-time Performance:
Considering the complexity of the LLM-based navigation model, how feasible is the real-time deployment of such models on resource-constrained UAVs? Is there any analysis of the computational requirements and latency when running these models on actual drones?

4. Ablation Studies:
Could more insights be provided on the contribution of individual components (e.g., assistant guidance, hierarchical trajectory generation, sensor configurations)? How would the model's performance change if fewer sensors were used, or if the advanced trajectory decoder based on MLLMs were removed?

5. Lack of Comparison with Ground-based VLN:
While the paper highlights the differences between ground-based and UAV navigation, there is a lack of direct comparison with existing ground-based VLN methods(Seq2Seq[1], DUET[2], NavGPT2[3]). It is unclear what unique challenges the proposed framework addresses. Specifically, what are the differences between ground-based VLN methods and the UAV Navigation LLM? Could ground-based VLN methods be applied to OpenUAV, and if so, how would they perform?

6. Extensibility to Other Tasks
Besides the object search task, can the proposed platform and dataset be generalized to other UAV tasks such as mapping, exploration, or surveillance? Would the dataset require significant modifications to accommodate these tasks?

[1]Anderson, Peter, et al. "Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[2]Chen, Shizhe, et al. "Think global, act local: Dual-scale graph transformer for vision-and-language navigation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[3]Zhou, Gengze, et al. "Navgpt-2: Unleashing navigational reasoning capability for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.
soundness: 3
strengths: 1. Comprehensive data set: The author contributes a large-scale goal-oriented UAV VLN data set, including 6-degree-of-freedom flight trajectories, providing a large number of training and evaluation resources for the research community.
2. Real UAV simulation: The OpenUAV platform significantly improves the authenticity of UAV navigation tasks, supports continuous flight trajectories, diverse environments and realistic flight dynamics, and solves the limitations of previous UAV VLN research.
3. Innovative benchmark task: The UAV-Need-Help benchmark introduces a challenge of practical significance, through different levels of help mechanisms, to cope with the challenges caused by the inherent dynamics of the aerial environment, obstructed views and changes in perspective.
4. Hierarchical trajectory generation: The proposed UAV navigation large model (LLM) combines long-range trajectory prediction and fine-grained control, and can handle the hierarchical complexity in UAV navigation tasks.
5. Parallelization and scalability: The platform supports efficient data collection and evaluation in parallel environments, which is crucial for large-scale UAV research.
summary: This paper proposes a new framework for UAV navigation, focusing on the different challenges of aerial environments and ground-based agent navigation. 
A realistic UAV simulation platform, OpenUAV, is proposed, which supports continuous six-degree-of-freedom (6-DoF) trajectories, realistic flight dynamics, and diverse environmental scenarios. The platform integrates the AirSim plugin and Unreal Engine 4 (UE4) to achieve high-fidelity simulation.

UAV-Need-Help Benchmark: A new benchmark task introduces an assistant-guided UAV target search task, where the assistant provides different degrees of navigation guidance to the UAV to help it cope with complex navigation tasks.

UAV Navigation Large Model (LLM): A multimodal model is proposed that can process visual and textual information simultaneously and generate hierarchical navigation trajectories to address the complexity of UAV navigation in three-dimensional space. The platform, dataset, and method aim to narrow the gap between simulated UAV navigation tasks and practical applications, and provide more realistic and challenging tasks for UAV VLN research.
weaknesses: Please see the Questions section for detailed improvement suggestions and questions.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: I am not an expert in the field of UAV but after looking at the related work it is still a bit unclear for me what the main contributions are and how it differentiates to existing work. It would be good if the authors could make it clearer what already exists and what is new. Secondly it would be really helpful to include more ablations on the importance of continuous actions or hierarchical predictions.

Minor suggestions/ questions:
1. Make sure that the abbreviations are introduced properly. E.g. line 016 is the first time you mention UAV, it would be good to have the long version when first using the short version.
2. Human baseline: I would be curious how well a human performs in the L3 setting. Is the human baseline with a person who tried it several times and is trained or is it a “zero-shot” baseline?
soundness: 3
strengths: S.1. Complete set to research UAVs. The paper provides a platform/ simulation environment, dataset, benchmark and a baseline model. This opens up a lot of possibilities for the community to do research in this field.

S.2. Baselines: Having multiple baselines for a new benchmark is important. I also appreciate the human baseline.

S.3. Hierarchical prediction: The concept of hierarchical predictions is interesting and makes sense. (But missing ablation to show that hierarchical is superior to directly predicting the fine-grained trajectory).
summary: The paper proposes a new UAV platform with several integrated scenes, continous action space and 6 DoF instead of previously 4. In addition they provide a VLN dataset together with a benchmark “UAV-Need-Help” together with an assistant that can provide guidance in three different levels. Lastly they have a model based on a LLM with a hirarchical strucutre that first predicts the target pose and then predicts the fine-grained path trajectory. The authors evaluate their model closed-loop and show qualitative results of the language predictions of the LLM.
weaknesses: **W.1. Comparison to existing work**

1. The works seems to be quite similar to [AerialVLN](https://arxiv.org/abs/2308.06735) or CityNav. One difference mentioned is the DoF and the use of discrete actions. The authors of AerialVLN state “Although the simulator **supports flying towards any given direction and speed/distance**, we consider the eight most common low-level actions in UAVs:Move Forward, Turn Left, Turn Right, Ascend, Descend, Move Left, Move Right and Stop.”, which means in my understanding that they do support continuous actions. There dataset also includes language instructions. So I was wondering why this new simulator and dataset is needed and can’t be build on top of the existing one? It would be great if the authors could highlight unique features or improvements their simulator offers, and explain why building on existing platforms was not sufficient for their research goals.
2. Difference to AirSim: AirSim does provide an API and a human control interface supporting PX4. In the paper it sounds like this is a contribution of the new platform OpenUAV (e.g. line 212 ff.). It would be good to make it more clear in the whole paper what is used from existing works and which parts are new in this paper.
3. Difference to related work that combines UE + AirSIM: The authors state that several works “have combined UE with AirSim to develop high-fidelity imaging platforms. However, these platforms either suffer from low rendering precision or lack algorithmic support for continuous trajectory VLN tasks”. (line 149) Since the authors also combine UE with AirSim, it would be great if the differences to the existing works could be described in more detail. 

**W.2. Details of descriptions/ Clarity**

1. Naming: There is already an OpenUAV platform (see[1]). It would be good to use an unique naming and also elaborate the differences.
2. Scenarios: It would be great to have more details of the available scenarios. For me it is not clear what a scenario means in this context? Is it a different environment (e.g. forest vs. city) or is it similar to CARLA a specific event happening (see https://leaderboard.carla.org/scenarios/)? What are the 22 distinct scenarios available in this simulator? If the authors could provide a list or table of the 22 scenarios, including a brief description of each it would help a lot.
3. Dataset splits: Figure 2 is great to get a rough overview of the objects but a detailed overview in the supplementary with all available objects and scenes and which are used for which split would be helpful.
4. Figure 3 (minor): In the image it seems that the front view image is input to the Trajectory completion module but in the text the authors say “it takes the front-view visual token T_img” (line 386). It would be good to have this clearer in the figure.
5. DAgger: What exactly is the teacher model? Can you provide more details?

**W.3. Evaluation/ Benchmark**

1. As I understood for all results an oracle path is necessary to generate the assistants. An additional mode without any assistants would be good to judge how hard the task is and in an ultimate fully autonomous setting this would be the wished setting I assume. The authors state “the inherent complexity and dynamic nature of aerial environments, with obstructed views and shifting perspectives, make basic target descriptions insufficient for UAV” (line 323). I am wondering if temporal input and a memory could help for this making the assistants unnecessary?
2. The authors say “UAV trajectories are inherently continuous and difficult to decompose into discrete actions, leading to unrealistic navigation when such simplifications are applied” (line 081). They use (different to previous works) continuous action space. To validate this statement it would be good to have a discrete baseline and qualitative example of the shortcomings (which are solved by the continuous version).

[1] H. Anand *et al*., "OpenUAV Cloud Testbed: a Collaborative Design Studio for Field Robotics," *2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)*

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - Why does the proposed method rely solely on the current observation? How does the model align visual history with the instructions? Additionally, the paper notes the use of `Llama-vid, Vicuna-7B, and EVA-CLIP`, similar to components used in another VLN paper [1]. A discussion comparing these approaches would be beneficial.

- In Tables 2 and 3, the OSR is significantly higher than the SR (almost double). Given the definition of OSR, methods can achieve high OSR by extensively exploring the environment. A discussion or additional experiments are needed to address this observation.

- Why not consider using existing LLM-based methods, such as [2], which employ waypoints in a manner similar to the proposed approach?

[1] Navid: Video-based VLM plans the next step for vision-and-language navigation

[2] NavGPT: Explicit reasoning in vision-and-language navigation with large language models
soundness: 3
strengths: This paper is highly comprehensive, including a platform, benchmark and method technique for the VLN task in the domain of UAV. The authors have clearly significant efforts in completing this project.  This benchmark could help researchers investigate realistic UAV control under the task of VLN. Besides, the proposed LLM-based navigation framework shows strong performance among the baseline methods.
summary: This paper introduces a UAV-based vision-and-language navigation benchmark built on the CARLA simulator. The simulator offers realistic environments and supports high-DoF control for UAV flight. In addition, the benchmark provides ground-truth human-controlled trajectories, trajectory descriptions summarized by a vision-language model (VLM), and assistive instructions for navigation evaluation. The paper also presents a vision-language model as a baseline for UAV-based vision-and-language navigation, achieving state-of-the-art performance on the proposed benchmark.
weaknesses: - This paper includes extensive content on the platform, benchmark, and method, but lacks many important details. For instance, `L398` mentions leveraging DAgger in a closed-loop simulation, yet no further experiments are provided to demonstrate its effectiveness.

- In my view, the L1 assistant (guiding the UAV to the oracle path) provides an oracle instruction by directly issuing ground truth commands, which should significantly enhance navigation performance. However, the experiment results show only a minor increase in SR (such as 16.1 in the full set). 

- The visual encoding strategy does not account for historical information, as the navigation method relies solely on the latest frame `L373`, resulting in only 16+1 tokens for encoding visual data. This design neglects navigation history, raising the question of how navigation history can align effectively with the instructions.

- The paper lacks video results to illustrate the ground truth trajectory and the method's performance on this platform. Such visual demonstrations would provide a more intuitive understanding of this benchmark.

Meta-review: The paper targets UAV-based vision-language navigation, recognizing that drones move in six degrees of freedom and therefore require continuous control, unlike ground-based robots. It presents the OpenUAV simulation platform, which offers realistic flight physics and varied environments. A dataset of around 12,000 drone trajectories is introduced, designed specifically for UAV navigation using language instructions. The authors also propose the UAV-Need-Help benchmark, which uses different levels of assistance to guide UAVs more effectively in challenging scenarios. To handle these tasks, they develop a hierarchical large language model that processes both images and text, then generates high-level navigation goals and detailed flight paths.

Strengths of the Paper:

-- Realistic UAV Flight Dynamics: By implementing full 6-DoF movement and continuous control, the paper tackles the limitations of simplified or discrete motion in previous UAV navigation research.

-- Comprehensive Toolkit: The authors provide a cohesive suite—including the OpenUAV platform, a large-scale dataset, and a structured benchmark—that can accelerate progress in UAV VLN research.

-- Hierarchical Model Design: Their hierarchical approach leverages LLM capabilities for high-level planning and detailed trajectory generation, demonstrating strong performance improvements compared to baseline methods.

Weaknesses of the Paper:

-- Limited Real-World Validation: There is no direct comparison with real-flight data or physical drones to confirm that the simulated environment’s flight dynamics transfer effectively to real-world conditions.

-- Insufficient Ablation and Comparison: While the authors mention a hierarchical LLM-based approach, the paper lacks thorough ablation studies (e.g., the impact of removing hierarchical layers, reliance on assistance levels) and deeper comparisons with related UAV VLN platforms or ground-based VLN methods.

-- Scalability and Real-Time Concerns: The paper does not adequately discuss the computational overhead, latency, or hardware constraints that might limit real-time deployment, an essential factor for practical UAV applications.

Overall, the paper provides a substantial contribution to UAV vision-language navigation research, and the reviewers all agree on accepting this paper. The Area Chair recommends accepting the paper.

Decision: Accept (Poster)