Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology
Authors: Xiangyu Wang
Donglin Yang
Ziqin Wang
Hohin Kwan
Jinyu Chen
Wenjun Wu
Hongsheng Li
Yue Liao
Si Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Unmanned Aerial Vehicle
Drone
Vision-Language Navigation
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a UAV simulation platform, an assistant-guided realistic UAV VLN benchmark, and an MLLM-based method to address the challenges in realistic UAV vision-language navigation.

Abstract: Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest. Most research has focused on ground-based agents, while UAV-based VLN remains relatively underexplored. Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navigation tasks between ground and aerial environments. To address these disparities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology. To enable realistic UAV trajectory simulation in VLN tasks, we propose the OpenUAV platform,  which features diverse environments, realistic flight control, and extensive algorithmic support. We further construct a target-oriented VLN dataset consisting of approximately 12k trajectories on this platform, serving as the first dataset specifically designed for realistic UAV VLN tasks. To tackle the challenges posed by complex aerial environments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help, which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks. We also propose a UAV navigation LLM that, given multi-view images, task descriptions, and assistant instructions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajectory generation. The evaluation results of our method significantly outperform the baseline models, while there remains a considerable gap between our results and those achieved by human operators, underscoring the challenge presented by the UAV-Need-Help task.

Published as a conference paper at ICLR 2025
TOWARDS REALISTIC UAV VISION-LANGUAGE NAVI- GATION: PLATFORM, BENCHMARK, AND METHODOL- OGY
Xiangyu Wang"', Donglin Yang"", Ziqin Wang"', Hohin Kwan', Jinyu Chen',
Wenjun Wu², Hongsheng Li Yue Liao Liu
1 Institute of Artificial Intelligence, Beihang University.
2 Hangzhou International Innovation Institute of Beihang University,
3 MMLab, CUHK, Centre for Perceptual and Interactive Intelligence
[wangxiangyu081 yangdongl in, liusi]
(alusi@bbuul aeuuuaa.eee. cn, liaoyue. ai@gmail com
ABSTRACT
Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest. Most research has focused on ground- agents, while Unmanned Aerial Vehicle (UAV)-based VLN remains relatively undercxplored. Recent efforts in UAV VLN predominantly adopt ground- groueddbbsedd based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navi- gation tasks between ground and aerial environments. To address these dispar- ities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology. To enable realistic UAV trajectory simulation in VLN tasks, we propose the TRAVEL platform, which features diverse environ- ments, realistic flight control, and extensive algorithmic support. We further construct target-oriented VLN dataset consisting of approximately 12k trajec- tories on this platform, serving as the first dataset specifically designed for real- istic UAV VLN tasks. To tackle the challenges posed by complex aerial envi- ronments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help. which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks. We also propose a UAV navi- gation LLM that, given multi-view images, task deseriptions, and assistant in- structions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajec- tory generation. The evaluation results of our method significantly outperform the baseline models, while there remains considerable gap between re- sults and those achieved by human operators, underscoring the challenge pre- sented by the UAV-Need-Help task. The project homepage can be accessed pttps://prince6371 github. 1o/Trave]
INTRODUCTION
Constructing embodied agents capable of understanding human commands remains a long-term objective in the field of artificial intelligence. Among these (Qi et al. 2020 Ku et al. [020 Shridhar jet al., 2020) Shen et al. 2021), visual-language navigation (VLN)- navigating to a target location based on language instructions and visual information has garnered significant research interest.
Current research in VLN focuses primarily on soounddbased agents (Krantz et al.] 2020 Blukis
et al. 2018), while bavebased VLN has received comparatively less attention. This area presents wealth of application scenarios, and due to the notable differences in action space and observations between UAVs and ground-based agents, it represents valuable area for research.
"Equal contribution.
"oorrsponniig author.

---

Published as a conference paper at ICLR 2025
TRAVEL Platform
UAV-Need-Help Benchmark
Chooed-leup Simulatiiu
Horaa Cantrol Interface
larget is person woaring brown jacket and pants. heartyy there a extansing pool and four chaire Pleace control the drane and find the sarget.
togee lecatel front right the drone
Asynchroneus Data Collection
Pardilleed Smadakke
Configurable Seasors
Tragenkcy Eeple
Reyenck
Ohjects
Realack Trajectory Support
Traditionll Fised Action
Ours Healistie Trajectory
Uarcal Engine 4 Sii + User-friendl API
Eegi
Figure I: We propose a realistic UAV simulation platform and a novel UAV-Need-Hood Help benchmark. The TRAVEL platform focuses on realistic UAV VLN tasks, integrating diverse environmental com- ponents, realistic flight simulations, and algorithmic support. The UAV-Need -elpp benchmark in- troduces an assistant-guide UAV object search task, where the UAV navigates to target object using object descriptions, environmental information, and guidance from assistants.
Recent UAV VIN benchmarks (L.iu et al. 2023b Fan et al., 2022 Lee et al. 2024) typically adopt ground-based VLN settings, relying on a fixed set discrete actions. However, we argue that a sig- nificant gap exists between bround-based movement and UAV flight characteristics, along with the distinct operational environments, making a direct transfer of ground-based approaches to UAVs in- sufficient for precise aerial navigation. The primary differences and challenges are manifested in two aspects: 1) Mismatch in agent movement dynamics. Ground-baad -asund agents (Anderson et al. 2018 Yan et al. 2019 Shah & Levine 2022) typically move along horizontal plane, which makes straightforward to plan navigation using discrete actions such as horizontal movement and rotation. In contrast, UAVs can operate freely in diree-iimessional airspace. As shown in Fig. I traditional methods attempt to define UAV movement using fixed action sets, which incorporate basic directions such as up. down, and horizontal movements. These approaches oversimplify UAV control and fail to capture realistic flight dynamics, as UAVs often perform maneuvers like pitching up, diving, and rolling simultaneously to achieve spatial movement. UAV trajectories are inherently continuous and difficult to decompose into discrete actions, leading to unrealistic navigation when such simplifica- tions are applied. 2) Disparity in navigation task complexity. UAVs frequently operate in diverse outdoor, open environments, where navigation paths are often long and complex. Unlike ground- based agents (Anderson et al., 2018) that navigate using well-defined descriptions of target objects, UAVs must deal with obstructed views and shifting perspectives due their high maneuverability. Therefore, relying solely on target descriptions inadequate for precise localization and navigation in such complex, dynamic scenarios. To address these disparities and associated challenges, we propose a UAV simulation platform TRAVEL, target-oriented VLN dataset, a novel benchmark UAV-Need-Help along with a UAV navigation LLM to create more realistic framework for UAV vision-language navigation which supports continuous trajectories and complex scenarios.
To narrow the gap to realistic UAV VLN tasks, a crucial advancement lies in replacing fixed action sequences with continuous flight trajectories. To facilitate this, we introduce TRAVEL, simulation platform Towards Realistic Aerial Vision and Exploratory Language navigation, offering realistic environments, authentic flight simulation, and extensive algorithmic support. As shown in Fig. we utilize UE4's realistic rendering capabilities, integrate 22 scenarios and 89 objects, and provide APIs for object placement and scenario configuration. We integrate AirSim plugin to translate trajectory sequences into continuous paths with realistic flight dynamics and support continuous flight control using real flight signals via remote controllers designed APIs. Building the features above, develop a parallel realistic trajectory collection framework and a closed-loop simulation framework, providing comprehensive algorithmic support for UAV tasks.
2

---

Published as a conference paper at ICLR 2025
Table I: Comparison of existing VLN datasets based on the DoF of navigation action space and dataset scale. Our UAV dataset incorporates 6 DoF trajectories for realistic VLN tasks. Neraj and N Noccab represent the number of collected trajectories and vocabulary size, respectively.
Dataset
DoF Kinematics
Ntraj
Nuccab
Traj Len.
Intr Len.
Actions
Environment
R2R
X
7189
3.1K
29
Matterport3D
TouchDown
2
X
9326
5K
310
90
35
Google Street View
LANI
2
x
6000
2.3K
16
57
116
CHALET
AVDN
3
x
6269
3.3K
144
89
7
XView
AerialVLN
4
X
8446
4.5K
661
83
204
AirSim
CityNav
4
X
32637
6.6K
545
SensatUrban
Ours
/
12149
10.8K
255
264
TRAVEL
To mitigate the scarcity of realistic UAV trajectory data for VLN tasks, We leverage the unique features of the TRAVEL platform to construct target-oriented VLN dataset, which is the first real- istic UAV VIN dataset to incorporate 6 degrees of freedom (DoF) motion, accurately capturing the complex flight dynamics of UAVs. Human annotators conducted continuous flights on the TRAVEL platform, regularly recording UAV states during annotation and asynchronously collecting addi- tional sensor data to obtain navigation trajectories. We utilize GPT-4 to generate target descriptions and follow up with manual quality checks to generate high-quality navigation instructions, which result total of approximately 12k rajectory-instructino pairs.
The high mancuverability of UAVs and the complex aerial environments pose significant challenges for UAV object search task relying solely on descriptive instructions. To support UAV accom- plish such tasks, we enhance UAV navigation capabilities by providing additional information and accordingly propose the UAV-Need-Help benchmark. As shown in Fig. [ that provides action guidance to UAVs in specific scenarios and categorize the assistant into three types based on the varying levels of guidance. The UAV performs the task based on the initial target description, environmental information, and instructions from the assistant.
we establish an assistant
To tackle the challenges presented by the UAV-Need-Heih benchmark and enhance the efficiency of UAVs in object search tasks, we propose UAV navigation LLM. We leverage MLLM's under- standing (Li et al. 2024 Han et al., 2024) and decision-making Zhang et al. (2024) capabilities to produce hierarchical outputs for long-distance and fine-grat fine-grained trajectories. Additionally, propose a backtracking sampling-base bassdd data aggregation strategy to enhance environmental under- standing and obstacle avoidance capabilities. Comprehensive closed-loop evaluations across various settings validate the effectiveness of our approach.
We hope that the proposed platform, benchmark, and methodology can promote research and de- velopment in VLN tasks based on continuous and realistic UAV trajectories, thereby facilitating the transfer of UAV VLN systems real-world applications.
RELATED WORK
Embodied Simulator. Simulation platforms erucial for intelligent systems research. For ground-based simulations, platforms like Habitat (Puig et al., 2024). Matterport3D (Chang et al.] 2017) and Google Street View (Anguelov al. 2010) have been pivotal simulated realistic indoor and outdoor scenes, significantly propelled research by improving data collection and algorithm evaluation in complex scenarios. However, UAV simulation platforms are still in their early stages of development. Some existing platforms, such as Gazebo (Koenig & Howard 2004), provide a foundation for UAV control and navigation, but they still exhibit limitations scalability, and adaptability to various scenarios. The x View platform (Lam et al., 2018) only offers remote sensing satellite images which are unsuitable for low-altitude navigation tasks. Recently,
terms of realism,
some studies (Liu al.) 2023b Ma et al. 2020 Haley et al., 2023 ) Gill et al. 2021 Madaan al.)
imaging plat- forms. However, these platforms typically lack algorithmic support for continuous trajectory VLN tasks. In contrast, our platform offers realistic environments, authentic flight simulation, and exten- sive algorithmic support, providing foundation for realistic UAV VLN research.
2020 Alvey et al., 2021) have combined UE with AirSim to develop
3

---

Published as a conference paper at ICLR 2025
Vision-Language Navigation Datasets. UAV VLN is an emerging task recently formed from the expansion of outdoor VLN tasks. Table presents a comparison between representative VLN datasets. Beginning with indoor navigation, R2R dataset (Anderson al.) (018) establishes a foun- dation for grounded VIN. Inspiring works (Vasudevan et al. 2020; Zhu et al. 2021 Chen et al.) 2019 Misra et al., 2018) such TOUCHDÓWN, LANI LANI have employed verbal navigation instruc- tions to address the challenges of outdoor long-range navigation. Moving beyond ground-level navigation, AVDN (Fan et al. 2022) collects aerial navigation trajectories with human dialogues. AerialVLN (Liu et al., 2023b) and CityNav (Lee et al. (024) propose VLN for UAVs and collect aerial trajectory datasets of discrete 4-DOF UAV action. In previous datasets, UAV trajectories were collected by simply modifying the UAV's position, which significantly deviates from real UAV flights. Based on our platform, we collect a target-oriented realistic UAV VLN dataset with contin- uous 6-DOF trajectories to enhance the realism of UAV navigation tasks.
LLM-based Navigation Agent. The application of LLLM-base methodologies has led to signif- icant advancements in developing navigation agents. LM-Nav (Shah al. 2022) propose a goal- conditioned robotic navigation method that enables robust generalization to real-world outdoor en- vironments without Imnguage-annotate data. LMDrive (Shao et al. 2024 further progressed the field by introducing a multi-modal sensor data, facilitating effective human-robot interaction in challenging urban scenar- ios. Some studies (Vemprala et al. 20233] /Zhong et al.] 2024) have utilized the zero-shot capabilities of LLMs to generate UAV navigation code, while (Lee et al.) (024) combined LLMs for coordinate optimization of drone target points, validating the basic capabilities of LLMs in drone navigation tasks. To further expand the application scenarios for drones, we proposed UAV navigation LLM composed of hierarchical trajectory generation models in solving the object search task.
guided, end-to-end autonomous driving framework that integrates
TRAVEL SIMULATION PLATFORM
As illustrated in Fig. the TRAVEL simulation platform is a fully open-source platform devoted to realistic UAV VLN tasks, integrating three modules: environment composition, flight simulation, and algorithmic support to achieve comprehensive functionality.
3.1 ENVIRONMENT COMPOSITION
Diverse Scenario Resources. TRAVEL offers a wide range of scenarios and achieves high-fidelity visual effects through the advanced graphical rendering capabilities of UE4. We integrate high- quality scenarios from online repositories and urban scenarios from the CARLA simulator (Doso-) vitskiy al. 2017), resulting in the TRAVEL platform featuring 22 distinct scenarios, including urban, rural, and natural landscapes, as detailed in Appendix A The platform can also simulate dy- namic environments, such as vegetation swaying and variations in lighting. This further enhances the platform's 's realism, reducing the gap when transitioning to real-world environments.
Tailored Object Assets. TRAVEL platform features wide variety of standalone object assets, in- cluding humans, vehicles, animals, road signs, tables and other items suited for urban and natural environments. Users can place objects in scenarios using different methods based on task require- ments. In addition to UE4's built-in scenario editor, we support the automatic placement method at runtime. The method is implemented through the TRAVEL API, where we have pre-defined feasi- ble regions within each scenario and classified various object categories. Annotators can utilize the object placement API to select the object type and place it in appropriate and suitable areas.
3.2 FLIGHT SIMULATION
Realistic UAV Flight Control. TRAVEL platform integrates the AirSim plugin to achieve realistic UAV flight simulation, enabling more precise flight control. Our platform can utilize the flight control API to achieve physics-based UAV maneuvers with a DoF trajectory representation. The pose of the UAV at each time is represented position coordinates, and (0, o, e)) denote the pitch, roll, and yaw angles respectively. The pose can be obtained at any time, allowing highly accurate simulation of UAV movement.
P {z, y. 2, 0, o, v. where (æ, v, represent the
4

---

Published as a conference paper at ICLR 2025
Configurable UAV Sensors. TRAVEL platform supports the simulation of various sensor pay- loads during UAV flight, including IMU, RGB and depth cameras, LiDARs and GPS. The platform initially configures the UAV with RGB and depth cameras covering front, rear, left, right, and down- ward views, along with a LiDAR sensor set up with a 360-degree horizontal field of view. Users can add necessary payloads and adjust detailed sensor configurations based specifie requirements, such as modifying image resolution to optimize response time.
Human Control Interface. We integrate AirSim's hardware-in-the-lon simulation capability and use PX4 to support remote controller operation. However, the control link in AirSim can be rela- tively complex for general users. Therefore, we develop simplified APIs that allows users simulate remote controller input via the keyboard, supporting functions such as data collection, and enabling operation both manual mode and position mode.
3.3 ALGORITHMIC SUPPORT
Data Collection Tools. TRAVEL platform has contained data collection framework to tackle the challenge of limited UAV training data. Collecting data from multiple sensors requires a certain amount of time, which can affect the continuity of annotation. Therefore, have implemented an asynchronous collection method that initially gathers UAV attitude information at equal time intervals, followed by the data collection of sensors in the background.
Closed-loop Simulation. The platform provides an extended interface for the UAV navigation model, allowing for flexible integration of model outputs to control drone flight and real-time feed- back of environmental information to the model. Moreover, we have implemented a dataset aggre- gation (DAgger) method enhance model training and proposed a backtracking policy. When the UAV encounters a collision, it is reverted to its previous position and uses teacher action to perform the next step, allowing it to recover from the collision and produce longer navigation trajectories.
Parallelization. TRAVEL platform employs an environment parallelization strategy. allowing mul- tiple simulated environments to run concurrently to enhance the efficiency of data collection and closed-loop evaluation. With 8 NVIDIA A100 GPUs, the simulation of a single UAV achieves performance boost of 16 times, reaching frame rates of between 160 and 1600 fps, depend on the amount of data captured by UAV payloads.
4 TARGET-ORIENTED REALISTIC UAV VLN DATASET
We construct a target-oriented realistic UAV VLN dataset using the proposed TRAVEL platform, which the first dataset accurately capturing complex flight dynamics for UAV VLN tasks. As shown in Fig. (a), we introduce this dataset in two aspects: data collection and data analysis.
4.1 DATA COLLECTION
Description Collection. In the UAV object search task, the target descriptions consist of three key components: 1) Target direction, indicating the relative position between the target and UAV's initial pose; 2) Object description, detailing the visual features of the target; and 3) Environmental information, describing the surrounding spatial context. During the description collection process, objects can be placed in the feasible regions of the scene using the TRAVEL object placement API. Object descriptions and environment information are then obtained by positioning a camera above the object to capture images from five different views (front, back, left, right, and down), followed by generating textual descriptions using GPT-4 (Achiam al.) (023). with the prompts detailed in Appendix B Human experts review and refine the generated content maintain data quality, removing any inaccuracies or hallucinations.
Asynchronous Trajectory Collection. Human common sense in target searching provides valu- able guidance for UAVs to learn effective search strategies. During the trajectory collection process, human experts utilize the human control interface of the TRAVEL platform to manually control the UAV according to the given instructions to search for the target. To minimize operational discon- tinuity caused by sensor data storage latency, only UAV states recorded at fixed time intervals during the flight. After completing the trajectory collection, sensor data is acquired through the parallel data collection framework of TRAVEL. The trajectory recording terminates when the UAV
5

---

Published as a conference paper at ICLR 2025
Description Collection
Asynchronous Trajectory Collection
There is target
OpanllAV
merii treer car area
Control
burlding cneet
S
Object Meta Indo
Review & Analysii
Replay
Pamilel
Sensor Data
Trajectory
(a) Data collection pipeline
-lard
Vahicle 39.7%
Text us
Train 75.33%
300
200
Takme 67
Human 31.9%
Other 8.76
11.0%%
200
400 600
100
200
300
(b) Trajectory length
(c) Distance to goal
(d) Object category distribution
(e) Data split distribution
Figure 2: Overview of our dataset construction and statistical analysis. (a) Data collection pipeline for generating high-quality target descriptions and realistic UAV trajectories. (b) (e) Statistical analysis of the dataset, covering trajectory lengths, task distances, object categories, and dataset splits. In (e), UM and UO represent Unseen Map and Unseen Object, respectively.
approaches within 5 meters of the target, and any trajectory is discarded if a collision is detected. Ultimately, a total of 12, 149 valid trajectories are obtained.
4.2 DATA ANALYSIS
Trajectory Analysis. As illustrated in Fig. (b), the UAV-Need-Heij dataset contains a total of 12,149 trajectories, where trajectories shorter than 250 meters classified easy, and those ceeding 250 meters are classified as hard. The diversity in trajectory lengths ensures the challenge and complexity of the tasks. Fig. 2 (c) illustrates that the distance to the targets ranges from 50 meters to 400 meters, representing the spatial scale of the environment.
Description Analysis. The most frequently occurring descriptions illustrated in Fig. (a), in- cluding building, tree, and car. These descriptions provide contextual information for UAVs, en- hancing their capability to estimate object locations through visual cues and thereby accurately find the target. Fig. 2 (d) shows the target object set consists of 89 distinct categories for search tasks, encompassing vehicles. humans, animals and other objects.
Dataset Split. As shown in Fig. 2 (e), to comprehensively evaluate the model's performance on both seen and unseen environments, and to analyze its ability to generalize to new maps and ob- jects, we divide the dataset into four subsets: Train, Test Seen, Test Unseen Map, and Test Unseen Object. Each test subset is further divided into easy and hard categories, consistent with the criteria mentioned above. Specifically, the data distribution for each subset is as follows:
Train 9152 trajectories with 76 objects across 20 scenes as the training set.
Test Seen 1410 trajectories generated using objects and scenes seen in the training set.
Test Unseen Map 958 trajectories with 2 scenes unseen in the training set.
Test Unseen Object 629 trajectories with 13 objects unseen in the training set.
UAV-NEED-HELP BENCHMARK
5.1 TASK FORMULATION
We propose assistant-guided UAV object search task named UAV-Need-Hod -eep. where the UAV navigates to the target object following the target description, environment information and guidance from the assistant. Fig. provides an illustration of the UAV-Need-Help task.

---

Published as a conference paper at ICLR 2025
uxtiatnnce
Trajectory
LI
Guoding UAY to aeacle path
LaRA
MLLM
viopaitt lostu:
Predictor
Difiicatt iiuntions
L.2
back oracle patts Hazard sinations
Target Pone
I,3
WWarpsisth
Guide UAV obstucle avoidance
Text Tokenizer +
HHiiiiea Enceder
Trajectory Completion
Multi-vier Image
Image
Instruction
Description
Information
Depch
Assistant
"Tore Rade"
Refined Pose Sequence
Available Data
Modal
(a) Assistant
(b) UAV Navigation LLM
Figure 3: Overview of the assistant mechanism and UAV Navigation LLM framework. (a) Three different assistant settings for providing varying levels of guidance. (b) UAV Navigation LLM framework: The instructions, multi-view images, and a learnable query encoded into the MLLM, where the query extracts features to predict a using frew inputs by trajectory completion model to generate fine- fine-grained trajectories.
target pose. This pose is then refined
Formally, in each episode, the UAV starts at initial position with posture Po and receives target description that specifies the target direction, object features, and its surrounding environment. At each time step, the UAV obtains its state S (position, posture, velocity), along with RGB images R and depth images D from five perspectives: front, left, right, rear, and below. An assistant monitors its status and provides additional instructions I' to suggest flight strategies when needed. In the closed-loop simulation, the UAV navigation model predicts 6 DoF trajectory sequence based on {I, S, R. D). Using the TRAVEL platform's flight API, the UAV navigates to each predicted posi- tion with 6 DoF attitude (x',y', z', 0', ø' v/') while adhering to its flight dynamics and updating its observations. The task is successful if the UAV lands within a 20-meter radius of the target.
5.2 ASSISTANT MECHANISM
As previously mentioned, the inherent complexity and dynamic nature aerial environments, with obstructed views and shifting perspectives, make basic target descriptions insufficient for UAV ob- ject searching. Thus, we introduce assistants to aid UAVs in this task, defining three distinct tiers based on the level of guidance provided, shown in Fig. (a).
LI assistant provides high-frequency guidance closely aligned with the ground truth (GT) trajec- tory. It continuously calculates the trajectory point in the oracle path that is closest the UAV's current position, then identifies the appropriate actions based on the UAV's orientation and the direction toward the nearest GT point. These actions may include cruising, turning, landing. This setting ensures that the UAV stays on the correct path at all times.
L2 assistant intervenes when the UAV encounters difficulties, providing www-frequecyy correc- tions to steer it back toward the ground truth trajectory. It activates only when collision risk is identified through depth maps or when the UAV deviates too far from the GT path, offering
corrective actions to guide the UAV back to the desired trajectory.
L3 assistant only provides obstacle avoidance assistance when the UAV is in a hazardous scenario. It determines the distance between the UAV's s current position and obstacles using depth maps, and when proximity to an obstacle is detected, it issues avoidance commands to prevent collisions.
6 UAV NAVIGATION LLM
UAV navigation LLM is a multimodal LLM capable of handling various input types, including images and texts. We first tokenize the the language space. These tokens then concatenated and fed into the LLM, with Vicuna-7B serving as the base model. Utilizing the features obtained by LLM. a hierarchical trajectory decoder generates both the UAV's target pose Ptarget and refined pose sequence ( (Porgy PI P.N Ptraj... P
input, where vision tokens are aligned with

---

Published as a conference paper at ICLR 2025
where N represents the length of this trajectory. The overview architecture is shown in Fig. (b). Additionally, we have designed backtracking sampling baaled based data aggregation to expand the dataset, thereby enhancing models' obstacle avoidance capabiliiiies in complex scenarios.
6.1 HIERARCHICAL TRAJECTORY GENERATION
Multimodal Tokenization. Given the task description Task. assistant instructions Tinstr and the UAV state information Istate. we tokenize them using a pre-trained language tokenizer (Li al. 2024) obtain corresponding tokens Tinstr. For the multi-view images, we utilize EVA- CLIP (Sun et al., 2023) and a Q-former structure to extract the visual features. Each image is con- verted into a set of tokens, consisting of I context token that captures global features and 16 content tokens that represent local details through grid pooling. Finally, we concatenate the three types of tokens: Ting for image tokens, Tiask for task description tokens, and Tinstr for instruction tokens, to construct the multi-modal input token sequence, denoted as Timput =< Timg: Trask, T Tinstr The details of the multimodal input can referenced Appendix D
Hierarchical Trajectory Decoder. The decoder is the core component that translates the input into the UAV's next step target pose. To enable complex trajectory planning within dynamic environ- ments, we employ hierarchical structure with two levels: high-level MLLM-based trajectory decoder and a fine-grained path decoder.
The MLLM-based trajectory decoder utilizes special learnable trajectory token Tup input to the LLM, which helps to extract trajectory-specie features Qwp- These features are fed into a multilayer perceptron (MLP) decode the target pose Prargets allowing the model to plan UAV trajectories based on environmental data and detailed language instructions.
The fine- fine-grained path decoder generates trajectory details to improve navigation efficiency. It takes visual tokens Ti encoded from the front-view image using pre-trained ViT and combines TV with processed poses derived from the MLLM- decoder. The concatenated features are passed through MIP to produce fine-grained trajectories Ptraj. The overall process defined as follows:
Peraj MLP imgg Starget (MLLM(<< Tinput: Qup >]))
where Starget represents the alignment process of target poses from the MILM output with the dimensions of visual tokens. To further improve UAV landing accuracy near the target, a Grounding DINO model (Liu et al.) (2023a) is utilized for detection. Once the target object is identified in multi-view images, the navigator initiates the landing process.
6.2 BACKTRACKING SAMPLING-BASER DATA AGGREGATION
We implement a DAgger (Ross et al. 201T) module for continuous and realistic UAV trajectories. During data collection, the module samples from both predicted trajectories and reference paths pro- vided by the teacher model. The teacher model selects the closest ground truth position to the UAV's current location and computes the next trajectory, which guides the UAV in VLN tasks. Considering that collisions often occur when simulating UAV flight, it becomes challenging to collect complete trajectories or those containing sufficient obstacle avoidance knowledge. Thus, we propose back- tracking sampling mechanism, where, if the UAV takes an action based on the model's output that reverted its state from two frames earlier, restoring its pose, velocity,
results in a collision, and other attributes. It will then follow the trajectory given by the teacher model, helping the UAV avoid collisions and stay on track. this way, we can obtaín more successful obstacle-avoidance trajectories, thereby enhancing the navigation capability of the model.
EXPERIMENTS
7.1 EXPERIMENTAL SETUP
Evaluation Metrics. We adapt evaluation metrics commonly used in VLN AAdersoo et al. 2018 Krantz et al.) 2020). including success rate (SR), oracle success rate (OSR), success weighted by path length (SPL.) and navigation error (NE). SR measures the percentage of tasks where the UAV
8

---

Published as a conference paper at ICLR 2025
Table 2: Results on Test Seen Set across different assistant levels. DA refers model trained using backtracking sampling-basee based data aggregation.
Full
Easy
Hard
Method
Assistant
NEJ
SR1
OSR1
SPL1
NEJ
SR1
OSR
SPL1
NEL
SR
OSR
SPL
Random
L1
222.20
0.14
0.21
0.07
142.07
0.26
0.39
0.13 320.12
0.00
0.00
0.00
Fixed
L1
188.61
2.27
8.16
1.40
121.36
3.48
11.48
2.14 270.69
0.79
4.09
0.49
CMA
L3
140.93
4.89
11.56
4.41
83.58
7.35
17.81
6.53
210.91
1.89
3.94
1.83
CMA
1.2
141.55
7.02
15.39
6.54
87.77
9.55
19.87
8.74
207.18
3.94
9.92
3.94
CMA
L1
135.73
8.37
18.72
7.90
84.89
11.48
24.52
10.68
197.77
4.57
11.65
4.51
Ours
L3
146.32
6.31 15.39
5.10
93.15
9.55
21.94
7.32 215.85 2.36
7.40
2.17
Ours
L2
120.57
12.98 37.38
11.30
76.89
17.55
43.48
15.01 186.22 7.40
29.92
6.76
Ours
L.
106.28
16. 10 44.26
14.30
68.78
18.84
47.61 16.39 152.04 12.76
40.16
11.76
Ours-DA
L1
98.66
17.45
48.87
15.76
66.40
20.26
51.23 18.10 138.04
14.02
45.98
12.90
Human
L1
14.15 94.51 94.51 77.84 11.68 95.44 95.44 76.19 17.16 93.37 93.37 79.85
successfully reaches the target. OSR measures whether the UAV reaches any location along the op- timal trajectory, even it doesn't 't exactly reach the final destination. SPL evaluates both the success rate and the efficiency of the path taken, rewarding shorter, more optimal paths. NE calculates the average distance between the UAV's final position and the target.
Comparison Baselines. We compare the following baselines against method the UAV-Need- Help task. 1) Random: The UAV randomly selects trajectory poses without any structured planning or guidance. This method is employed to illustrate the extent of the solution space. 2) Fixed Action: The UAV maps the assistant's 's instructions into predefined fixed actions. For example, cruise means moving forward by 5 meters, while tum left results in 30-degree turn followed by 5-meter forward movement. 3) Cross-Modal Attention (CMA): The CMA model (Anderson et al.) (018) commonly used in grounded VLN tasks and employs bi-directional LSTM simultaneously process image inputs and instruction comprehension. To adapt to our task setting, we modify its recurrent predictor output set of trajectories instead of traditional navigation actions.
7.2 QUANTITATIVE RESULT
Comparison with Baselines. Table shows that our method outperforms across all metrics at dif- ferent difficulty levels on the test seen set. At various assistant levels, the SR metric improves by an average of 5% over the CMA model. This demonstrates that our MLLM-based hierarchical trajec- tory generation approach enhances scene understanding and produces more accurate and adaptable pose sequences, ultimately improving decision-makinn and overall performance.
Table 3: Generalization capabilities across different test sets with LI assistant, where UO and UM represent the Test Unseen Object Set and the Test Unseen Map Set, respectively.
Full
Easy
Hard
Method
Test Set
SR1
OSR
SPL*
NEG
SRT
OSR SPL1
NEL
SR1
OSR
SPL
Random
UO
260.14
0.16
0.16
0.16
174.10
0.48 0.48
0.48 302.96 0.00
0.00
0.00
Fixed
UO
212.84
3.66
9.54
2.16
151.66
6.70
13.88
3.72 243.29
2.14
7.38
1.38
CMA
UO
155.79
9.06
16.06
8.68
102.92
14.83
22.49
13.90 182.09 6.19
12.86
6.08
Ours
UO
118.11
22.42
46.90
20.51
86.12
24.40
49.28
22.03
134.03
21.43
45.71
19.75
Random
UM
202.98
0.00
0.00
0.00
158.46
0.00
0.00
0.00
265.88
0.00
0.00
0.00
Fixed
UM
180.47
0.52
2.61
0.39
132.89
0.89
4.28
0.67
247.72
0.00
0.25
0.00
CMA
UM
141.68
2.30
10.02
2.16
102.29
3.57
14.26
3.33
197.35
0.50
4.03
0.50
Ours
UM
138.80
4.18
20.77
3.84
102.94
4.63
22.82
4.24
189.46
3.53
17.88
3.28

---

Published as a conference paper at ICLR 2025
8
10
11
12
Instruction: There is vehicle in left front of UAV. In the center of the figure, there is yellow vehicle. The surrounding area consists of multiple multi-story buildings with varying architectural styles...
Instruction: There in right front of In the center the figure, there standing next
Figure 4: Visualization of object search results of our method. First two rows demonstrate our UAV successfully follows the instruction. Notably, the third fifth images depict the drone executing turning maneuver, resulting in a change in the drone's perspective. The third row illustrates failed example, depicting collision with trees forest scenario.
The classical baseline model CMA faces several challenges due limitations in model size and task complexity and performs poorly on multiple metrics. The Random and Fixed methods struggle to complete the tasks, demonstrating that achieving the target search tasks without understanding instructions and visual information is extremely difficult. We further evaluate our model trained with backtracking sampling-base based data aggregation, which shows an improvement in SR compared to the original model, indicating this method has enhanced the UAV's navigation capability.
Additionally, we evaluate the performance of humans operating UAVs with guidance from LI assis- tant. As shown in Table 2 humans achieve high success rate but sometimes take longer paths. This suggests that while humans make correct decisions, they may choose more cautious or exploratory routes, leading slightly less efficient path planning.
Comparison under Different Assistant Level. Under the continuous guidance of the L1 assistant, method achieves the highest success rate. As the assistant level increases, the UAV agent needs to rely more on its navigation planning capabilities to complete tasks. The results show that both our method and the CMA method experience decline success rate and other metrics at higher levels of assistance, demonstrating that long-term VLN for UAVs is extremely challenging.
Generalization on Unseen Cases. Table shows the performance on unseen datasets, highlight- ing our method's 's superior zero-shot capability and its adaptability to new scenarios. Our method demonstrates a slightly higher success rate in the unseen object test set than in the seen set. This phenomenon can be attributed to the inherent generalization capabilities of the Grounding DINO module (Liu et al.) (223a) used in our approach and the relatively simpler nature of the data. We have still partitioned this dataset for future research aimed at models that do not require external detectors. In the unseen scene test set, results the fixed action method reflect the complexity of the environment through its low success rate. Our method shows noticeable decline in the unseen scene test set, yet still outperforms other methods.
Table 4: Performance scalability with varying amounts of training data.
Performance Scalability. Table Treorts the model's per- formance across various quantities of training data with
LI assistant. The results show that as the data volume in- creases, the model performs better, indicating that a larger and more diverse dataset could enhance the model's under- standing and decision-making capabilities.
Data Amount
NEJ
SR1
OSRt SPLt
10%
126.50
10.78 31.21
9.68
50%
122.63
13.19
37.16
11.71
100%
106.28
16.10
44.26
14.30
7.3 QUALITATIVE RESULT
Fig. presents two examples evaluated in TRAVEL platform. The first two rows show the UAV successfully following the instructions, maneuvering through the buildings, and ultimately locating yellow vehicle. During this process, the UAV experienced camera view shifts due its attitude
10

---

Published as a conference paper at ICLR 2025
changes, highlighting the realistic fidelity of our platform. In contrast, the third row demonstrates a collision caused by insufficient altitude while navigating through forested area, showcasing the challenges posed by complex environments. Additional results can be found in Appendix E
CONCLUSION
We address the challenges of realistic UAV VLN from three aspects: platform, benchmark, and methodology. To achieve this, we develop the TRAVEL platform, which provides realistic environ- ments, flight simulation, and comprehensive algorithmic support. We also construct target-oriented realistic UAV VLN dataset and propose the UAV-Need-Heih benchmark, which provides assistance guide UAVs through complex VLN scenarios. Additionally, we propose UAV navigation LLM along with backtracking sampling- sampling-based based data augmentation strategy, which together effectively enhance the performance of realistic trajectory-based VLN tasks. Our contributions establish uni- fied framework for realistic UAV VI.N research, making significant strides toward bridging the gap between simulation and real-world UAV navigation applications. Furthermore, there are two promis- ing directions for future research in realistic UAV VI.N tasks. The first is to enhance the autonomous navigation capabilities UAVs, enabling them to operate effectively in complex environments with minimal guidance. The second is improve the transferability from UAV simulation to real-world deployment, facilitating the application of UAVs in real-world scenarios.
Acknowledgements This research is supported in part by National Key R&D Program of China (2022ZD01 15502), National Natural Science Foundation China (NO. 024611603002 U23B2010), "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01161).

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Brendan Alvey, Derek Anderson, Andrew Buck, Matthew Deardorff, Grant Scott, and James M Keller. Simulated photorealistic deep learning framework and workflows to accelerate computer vision and unmanned aerial vehicle research. In ICCV, 2021.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Fruch, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Lue Vincent, and Josh Weaver. Google street view: Capturing the world at street level. Computer, 2010.
Valts Blukis, Dipendra Misra, Ross Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position- rosition-visitationn prediction. In CRL, 2018.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterportåd: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:/709 06158, 2017.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural
language navigation and spatial reasoning visual street environments. CVPR, 2019.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In CRI, 2017.
Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, and Xin Eric Wang. Aerial vision- and-dialog navigation. arXiv preprint arXiv:2205. 12219, 2022.
Jappreet Singh Gill, Mahdi Sacedi Velashani, Jordan Wolf, Jonathan Kenney, Mohsen Riahi Manesh, and Naima Kaabouch. Simulation testbeds and frameworks for uav performance evaluation. In EIT, 2021.
Joshua Haley, Jonathan Tucker, Jonathan Nesper, Brian Daniel, and Trisha Fish. Multi-agent collab- oration environment simulation. In Synthetic data for artificial intelligence and machine learning: tools, techniques, and applications, volume 12529, pp. 208-213, 2023.
Songhao Han, Wei Huang. Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:241 14794, 2024.
Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open-source multi- robot simulator. In IROS, 2004.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, 2020.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and lad-uuug navigation with dense spatiotemporal grounding. In EMNLP, 2020.
Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. xview: Objects in context in overhead imagery. arXiv preprint arXiv:1802.07856 2018
Jungdae Lee, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, and Nakamasa Inoue. Citynav: Language-gon goal aerial navigation dataset with geographic information. arXiv preprint arXiv:2406. 240.244446002.... 2024.
12

---

Published as a conference paper at ICLR 2025
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid, vid: An image is worth 2 tokens in large language models. In ECCV, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023a.
Shubo Liu, Hongsheng Zhang, Yuankai Qi. Peng Wang. Yanning Zhang, and Qi Wu. AcrialvIn: Vision-and-languagg navigation for uavs. In ICCV, 2023b.
Chenxiang Ma, You Zhou, and Zhiqiang Li. A new simulation environment based on airsim, ros, and px4 for quadcopter aircrafts. In ICCAR, 2020.
Ratnesh Madaan, Nicholas Gyde, Sai Vemprala, Matthew Brown, Keiko Nagami, Tim Taubner, Eric Cristofalo, Davide Scaramuzza, Mac Schwager, and Ashish Kapoor. Airsim drone racing lab. In NIPS, 2020.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions to actions in 3d environments with visual goal prediction. In EMNLP, 2018. 8.
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote. Tsung- Yen Yang, Ruslan Part- sey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr- ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. In ICLR, 2024.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In CVPR, 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell, reduction of imitation learning and struc- tured prediction to no-regret online learning. In JMLR, 2011.
Dhruv Shah and Sergey Levine. ViKiNG: Vision-Based Kilometer-Seaa Navigation with Geo- graphic Hints. In Proceedings of Robotics: Science and Systems, 2022.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, 2024.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simula- tion environment for interactive tasks in large realistic scenes. In IROS, 2021.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. techniques for clip at scale, 2023.
Eva-clip: Improved training
Arun Balajee Vasudevan, Dengxin Dai, and Lue Van Gool. Talk2nav: Long-range vision-and- language navigation with dual attention and spatial memory. IJV,, 2020.
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Technical Report ASR-TR--023-8, Microsoft, February 2023.
An Yan, Xin Eric Wang, Jiangtao Feng, Lei Li, and William Yang Wang. Cross-lingual vision- language navigation. arXiv preprint arXiv: 19/0. 1301, 2019.
Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-basee vim plans the next step for vision-and- language navigation. arXiv preprint arXiv:2402. 15852, 2024.
13

---

Published as a conference paper at ICLR 2025
Jiageng Zhong, Ming Li, Yinliang Chen, Zihang Wei, Fan Yang, and Haoran Shen. A safer vision- based autonomous planning system for quadrotor uavs with dynamic obstacle trajectory predic- tion and its application with llms. In WACV, 2024.
Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang. Multimodal text style transfer for outdoor vision-and-languag navigation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), ACL, 2021.
14

---

Published as a conference paper at ICLR 2025
A PLATFORM AND DATASET DETAILS
We visually present some scenarios and complete object assets supported by the platform in Fig- and Fig. 6 respectively. In addition to the scenarios shown in Fig. | we have also included 11 urban maps transferred from the CARLA simulator, and described the styles of different maps in Table The dataset splits based on maps and objects are respectively presented in Table 6 and Table 7
WHITE
LIVES
AAY
Figure 5: Platform Scenarios Overview: We present a selection of scenarios, respectively depicting city, town, forest, and desert environments.
Figure 6: Platform Objects Overview: We present the platform's object assets, including vehicles,
people, animals, and other objects.
B PROMPT ENGINEERING IN DESCRIPTION COLLECTION
As shown in Fig- Z we use the following prompt collect object descriptions (Section The prompt integrates information from multiple camera views to provide a comprehensive represen- tation of the target object's visual characteristics. The generated instruction incorporates detailed descriptions of the surrounding environment's colors and shapes to facilitate accurate visual iden- tification. Furthermore, it specifies the spatial relationship between the target and nearby objects, enabling precise localization of the target within its context.
15

---

Published as a conference paper at ICLR 2025
Map
Style
Map
Style
NYCEavioommenteegaaa
City, Park, River Village, Forest, Park
New YorkCryyity
City
ModularParkk
BrushifyForesttPck
Forest
BattlefieldKitDesee
Desert
Carla_Towni
Town, River
ModularFuropean
City, Park
Carla_Town02
Town
TropicalIsland
Mountain, Forest
Carla_Town03
City
CrushifyCountryRee
Mountain, Village
Carla.Town04
Town, Mountain
NordicHarbour
Town, River
Carla TownO5
Town
Japanese_ Sttrett
City
Carla.TownOf
Highway
BrushifyUrban
City. River
Carla_Town0?"
Village
London_ Stteet
City
Carla_ Ionl
City
WesterTown
Desert, Village
Iarla_TownI5
City, Forest
Table 5: Maps and corresponding style classifications.
Dataset
Maps
Train set
NY YCEaviroommennMegaaaa New YorkCity, BattlefieldKiDDesert,, Modu-
larEuropean, Tropicallsland, BrushifyCountryRodde HordicHarbouur
Srrunese_ttreet Brushify Urban, London_Street, WesterTown, Brushify- ForestPack, Carla.TownO1. Carla.Town02, Carla_Town04, Towla_ Carla Town06, Carla Town07, Carla TOWn Carla Town15
Test seen
NYCEnviroonmentMeaaaa., YorCCtkCitt, ModularEuropean, Tropicallsland, TownO1, Carla_ Town02, Carla_ Town04, TownO5, Carla, TownO6, Tarla.TownnOee Carla.Townl
Test unseen
Carla_TownO2, ModularPark
Table 6: Map-based dataset split.
Dataset
Object
Train / Test seen
Volkswagen 2021 Parked, Tazzari, AmbulanceParked, CarlaCola, FiretruckParked, Truck, AudiA2, AudiTT, VolkswagenBeettl, BMW- GrandTourer, BMWIsetta, Chevroletlmpala, Citroen_C3, Cybertruck, Charger-parked, LincolnMMk22011-pprgg MiniCooperS. Mini2021 Mustang prop. NissanPatrolST, ChargerParked. FordCrown parked, LincolnParked, NissanPatro12021 parked, TeslaM3 TeslaM3.parkke, RoadBike, Yamaha, Vespa, Kawasaki, Harley, Bin, TrashCan01, TrashCan03, TrasdhBag. ShoppingBag, ShoppingTrollle, TravelCase, PlasticTable, PlasticChair, Table, Round_ hayBale, Tent_02, Tent_03, Tent_04, Warningonsstructio, WarningAccident, TrafficCones.01 StreetBarrier, FireHdrant, MailBox, Couple02, Couple04, Couple Dog01, Dog03, Man01, Man02, Man10, Man17, Man32, Man33, Man36, Man39, Man42, Man54, Woman(4, Woman27, Woman44, Woman51, Woman52, Woman53, African plepann, camel, Domestic_sheep. horse_skeleton, leopard, Longhom_cattie cattle, Longhorn V2. mud_pig, rhino, Scot- hand_cattle tiger, zebra
Test unseen
Man17, Man54, Woman51, Indian elephant, Cape Buffalo, Domestic pig. Dog02, Tent01, NissanMicra, EtronParked, Seat Li0coln2020Parked, TravelCase
Table 7: Object-based dataset split.
C EXPERIMENTAL DETAILS
Implement Detail. We include the implementation details necessary for reproducing results. In general, we adopted training strategy similar to that described in Li et al. (2024). During the train- ing of the MLLM, we freeze most of the model's parameters and only compute gradients on the visual projector, trajectory prediction head, and LoRA layers. These trainable parameters make up just 4% the total model parameters and significantly reduce computational costs while maintain- ing the model's core capabilities. We supervise the predicted 3D angles using cosine similarity loss
16

---

Published as a conference paper at ICLR 2025
Prompt:
CONTEXT Describe the object in the image and its surroundings. These are images captured from the top, front, back, left, and right views, and the center object has a mesh body name of (object name). OBJECTIVE # Please relate information from different views to describe the visual characteristics of the central object and the surroundings as a whole, prioritize describing the surrounding objects surrounding buildings > vegetation, and descriptions prioritize color and shape.
STYLE # Describe objectively; do not use jargon such as grid body, etc.
# TONE # Do not use subjective evaluation words. For example: sunny. beautiful, etc.
# AUDIENCE # Describes the object and its surroundings to make it easier for the drone pilot to find the target.
RESPONSE Please write the above tasks together paragraph, keep it concise, just information, don't have any additional output, don't emphasize that there are multiple viewpoints, blend the information together, pay attention to differentiate whether it's tall building short building, and what position the target in.
Target direction:
There is target in front of UAV. Using your front the x-axis and your right the y-axis, The target yaw angle of 213.0 degrees from you.
Object description and environment information (GPT4):
The central object green-colored vehicle positioned on road. The vehicle surrounded by urban architecture, including multiple feature mix stone or brick facades. One the notable buildings includes "Leartes Bank" sign, indicating purpose as a commercial building. There around windows and doors. Some the buildings have fire escapes visible, typical city settings.. The streets are wide and largely empty, except for a few scattered objects like street lamps, signage, and the parked green vehicle. There is no visible vegetation or trees the immediate area. The scene is devoid of people or other vehicles, adding to sense of spaciousness. The road is straight and clean, with buildings positioned symmetrically on both sides. billboard can also be seen on one of the buildings, contributing to the urban setting.
short buildings. buildings primarily multi-story, rectangular,
variety structures, some more ornate designs, particularly
Figure 7: The instruction includes the creation of target direction, object descriptions, and environ- mental information. The target direction is determined based on the coordinates of the UAV's start and end positions, while the object descriptions and environmental information are generated by GPT-4 using the provided prompts, incorporating the top, front, rear, left, and right views and the object's name.
and apply LI loss between the predicted waypoints and the ground truth. Similarly, for the trajec- tory completion model, we also freeze the parameters of the vision encoder. The MLP trajectory prediction layer trained using L1 loss between the predicted trajectory and the labelled trajectory.
The MLLM model is trained on 8 NVIDIA A100 GPUs with a batch size of 128 for 2 epochs, while the fine-grained model and CMA model are trained with a batch size of 128 for 10 epochs. We use Adam optimizer with one-cycle learning rate de- cay schedule to train all models, where we set the maximum learning rate 5e-4.
Table Ablation on model structure.
Component
NEG
SRt
OSR1
SPL1
w/o LoRA
124.28
12.20
36.52
10.76
w/o Leas
113.53
13.55
40.28
12.34
w/o up
111.53
15.18
41.13
13.61
Ours
106.28
16.10 44.26
14.30
Ablation Study. To validate the effectiveness each part, we summarize experiments results in Table 8 where Loos indicates the cosine loss of poses, Qup indicates the learnable query. The ablation of model architecture highlights the critical role of LoRA. The exclusion of LoRA resulted in an 8% decrease in OSR and 4% in SR. Removing the cosine loss and the learnale query also leads to a decline in performance, but the impact relatively smaller. The complete model performs best across all metrics, validating the effectiveness of each component in enhancing model performance.
17

---

Published as a conference paper at ICLR 2025
We further sample part of the test seen set for more extensive ablation experiments. First, val- idate the effectiveness of the hierarchical prediction design. As shown in Fig. g the hierarchical trajectory prediction method achieves a balance between performance and computational efficiency, completing one inference in 0.407 seconds, which matches real-time drone navigation requirements. Using MIL.M alone for long-distance prediction results in decrease in success rate with similar in- ference times, Fine-grained predictions using MLLM improve performance by receiving assistance information more frequently. However, this comes at the cost of significant increase in inference time, making it unsuitable for real-time navigation.
Table 9: Ablation on assistant level.
SR
25.94/ 0.52
Hicrarchical trajectory generation
Assist Level
SR1
OSR1
NEG
SPL
MLLM for fine- grained prediction
24.06 / 2.46
No assist
6.25
11.88
167.10
4.95
L3 assist
7.20
13.71
163.55
6.17
20.94 1244/2.66 MLLM for long-range prediction
LI assist
24.06
43.13
130.57
20.42
20
Table 10: Ablation on sensor configurations.
Model
SR
OSR1
NEІ
SPL1
FPS
Front&top view 22.50
41.88
132.19
18.69
Bascline
24.06
43.13
130.57
20.42
Figure 8: Ablation of hierarchical prediction.
We also conduct experiments on the UAV VLN task without relying on assistant to validate the current capabilities of autonomous UAV navigation in Table 9 The results show that similar to previous UAV VLN tasks (Liu et al. 2023b), completing trajectryy VLN tasks without guidance information remains a significant challenge.
In Table 10 explore the impact of using fewer sensors. We believe the front and top views more important for detection in search tasks, so we kept only these two perspectives for the VLN tasks. The experimental results show slight performance drop across various metrics after reducing the number of sensors, indicating that visual information plays a supportive role in decision-making.
D MLLM PROMPT TEMPLATE
In this section, provide a detailed prompt for the input to MLLM, which organizes assistant instructions, drone states, task description information, and multi-view images into a specific format for input into MLLM.
A chat between a user and an Al agent. The agent is a navigation model that outputs UAV waypoints based on the user's instruction and UAV state information.
Assistant: <assist info> Previous displacement: <delta>
Current position: <cur_post
Current image: <img>
Instruction: <task deac>
Please control the drone and rind the ta target.
Next goal position: <waypoint>
Figure 9: We present the specific template for the input to MLLM, where <delta>>> represents the UAV's position change from the previous moment, <cur pos>> denotes the current global position in the coordinate system of the first frame, and <waypoint>> is the learnable special query.
E ADDITIONAL QUALITATIVE RESULTS
Here, we provide more visualization results of our method in closed-loop evaluation.
18

---

Published as a conference paper at ICLR 2025
AEG
Instruction: There a left front of UAV. Using your front the x-axis and your right as the y-axis, The target
yow ungle 40 degrees from you. The object is red fire trunk positioned on a dark asphalt surface. There are sidewalks adjucent buildings with clean, light-colored pedestrian the downtown-like area with mix of buildings, and trees along the streets. Please control the drone and find the target.
and several evenly spaced streetlights. To locate the red trunk, focus
Corresponding Top- Top-Down View
Instruction: There is target in the right front of UAV. Using your front as the s-axis and your right the y-axis, The larget is at yaw angle of dogrees from you. The center object brown elephant lying on the ground surrounded by donse area of displaying
vegetation. To the fromt, back, left, and right, there lighter of soil and rocks. The terrain is mostly flat with gentle slapes. In the backdrup. there are sanull bills and elevated areus visible under clear sky, providing additional reference points for locating the borse. Please control the drone and find the targot.
scattered throughout the landscape, with
Figure 10: More examples of the UAV object search task using our method illustrate longer trajec- tories and improved navigation performance across various scenarios.
19


Reviews:
[{"review_id": "g4oxnRBYXS", "reviewer": "Reviewer_DUG4", "summary": "This paper makes three key contributions to advance realistic UAV vision-language navigation. First, they introduce OpenUAV, a simulation platform that properly models realistic drone flight dynamics with 6 degrees of freedom, rather than using simplified discrete movements like previous work. Second, they create what appears to be the first UAV navigation dataset (about 12k trajectories) that incorporates realistic flight physics. Third, they propose a novel \"UAV-Need-Help\" benchmark where an AI assistant provides varying levels of guidance, along with a hierarchical navigation model using large language models.\nThe approach significantly outperforms baselines in experiments, though still falls well short of human performance. What makes this work valuable is how it addresses fundamental limitations in current UAV navigation research by developing tools and datasets that better match reality, particularly through the inclusion of realistic flight dynamics. While substantial work remains before real-world deployment would be feasible, this paper lays important groundwork and provides valuable resources for future research in autonomous drone navigation.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper address realistic flight dynamics in UAV navigation instead of using oversimplified discrete actions.\n2. Provides a comprehensive open-source simulation platform that combines realistic environments, flight physics, and algorithmic support.\n3. Creates the first large-scale UAV navigation dataset (12k trajectories) that captures true 6-DOF flight dynamics.\n4. Introduces a novel assistance mechanism with three levels of guidance, making the navigation task more adaptable and learnable.\n5. Proposes a practical hierarchical architecture combining language models with trajectory planning for complex navigation.\n6. Includes thorough experimental validation across different difficulty levels and scenarios, with clear comparisons to human performance.\n7. Shows strong generalization ability on unseen objects, though with expected performance drops on unseen environments.", "weaknesses": "1. The dataset size (12k trajectories) is relatively small compared to other vision-language datasets, which might limit model learning.\nLacks detailed comparison with real drone flight data to validate how well their simulation matches actual flight dynamics.\n2. No discussion of computational requirements or real-time performance, which is crucial for practical drone navigation.\n3. Success rate metrics (around 16-17%) are still quite low even with the highest level of assistance.\n4. Missing ablation studies on different environmental conditions and their impact on navigation performance.\n5. No discussion of safety mechanisms or fallback strategies when the navigation system fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PAyv6HoawW", "reviewer": "Reviewer_LL2n", "summary": "This paper proposes a new framework for UAV navigation, focusing on the different challenges of aerial environments and ground-based agent navigation. \nA realistic UAV simulation platform, OpenUAV, is proposed, which supports continuous six-degree-of-freedom (6-DoF) trajectories, realistic flight dynamics, and diverse environmental scenarios. The platform integrates the AirSim plugin and Unreal Engine 4 (UE4) to achieve high-fidelity simulation.\n\nUAV-Need-Help Benchmark: A new benchmark task introduces an assistant-guided UAV target search task, where the assistant provides different degrees of navigation guidance to the UAV to help it cope with complex navigation tasks.\n\nUAV Navigation Large Model (LLM): A multimodal model is proposed that can process visual and textual information simultaneously and generate hierarchical navigation trajectories to address the complexity of UAV navigation in three-dimensional space. The platform, dataset, and method aim to narrow the gap between simulated UAV navigation tasks and practical applications, and provide more realistic and challenging tasks for UAV VLN research.", "questions": "1. Real-world Deployment:\nHow well do the models and methods developed on the OpenUAV platform perform in real-world UAV navigation tasks? Has there been any experimental validation involving physical UAVs? Although the simulation platform is quite realistic, it remains unclear to what extent the models trained on this platform can generalize to real-world UAV tasks. A comparison with experiments using real UAVs is lacking.\n\n2. Autonomy and Reliance on Assistance:\nThe UAV-Need-Help benchmark seems overly reliant on the assistance mechanism, which could reduce the autonomy of the UAV. It is unclear how the UAV Navigation LLM would perform without such assistance. Can the UAV autonomously recover from navigation failures in the absence of external help, particularly when faced with obstacles?\n\n3. Efficiency and Real-time Performance:\nConsidering the complexity of the LLM-based navigation model, how feasible is the real-time deployment of such models on resource-constrained UAVs? Is there any analysis of the computational requirements and latency when running these models on actual drones?\n\n4. Ablation Studies:\nCould more insights be provided on the contribution of individual components (e.g., assistant guidance, hierarchical trajectory generation, sensor configurations)? How would the model's performance change if fewer sensors were used, or if the advanced trajectory decoder based on MLLMs were removed?\n\n5. Lack of Comparison with Ground-based VLN:\nWhile the paper highlights the differences between ground-based and UAV navigation, there is a lack of direct comparison with existing ground-based VLN methods(Seq2Seq[1], DUET[2], NavGPT2[3]). It is unclear what unique challenges the proposed framework addresses. Specifically, what are the differences between ground-based VLN methods and the UAV Navigation LLM? Could ground-based VLN methods be applied to OpenUAV, and if so, how would they perform?\n\n6. Extensibility to Other Tasks\nBesides the object search task, can the proposed platform and dataset be generalized to other UAV tasks such as mapping, exploration, or surveillance? Would the dataset require significant modifications to accommodate these tasks?\n\n[1]Anderson, Peter, et al. \"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n[2]Chen, Shizhe, et al. \"Think global, act local: Dual-scale graph transformer for vision-and-language navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[3]Zhou, Gengze, et al. \"Navgpt-2: Unleashing navigational reasoning capability for large vision-language models.\" European Conference on Computer Vision. Springer, Cham, 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive data set: The author contributes a large-scale goal-oriented UAV VLN data set, including 6-degree-of-freedom flight trajectories, providing a large number of training and evaluation resources for the research community.\n2. Real UAV simulation: The OpenUAV platform significantly improves the authenticity of UAV navigation tasks, supports continuous flight trajectories, diverse environments and realistic flight dynamics, and solves the limitations of previous UAV VLN research.\n3. Innovative benchmark task: The UAV-Need-Help benchmark introduces a challenge of practical significance, through different levels of help mechanisms, to cope with the challenges caused by the inherent dynamics of the aerial environment, obstructed views and changes in perspective.\n4. Hierarchical trajectory generation: The proposed UAV navigation large model (LLM) combines long-range trajectory prediction and fine-grained control, and can handle the hierarchical complexity in UAV navigation tasks.\n5. Parallelization and scalability: The platform supports efficient data collection and evaluation in parallel environments, which is crucial for large-scale UAV research.", "weaknesses": "Please see the Questions section for detailed improvement suggestions and questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULrgGjRCvZ", "reviewer": "Reviewer_7iwn", "summary": "The paper proposes a new UAV platform with several integrated scenes, continous action space and 6 DoF instead of previously 4. In addition they provide a VLN dataset together with a benchmark “UAV-Need-Help” together with an assistant that can provide guidance in three different levels. Lastly they have a model based on a LLM with a hirarchical strucutre that first predicts the target pose and then predicts the fine-grained path trajectory. The authors evaluate their model closed-loop and show qualitative results of the language predictions of the LLM.", "questions": "I am not an expert in the field of UAV but after looking at the related work it is still a bit unclear for me what the main contributions are and how it differentiates to existing work. It would be good if the authors could make it clearer what already exists and what is new. Secondly it would be really helpful to include more ablations on the importance of continuous actions or hierarchical predictions.\n\nMinor suggestions/ questions:\n1. Make sure that the abbreviations are introduced properly. E.g. line 016 is the first time you mention UAV, it would be good to have the long version when first using the short version.\n2. Human baseline: I would be curious how well a human performs in the L3 setting. Is the human baseline with a person who tried it several times and is trained or is it a “zero-shot” baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S.1. Complete set to research UAVs. The paper provides a platform/ simulation environment, dataset, benchmark and a baseline model. This opens up a lot of possibilities for the community to do research in this field.\n\nS.2. Baselines: Having multiple baselines for a new benchmark is important. I also appreciate the human baseline.\n\nS.3. Hierarchical prediction: The concept of hierarchical predictions is interesting and makes sense. (But missing ablation to show that hierarchical is superior to directly predicting the fine-grained trajectory).", "weaknesses": "**W.1. Comparison to existing work**\n\n1. The works seems to be quite similar to [AerialVLN](https://arxiv.org/abs/2308.06735) or CityNav. One difference mentioned is the DoF and the use of discrete actions. The authors of AerialVLN state “Although the simulator **supports flying towards any given direction and speed/distance**, we consider the eight most common low-level actions in UAVs:Move Forward, Turn Left, Turn Right, Ascend, Descend, Move Left, Move Right and Stop.”, which means in my understanding that they do support continuous actions. There dataset also includes language instructions. So I was wondering why this new simulator and dataset is needed and can’t be build on top of the existing one? It would be great if the authors could highlight unique features or improvements their simulator offers, and explain why building on existing platforms was not sufficient for their research goals.\n2. Difference to AirSim: AirSim does provide an API and a human control interface supporting PX4. In the paper it sounds like this is a contribution of the new platform OpenUAV (e.g. line 212 ff.). It would be good to make it more clear in the whole paper what is used from existing works and which parts are new in this paper.\n3. Difference to related work that combines UE + AirSIM: The authors state that several works “have combined UE with AirSim to develop high-fidelity imaging platforms. However, these platforms either suffer from low rendering precision or lack algorithmic support for continuous trajectory VLN tasks”. (line 149) Since the authors also combine UE with AirSim, it would be great if the differences to the existing works could be described in more detail. \n\n**W.2. Details of descriptions/ Clarity**\n\n1. Naming: There is already an OpenUAV platform (see[1]). It would be good to use an unique naming and also elaborate the differences.\n2. Scenarios: It would be great to have more details of the available scenarios. For me it is not clear what a scenario means in this context? Is it a different environment (e.g. forest vs. city) or is it similar to CARLA a specific event happening (see https://leaderboard.carla.org/scenarios/)? What are the 22 distinct scenarios available in this simulator? If the authors could provide a list or table of the 22 scenarios, including a brief description of each it would help a lot.\n3. Dataset splits: Figure 2 is great to get a rough overview of the objects but a detailed overview in the supplementary with all available objects and scenes and which are used for which split would be helpful.\n4. Figure 3 (minor): In the image it seems that the front view image is input to the Trajectory completion module but in the text the authors say “it takes the front-view visual token T_img” (line 386). It would be good to have this clearer in the figure.\n5. DAgger: What exactly is the teacher model? Can you provide more details?\n\n**W.3. Evaluation/ Benchmark**\n\n1. As I understood for all results an oracle path is necessary to generate the assistants. An additional mode without any assistants would be good to judge how hard the task is and in an ultimate fully autonomous setting this would be the wished setting I assume. The authors state “the inherent complexity and dynamic nature of aerial environments, with obstructed views and shifting perspectives, make basic target descriptions insufficient for UAV” (line 323). I am wondering if temporal input and a memory could help for this making the assistants unnecessary?\n2. The authors say “UAV trajectories are inherently continuous and difficult to decompose into discrete actions, leading to unrealistic navigation when such simplifications are applied” (line 081). They use (different to previous works) continuous action space. To validate this statement it would be good to have a discrete baseline and qualitative example of the shortcomings (which are solved by the continuous version).\n\n[1] H. Anand *et al*., \"OpenUAV Cloud Testbed: a Collaborative Design Studio for Field Robotics,\" *2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlChmgNDlC", "reviewer": "Reviewer_zwf5", "summary": "This paper introduces a UAV-based vision-and-language navigation benchmark built on the CARLA simulator. The simulator offers realistic environments and supports high-DoF control for UAV flight. In addition, the benchmark provides ground-truth human-controlled trajectories, trajectory descriptions summarized by a vision-language model (VLM), and assistive instructions for navigation evaluation. The paper also presents a vision-language model as a baseline for UAV-based vision-and-language navigation, achieving state-of-the-art performance on the proposed benchmark.", "questions": "- Why does the proposed method rely solely on the current observation? How does the model align visual history with the instructions? Additionally, the paper notes the use of `Llama-vid, Vicuna-7B, and EVA-CLIP`, similar to components used in another VLN paper [1]. A discussion comparing these approaches would be beneficial.\n\n- In Tables 2 and 3, the OSR is significantly higher than the SR (almost double). Given the definition of OSR, methods can achieve high OSR by extensively exploring the environment. A discussion or additional experiments are needed to address this observation.\n\n- Why not consider using existing LLM-based methods, such as [2], which employ waypoints in a manner similar to the proposed approach?\n\n[1] Navid: Video-based VLM plans the next step for vision-and-language navigation\n\n[2] NavGPT: Explicit reasoning in vision-and-language navigation with large language models", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is highly comprehensive, including a platform, benchmark and method technique for the VLN task in the domain of UAV. The authors have clearly significant efforts in completing this project.  This benchmark could help researchers investigate realistic UAV control under the task of VLN. Besides, the proposed LLM-based navigation framework shows strong performance among the baseline methods.", "weaknesses": "- This paper includes extensive content on the platform, benchmark, and method, but lacks many important details. For instance, `L398` mentions leveraging DAgger in a closed-loop simulation, yet no further experiments are provided to demonstrate its effectiveness.\n\n- In my view, the L1 assistant (guiding the UAV to the oracle path) provides an oracle instruction by directly issuing ground truth commands, which should significantly enhance navigation performance. However, the experiment results show only a minor increase in SR (such as 16.1 in the full set). \n\n- The visual encoding strategy does not account for historical information, as the navigation method relies solely on the latest frame `L373`, resulting in only 16+1 tokens for encoding visual data. This design neglects navigation history, raising the question of how navigation history can align effectively with the instructions.\n\n- The paper lacks video results to illustrate the ground truth trajectory and the method's performance on this platform. Such visual demonstrations would provide a more intuitive understanding of this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)