CityNav: Language-Goal Aerial Navigation Dataset Using Geographic Information

Authors: Jungdae Lee, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, Nakamasa Inoue
Primary area: datasets and benchmarks
Keywords: City-scale 3D Vision, Aerial Navigation, 3D Vision and Language

Abstract:
Vision-and-language navigation (VLN) aims to guide autonomous agents through real-world environments by integrating visual and linguistic cues. Despite notable advancements in ground-level navigation, the exploration of aerial navigation using these modalities remains limited. This gap primarily arises from a lack of suitable resources for real-world, city-scale aerial navigation studies. To remedy this gap, we introduce CityNav, a novel dataset explicitly designed for language-guided aerial navigation in photorealistic 3D environments of real cities. CityNav comprises 32k natural language descriptions paired with human demonstration trajectories, collected via a newly developed web-based 3D simulator. Each description identifies a navigation goal, utilizing the names and locations of landmarks within actual cities. As an initial step toward addressing this challenge, we provide baseline models of navigation agents that incorporate an internal 2D spatial map representing landmarks referenced in the descriptions. We have benchmarked the latest aerial navigation methods alongside our proposed baseline model on the CityNav dataset. The findings are revealing: (i) our aerial agent model trained on human demonstration trajectories,  outperform those trained on shortest path trajectories by a large margin; (ii) incorporating 2D spatial map information markedly and robustly enhances navigation performance at a city scale; (iii) despite the use of map information, our challenging CityNav dataset reveals a persistent performance gap between our baseline models and human performance. To foster further research in aerial VLN, we have made the dataset and code available at https://anonymous.4open.science/w/city-nav-77E3/.

Decision: reject
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729510256828

thread_type:
official_review

<thread_object>
creator: reviewer_jguw

<content>
summary:
The paper proposes a novel dataset for aerial VLN (vision-language-navigation) called CityNav. It consists of about 32k pairs, where the first entry = natural language description of goal, and second entry = human demonstration of reaching the goal. The latter is collected using Mechanical Turk in a newly developed web-based 3D simulator. The dataset depicts photorealistic urban scenes. In addition to the dataset, several learning-based baseline methods are evaluated (and one is specifically developed within the paper), to provide a suite of baseline results against which future researchers can compare. The various methods are evaluated in two settings: (i) imitation learning based on shortest path trajectories; (ii) same but based on human trajectories. It is shown that results in general get better in the setting (ii).

strengths:
* The main and strongest contribution is (quite naturally) the dataset itself, CityNav. It is a factor 4x or so larger than the 2nd largest similar dataset, and with intriguing advantages relative to recent similar works:
    - Based on real-world point clouds (although this work is based on SensatUrban from 2022, so it is intself not a contribution).
    - Takes into account realistic challenges related to 3D geometries in real UAV missions.

* Many relevant experiments to showcase the strength of the proposed dataset are included, e.g.
    - Several imitation learning-based baselines are evaluated / developed, to give a strong suite of baseline results against which future work can be compared.
    - Great to see the various results with methods trained based on either shortest path or human demonstrations, which showed human demonstrations lead to better results overall.
    - Several relevant "additional results", including proper ablations of the proposed MGP method, examination into the effect of training size (interesting to see that the size is more important when training based on human trajectories vs when trained on shortest path), evaluations on different difficulty levels, and some results on challening conditions such as gnss-disturbed environments.
    - Good that several relevant and widely used metrics were used.

* I think the stuff written around Line 311 on "Quality control" was good and made it feel like the dataset was carefully designed. For example, the rigorous selection of which human trajectories were kept in the dataset, etc. Also, speaking of the dataset, I liked the plots that showed some key characteristics about the data, e.g. Fig 4 e-f.

* The anonymous webpage had great visualizations which further helped me understand the task setup.

* Great that the code was accompanied with the submission, it further strengthens the validity of the work.

weaknesses:
* Some things were not quite clear to me regarding the use / not use of maps. So the proposed MGP method seems to use a map (but see questions below, I might have misunderstood), and so the the human annotators. Meanwhile, the other learning-based methods do not. I'm not quite sure I get the reasoning about why maps make sense to use as input, or in particular I dont see the realism behind it, e.g. if considering gnss-free settings. Now that said, it's fine IMO to have such results, but I found the "focus" on these map-based things to be a bit.. "off" relative to the applications I envision.

* Adding to the above, it'd be great to see the dataset accompanied by human trajectories that were collected by humans that dit NOT get to use maps. Perhaps training on such trajectories is beneficial especially for learning-based methods that do not have maps. (I understand that this is beyond the scope of addressing within this rebuttal, but still adding as a weakness for now.)

* I think some results on RL-based approaches would significantly improve the suite of learning-based baselines. Currently, only imitation learning-based methods are shown. In the RL case, some appropriate reward must of course be specified.

* To the best of my knowledge, no inference runtimes (e.g. time per action) were mentioned for the various methods, and I think it should be reported alongside the other metrics.

* I think the paper would be strengthened by explicitly showing results (both for learning-based methods and for the human trajectories) at category-level. For instance, it can provide insights about whether the class imbalance (that e.g. much more "Building" than "Parking lot") affects results in any way, e.g. if results get worse on "Parking lot" than "Building".

* Maybe I missed it but I didn't see the distribution geographically of which and how many cities / urban areas were included. This should be included in the paper.

* Misc (do not affect my rating):
    - I don't quite agree that the aerial VLN task is in general more challenging than ground VLN, necessarily as stated on Line 170 (since ground based counterparts have more challenging with occlusions etc). Please consider rephrasing a bit.
    - I think Fig 4 can be improved by mentioning what data splits are used in the various subfigures, e.g. if some only based on Train, if some based on overall union, etc.
    - I think that the shortest path "oracle method" can be reported below "Human" in Table 2, it's a nice upper-bound method to compare the rest with. Also, speaking of tables, I think it'd be great if the captions included what splits were used in tables such as Table 3 (e.g. Test unseen and so on -- i know this is mentioned in the main text, but still helpful).

questions:
* Please try to address as many of the weaknesses as possible but I understand that time is a very limiting factor so here are some priorities:
    - Is there a particular reason why RL-based method(s) were not used, i.e. why only imitation learning?
    - What are runtimes per action / trajectory during inference?
    - Is the proposed MGP method using actual maps as input or are they somehow generated by the model? If the former, do you not agree that it may be unrealistic to provide such a map, e.g. in GNSS-free settings?
    - Did I miss it, or did you not include geographical distribution of the various cities / urban areas of the dataset? If not included, please add to cam-ready, and also please comment on the distribution here.

* I did not get this part in the box on Line 295: "Note that you cannot replace the marker." <-- It seems to me that this constraint leaves room for lots of mistakes that are unnecessary. Why not include an "undo" button..?

* The 170+ MTurk annotators, where were they distributed across the globe? Were the randomly selected?  Also, were exactly one person used as demonstrator per "game", or multi-annotator setup used?

* Why were the easy-medium-hard thingys mentioned as "-" on "Train" in Fig. 4a? Why could they not be listed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730304160289

thread_type:
official_review

<thread_object>
creator: reviewer_nnad

<content>
summary:
The paper introduces CityNav, a dataset designed for language-guided aerial navigation in real-world, photorealistic 3D city environments. CityNav encompasses over 32k natural language descriptions paired with human-generated trajectories. The dataset leverages 3D scans of actual cities, incorporating a web-based 3D flight simulator that syncs with world maps for trajectory collection. This dataset provides a diverse set of realistic trajectories and linguistic descriptions. Through their analysis, the authors reveal that models trained with human demonstrations substantially outperform those relying on shortest path trajectories, and the integration of 2D spatial maps markedly enhances navigation performance. However, there remains a significant performance gap between automated models and human navigators.

strengths:
1. The paper introduces CityNav, a new dataset for outdoor aerial vision-and-language navigation in real-world urban settings.
2. The dataset includes high-quality data with detailed annotations.
3. The authors provide access to the dataset and the code, enhancing reproducibility and facilitating further research.

weaknesses:
1. The analysis of the experiments is not entirely convincing, as the discussion in the paper lacks explanations for the underlying reasons of the observed performance. This insufficiency makes it challenging to fully assess the effectiveness of the proposed dataset and benchmark.
2. The authors do not adequately explain the practical value of the Aerial VLN task that the CityNav dataset is designed to support. Without a clear connection between the Aerial VLN and its potential applications, the significance of the proposed dataset remains unclear.

questions:
1. Could the authors provide a more detailed analysis of the differences between human and shortest-path trajectories? In the abstract, the authors mention that "human demonstration trajectories outperform those trained on shortest path trajectories by a large margin." I am curious about the reasons behind the superior performance of human demonstrations. Specifically, do human demonstrations provide more information within the trajectory compared to shortest paths, which might only have start and end points? If this is the case, the comparison lacks persuasiveness because the information provided by human demonstrations is much denser and more extensive than that of the shortest path. 
2. The authors do not provide a convincing explanation of the value of the Aerial VLN task when introducing it. From an academic perspective, what distinguishes Aerial VLN from other Indoor VLN tasks that warrant attention? Could the authors please provide a more detailed comparison between Aerial VLN and Indoor VLN, highlighting specific challenges or opportunities unique to the aerial domain? 
3. Meanwhile, could the authors please discuss real-world scenarios where language guidance might offer advantages over GPS, perhaps drawing on existing literature or use cases? Specifically, why opt for language guidance over traditional positioning systems like GPS, especially when maps are available? 
4. Furthermore, I am confused about the experiments in disaster scenarios. The additional experiments mentioned in the appendix, which assume the failure of GPS systems during major natural disasters, further raise concerns about the practical application of this task. It is questionable whether GPS systems would indeed be significantly disrupted by such events (±100m). Moreover, the simulation assumes that the map-based VLN system remains stable under such conditions. What evidence supports these disaster scenario simulations, and do they reflect realistic conditions, or do they indicate a lack of real-world applicability for the task? Could the authors please provide evidence or citations supporting the GPS disruption levels used in their disaster scenarios, and explain how their VLN system's stability was modeled in these conditions?

If the authors are willing to address my questions, I would be very pleased to continue this discussion with them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730338721497

thread_type:
official_review

<thread_object>
creator: reviewer_bznw

<content>
summary:
The paper introduces CityNav, a new 3D simulator and dataset for language-guided aerial navigation, filling the resource gap in city-scale aerial navigation. The dataset includes 32,000 descriptions paired with human trajectories and proposes a baseline model using spatial maps. However, high-altitude operations may impact the detection of small objects, and the lack of environmental interaction raises concerns about realism. Moreover, more balanced evaluation and clearer experimental setups are needed.

strengths:
1. CityNav introduced a new web-based 3D simulator designed to collect and test language-guided aerial navigation tasks.
2. The CityNav dataset includes 32,000 natural language descriptions paired with human demonstration trajectories. This dataset fills the resource gap in the aerial navigation domain, providing valuable resources for city-scale aerial navigation research.
3. By incorporating spatial map information, the baseline model provides a foundation for future city-scale aerial navigation tasks, exploring how to effectively utilize linguistic and visual cues for navigation.

weaknesses:
1. The prevalence of forward and down actions suggests that the drone operates at relatively high altitudes, which may hinder object recognition and make it challenging to observe smaller objects, such as vehicles. 
2. The paper mentions a lack of interaction with environmental objects during navigation, indicating that the model may do not account for realistic challenges like collision. 
3. The use of the landmark map as an additional prior for the MGP model raises concerns about fairness in evaluation. This additional information might skew the comparison with models that do not utilize such priors.  
4. Several experimental concerns are raised in the study, which will be detailed in the questions.

questions:
1. The majority of annotated actions are forward and down. When the drone is at a high altitude, does the annotated data make it difficult to detect small objects like vehicles?
2. Were collisions with environmental objects considered during data collection and model evaluation, particularly in the shortest path annotations?
3. Why do Seq2Seq and CMA models perform better under the SP setting compared to the HD setting in Table 2?
4. Why is the Navigation Error (NE) difference between different difficulty levels in Table 3 so small?
4. Does the landmark map give the MGP model an unfair advantage during evaluation? Would introducing similar priors to other models create a more balanced comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732417807

thread_type:
official_review

<thread_object>
creator: reviewer_6nak

<content>
summary:
The manuscript introduces CityNav, a comprehensive dataset designed for language-guided aerial navigation using real-world 3D urban environments. This dataset addresses a notable gap in the field by enabling aerial vision-and-language navigation (VLN) research with realistic data. CityNav includes over 32,000 natural language descriptions paired with human demonstration trajectories, collected through a novel web-based 3D flight simulator. The paper presents benchmark results for navigation models, highlighting significant performance enhancements through the use of a map-based goal prediction model.

strengths:
1. The introduction of CityNav fills an important void in aerial VLN research, offering a large-scale, realistic dataset with geo-referenced human trajectories.

2. The dataset's scale and the use of actual 3D scans of urban areas significantly improve realism compared to synthetic or satellite-only data.

weaknesses:
1. The current dataset lacks dynamic elements and agent-object interactions, such as moving vehicles, pedestrian, the varying light, etc., reducing applicability to real-world, complex urban settings.

2. The generation of the dataset seems very engineering and laborious, where the instructions and trajectories are obtained by MTurk. The dataset is mostly based on existing city-scale point cloud data from SensatUrban. Actually, SensatUrban just contain the images from some view of the field and sparse point clouds, while not images from each view as NeRF and Gaussian Splatting. In this case, view rendering works should be done to make the dataset more applicable. Besides, I wonder the gap between the simulation and real flight.

3. This work just implement a simple baseline, which lacks novelty and insight about the special challenges in this field. In my view, the object on the street is structural, such as road, cars, pedestrains, etc, and these objects are very small. During flight, the geographical objects are captured from differnet views and distance, which increase the difficulty to recognition. In this case,  more specific and innovative method should be proposed.

questions:
please address my concerns in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732434774284

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Comment by Authors

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. We have carefully addressed the points raised by the reviewers, particularly **potential misunderstandings (Reviewers 6nAK and nnaD)**, those concerning **the contribution of using the navigation map (Reviewer BzNw)** and **the advantages of using human demonstrations (Reviewer nnaD)**, through additional experiments. 

Please let us know if we have addressed your concerns or if you have additional feedback or suggestions. We greatly appreciate your time and effort and look forward to the discussions.

We also appreciate the many recommended improvements. Based on your suggestions, we have uploaded a new paper version that incorporates most of the reviewer's comments:
* **[6nAk]** Including a discussion on the visibility for distant objects in Section 5.
* **[BzNw]** Adding results for the baselines with the navigation map in Appendix D.
* **[BzNw]** Adding images of the aerial agent’s view in Appendix B.
* **[nnaD]** Adding analysis of the human and shortest-path trajectories in Appendix B.
* **[JGUw]** Adding the geographical distribution of cities in Section 3.
* **[JGUw]** Revising some wording in Section 2.
* **[JGUw]** Adding category-level performance in Appendix D.

Best regards,

Authors of Submission 9167
</content>
</thread_object>

</thread>
<thread>
creation date:
1733027150152

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate the significant effort all reviewers have dedicated to evaluating our work. Additionally, many thanks to the reviewers who provided thoughtful and constructive comments on our initial responses. 

Regarding the follow-up questions from **BzNw** and **nnaD**, we have carefully reviewed them and provided detailed responses, which we kindly invite you to examine. For the question from **6nAk**, we believe it has been sufficiently addressed in our initial reply, and we encourage you to refer to that section for further details. 

Through this constructive discussion, we believe that all concerns raised have been resolved. We hope the reviewers’ fair and balanced evaluation, based on the collaborative and insightful discussion we have shared. We are confident that our research offers valuable contributions to the community and helps advance the field.

Best regards,

Authors of Submission 9167
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary and Recommendation for Rejection

---

## Strengths:
1. **Novel Dataset Contribution**:
   - The **CityNav dataset** introduces a city-scale language-guided aerial navigation dataset using realistic 3D urban environments.
   - Includes 32,000 natural language descriptions paired with human demonstration trajectories collected via a novel web-based 3D simulator. 

2. **Benchmarking and Analysis**:
   - Offers baseline models that incorporate 2D spatial map representations, with results highlighting the importance of map information for navigation performance.
   - Conducts analyses of human versus shortest-path trajectories, emphasizing the advantages of human demonstrations in complex navigation tasks.

3. **Community Impact**:
   - Promises public availability of the dataset and code, fostering reproducibility and future research.

 
---

## Weaknesses:
1. **Limited Practical Relevance**:
   - The real-world application of language-guided aerial navigation tasks remains unclear, particularly given the availability and reliability of GPS-based systems.
   - The task design and justification for using language-based descriptions over GPS signals lack strong practical motivation.

2. **Baseline and Methodological Gaps**:
   - The proposed baseline model (MGP) relies heavily on map-based priors, which may skew evaluations and comparisons with other models that lack similar inputs.
   - The absence of reinforcement learning (RL)-based baselines and insufficient exploration of advanced techniques weaken the methodological contribution.

3. **Dataset Limitations**:
   - The dataset does not account for dynamic elements like moving objects or environmental interactions, reducing its applicability to real-world urban navigation scenarios.
   - Generated data relies heavily on existing datasets like SensatUrban, which primarily consists of point cloud data, limiting the dataset's originality.

4. **Unconvincing Experimental Analysis**:
   - Analysis of the differences between human and shortest-path trajectories lacks depth and fails to robustly explain the observed performance gaps.
   - Results across difficulty levels (easy, medium, hard) show minimal variation, raising concerns about task diversity and evaluation robustness.

5. **Ethical Concerns**:
   - The use of real-world geographic data raises potential privacy and licensing issues, as flagged by one reviewer.
 
---

## Authors' Mitigation:
1. **Expanded Analysis**:
   - Authors added detailed comparisons of human versus shortest-path trajectories and conducted experiments to validate the role of map information.

2. **Enhanced Dataset Presentation**:
   - Included additional data visualizations, geographic distribution details, and annotations to improve dataset transparency.

3. **Addressed Practical Concerns**:
   - Provided theoretical use cases for language-guided navigation, including search-and-rescue operations in disaster scenarios and privacy-driven applications.

---

## Remaining Weaknesses:
1. **Insufficient Practical Justification**:
   - The use of language-based navigation remains unconvincing when GPS systems could offer a more efficient and straightforward solution for the described tasks.

2. **Dataset and Task Limitations**:
   - The dataset lacks dynamic or real-world complexity, reducing its relevance for practical applications.
   - The absence of diverse and challenging navigation scenarios limits the dataset’s potential impact.

3. **Baseline Model Dependence**:
   - The reliance on map-based priors for the baseline model undermines its fairness and generality, making comparisons with other models less meaningful.

4. **Ethical Concerns**:
   - Privacy and licensing issues related to real-world geographic data remain unresolved.

---

## Decision Justification:
While the **CityNav dataset** introduces a novel contribution to aerial VLN research, its practical relevance and methodological impact are limited:

1. **Scope and Significance Misalignment**:
   - The task design lacks clear practical utility, particularly given the availability of more reliable and efficient navigation systems like GPS.

2. **Methodological Weaknesses**:
   - The absence of advanced techniques like RL-based approaches and reliance on map-based priors limit the paper’s technical contributions.

3. **Dataset Constraints**:
   - The dataset's reliance on static environments and point cloud data reduces its applicability to real-world scenarios, undermining its broader impact.

4. **Unaddressed Concerns**:
   - Ethical and practical concerns raised by reviewers remain inadequately addressed, further weakening the submission.

Given these considerations, I recommend rejecting the paper. While it provides a foundation for future research, substantial revisions and stronger justifications are required to meet the standards of a top-tier conference like ICLR.