DivScene: Benchmarking LVLMs for Object Navigation with Diverse Scenes and Objects

Authors: Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman Pan, Yangqiu Song, Dong Yu
Primary area: foundation or frontier models, including LLMs
Keywords: Embodied AI, Object Navigation, Large Vision Language Models, LVLM, Imitation Learning

Abstract:
Object navigation in unknown environments is crucial for deploying embodied agents in real-world applications.
While we have witnessed huge progress due to large-scale scene datasets, faster simulators, and stronger models, previous studies mainly focus on limited scene types and target objects. In this paper, we study a new task of navigating to diverse target objects in a large number of scene types. To benchmark the problem, we present a large-scale scene dataset, DivScene, which contains 4,614 scenes across 81 different types. With the dataset, we build an end-to-end embodied agent, NatVLM, by fine-tuning a Large Vision Language Model (LVLM) through imitation learning. The LVLM is trained to take previous observations from the environment and generate the next actions. We also introduce CoT explanation traces of the action prediction for better performance when tuning LVLMs. Our extensive experiments find that we can build a performant LVLM-based agent through imitation learning on the shortest paths constructed by a BFS planner without any human supervision. Our agent achieves a success rate that surpasses GPT-4o by over 20%. Meanwhile, we carry out various analyses showing the generalization ability of our agent.

Decision: reject
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729511961473

thread_type:
official_review

<thread_object>
creator: reviewer_am4v

<content>
summary:
This paper presents a new benchmark for object-goal navigation as well as a LVLM-based baseline method. For data generation, this work adopts GPT-4 to generate diverse scene data and samples episodes accordingly. Then NatVLM is proposed as a baseline method, which tunes a LVLM using chain-of-thought. The authors compare NatVLM with other LLM or LVLM navigation methods on the proposed benchmark and demonstrates leading performance.

strengths:
1. This paper is easy to follow.
2. DivScene contains a large amount of houses of different types and object categories, which can serve as a good evaluation benchmark for open-vocabulary object navigaion.

weaknesses:
Although the proposed dataset is diverse, the experimental comparison is weak:
The authors claim in L68-73 "However, they still face a substantial domain gap between navigation tasks and the LLM training corpus (Lin et al., 2024). Moreover, those methods (Chen et al., 2023; Zhu et al., 2024) require a captioning model to generate textual descriptions of each scene, serving as the perception of the backbone LLMs. The captioning model might miss crucial information about objects and lead to suboptimal results, like spatial or color details.". If this is true, the authors should compare with state-of-the-art open-vocabulary object navigation methods [1,2,3] to validate this. Otherwise, only comparing with LLM and LVLMs are less meaningful.

Also, some details about the dataset need to be further clarified:
1. How does the objects arranged in the scenes? How to make sure their relationship is reasonable?
2. How to define scene type and what is the criteria for choosing them?

Moreover, the whole approach, including data generation and NatVLM, is too naive. There are many ways to exploit the ability of LLM/LVLMs for navigation [2,3] and a thorough comparison is required.

Typos:
Line 259: Similarly, we randomly select 27 houses from distinct scene types as the validation set. (validation-->test)

[1] Esc: Exploration with soft commonsense constraints for zero-shot object navigation. ICML 2023.

[2] Vlfm: Vision- language frontier maps for zero-shot semantic navigation. ICRA 2024.

[3] SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation. NeurIPS 2024.

questions:
See weakness. I hope my questions can be well addressed and more experimental results are required to support the authors' claim.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730192790813

thread_type:
official_review

<thread_object>
creator: reviewer_hdkj

<content>
summary:
The paper introduces a new dataset with diverse objects and scenes. The scenes are generated using Holodeck, and the trajectories are collected following the shortest path principle. The dataset contains 4,614 scenes across 81 distinct types. It also provides ground-truth actions along with explanation traces to support Chain-of-Thought (CoT) reasoning. The experimental results demonstrate the effectiveness of these explanation traces in enhancing model performance.

strengths:
1. The paper demonstrates that explanation traces are important for action prediction, providing useful insight for model design.
2. The authors utilize a postprocessing procedure to guarantee the data quality.
3. The authors provide the data examples and the source code of open-sourced API evaluation.

weaknesses:
1. Overclaim. The authors claim that they are the first to build a navigational agent based on LVLMs. However, to my knowledge, NaVid [1] has already introduced a navigation agent using LVLMs and deployed the model in real-world scenarios. In contrast, the current work operates entirely within synthetic scenes and trains its model on a single simulator, which limits the novelty and practical significance. This weakens the overall contribution, as the work does not extend beyond what NaVid achieved.

2. Missing Comparison with Existing Navigation Datasets. The manuscript lacks a proper comparison with existing navigation datasets. It is essential to clarify how this dataset differs from previous datasets in terms of data domain, quantity, and quality. Important details such as the number of scenes, object categories, and trajectory complexity in existing datasets should be explicitly mentioned. Without this information, it is challenging to assess the uniqueness or relevance of the proposed dataset.

3. Limited Technical Contribution. Sections 3, 4, and 5 primarily describe a standard pipeline for collecting navigation trajectories, which follows established methodologies from prior works. The paper should make a clearer distinction regarding its unique contributions in these sections to better articulate the novelty. Without this, the technical advancements of the current work appear minimal.

4. Insufficient Baseline Models and Ablation Study. The experimental section is limited, as the authors train only a single model based on Idefics 2 (8B) and compare it with zero-shot models. This narrow focus prevents a thorough evaluation. Including additional baselines, such as traditional reinforcement learning (RL)-based or imitation learning-based navigation models, would provide a more comprehensive comparison. Furthermore, the influence of different LLM backbones or LVLM pre-trained weights should be studied to offer deeper insights into how these components affect navigation performance.


[1] NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

questions:
1. Zero-shot transfer experiments are not convincing. I visualize the scenes in iTHOR, ProcTHOR, and AI2-THOR and find their styles are very similar, which means a marginal domain gap in your cross-domain setting. So comparing zero-shot models such as llava and qwen-VL with your fine-tuned models makes no sense. A more valuable comparison should be the in-domain experiments on iTHOR and ProcTHOR with your NATVLM.

2. Insufficient Literature Review on Object Navigation. The manuscript does not adequately cover prior research on object navigation, particularly recent works that explore navigation agents built with Vision-Language Models (VLMs). A comprehensive discussion of related works is essential for positioning this study within the existing literature. The omission of relevant research leaves a gap in understanding how the proposed approach compares to prior VLM-based navigation agents and whether it builds on or diverges from previous methodologies.

[1] VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models
[2] NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575335840

thread_type:
official_review

<thread_object>
creator: reviewer_yjpb

<content>
summary:
The authors present an open-vocab version of the closed-vocab object navigation. They introduce both a benchmark dataset for this setting, and demonstrate a new VLM-based method (NatVLM).


Benchmark:
* They generate 4k scenes using Holodeck, and aim for diverse generated scenes by using 81 scene types from MIT 365 along with LLMs to generate scene descriptions. They use 22k objects from Objaverse (looks like they subselect 5k somehow).
* They use AI2Thor to compute shortest paths and compute episodes for trianing. Presumably they also use aI2Thor for online evaluation

Baselines:
They present a few baselines, using existing vlms and a method NATVLM  (a finetuned idefics2). They show a few ablations of prompts in the NatVLM baseline. They show zero-shot transfer performance of these baselines on closed-vocab versions ProcThor and iThor.

strengths:
### Significance
Open-vocab version of existing closed-vocab object nav. Reasonably important problem with a large enough community that would be interested in this benchmark. 

### Originality
Seems sufficiently novel

### Quality
The authors do a good job of introducing a benchmark using existing tools in the community. They provide results for some reasonable baselines on this new benchmark, and show zero-shot performance on closed-vocab scenes (but still in procthor). Generally they do a very good job describing experimental settings, though some details are missing from the paper -- but would be present in a code release. E.g. which episodes were used for eval on procthor, ithor. The extensive appendix helps here.

### Clarity
Good references to related work, motivation is clear

weaknesses:
My main concern is that the paper does too many things for these experiments to sufficiently support. It reads like a task paper, and a dataset paper, and also a method paper.

1. As a method paper, the submission is missing comparisons to sota on closed-vocab benchmarks (e.g. sota in procthor, or on MP3D in habitat).
2. As a dataset paper: the submission could use more analysis of well these scenes approximate real-world settings -- should we use/trust this data
3. As primarily proposing the task of open-vocab object nav: I would like to see more of a breakdown of where current methods fail, and why this problem is hard

I think there is something here, but it’s not quite ready in the current form. I'll ask/recommend specific experiemnts these in the "questions" section, since I think they could make the submission stronger.

questions:
### As a method:
- In this case the paper would be a method for open-vocab object navigation, where they generate a dataset for the proposed method. But for an open-vocab method, I want to be able to compare its performance to SotA on existing object nav benchmarks, even if they are closed vocab. Only comparisons are to other VLM-based methods, which are not SotA on existing benchmarks. 
- You don't need to be the best in closed-vocab objnav. It's okay to break out the open-vocab and closed-vocab results separately in a table. But it would be helpful to show the reader how big the gap is. If there's a large gap, that's an even better motivation of why closed-vocab approaches do not work in real-world settings.
- If you do want to be the best on closed-vocab objnav, you don't need to show zero-shot performance but can finetune/anneal the model on the downstream dataset. Does the more general open-vocab object nav help them improve over sota?

### As a task: 
No breakdown on where the failures were. Why is open-vocab much harder than closed-vocab? E.g. how many of the failures were due to missing the object, vs navigation failure. That would allow this to tie into existing literature like 3d object detection (e.g. on scannet, mp3d, etc).

### As a benchmark dataset
I would like to see more analysis of the decisions made for this benchmark, and how well models trained on this data transfer to real-wold settings. 
* Existing zero-shot transfer experiments are to other benchmarks also in AI2Thor. So largely this concerns whether the model will work in other ai2thor scenes from procthor/ithor, and using those closed-vocab objects instead of the ones from objaverse
* I would love to see generalization to some obj nav in other simulators and using more real-world data (e.g. ARKit or MP3D, which the authors mention but do not compare to).
* Is generating new scenes Holodeck better than using an existing artist-deigned dataset like Structure3d, which has 3.5k scenes? ARKit which is 4500 real ones and already has a bunch of CAD annotations with cad-estate?
* Is 4k scenes the right number?  Would want to see an experiment when training on subsets of the data to show performance isn't saturating (e.g. 1%, 10%, 50%, 100% of scenes).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645913041

thread_type:
official_review

<thread_object>
creator: reviewer_441m

<content>
summary:
his paper tackles the challenge of object navigation in varied environments by introducing DIVSCENE, a dataset with over 4,600 scenes across 81 types. Building on this, the authors developed NATVLM, a navigation agent fine-tuned on a large vision-language model. NATVLM uses Chain of Thought reasoning to help it understand navigation steps better, which improves its accuracy and generalization in diverse settings. The experiments show strong performance against other models and confirm the agent's adaptability across different datasets.

strengths:
1. This paper introduces the DIVSCENE dataset, which is a substantial addition to the field of embodied AI. Unlike existing datasets that are often limited to a narrow range of scenes or objects, DIVSCENE covers 81 diverse scene types with over 22,000 object types, enhancing the scope for studying generalizable object navigation.
2. The method is rigorous, with extensive experiments that include comparisons to baseline models (both open-source and closed-source) and ablation studies to evaluate the impact of CoT reasoning. The experiments provide solid evidence that NATVLM significantly outperforms existing methods in diverse navigation scenarios.
3. The paper is well-organized and presents the main ideas in a clear and logical structure.

weaknesses:
1. While DIVSCENE provides a broad range of synthetic environments, the reliance on simulated platforms (like AI2THOR) limits the real-world applicability of NATVLM. Simulated environments often lack the unpredictable variability and physical constraints found in real-world scenarios. Expanding experiments to include real-world datasets or environments would strengthen the evidence for the model’s generalization abilities and applicability. Including a discussion on the challenges of transferring synthetic-trained models to real-world settings would also be beneficial.
2.  The manual creation of CoT explanation traces for training NATVLM may limit scalability and automation in more complex or dynamic environments.
3. While DIVSCENE offers diverse scenes and objects, the paper lacks a detailed analysis of the scene complexity and its impact on navigation performance. For example, understanding how NATVLM performs in cluttered versus minimal scenes, or in spaces with varying levels of visual occlusion, would add depth to the evaluation. Such insights could help clarify the scenarios where NATVLM’s CoT-based reasoning is most beneficial and identify cases where additional model improvements may be needed.
4. Although the paper emphasizes the effectiveness of NATVLM, it does not address the computational cost of training and deploying the model, especially given the integration of large-scale LVLMs and manually designed CoT traces. Providing a comparison of training and inference times with baseline models would offer readers a clearer understanding of the model’s practicality.

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310900218

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank all reviewers for their detailed and constructive feedback on our manuscript. **We are delighted that they acknowledge our pioneering contributions and recognize the efforts and strengths of our work:**
1. Open-vocab object navigation is important (Reviewer 441m, YJpB); 
2. The method is rigorous, with extensive experiments and ablation study (Reviewer 441m, YJpB); 
3. The paper is well-organized with logical structure and clear motivation (Reviewer 441m, YJpB, AM4v); 
4. The work is very novel (Reviewer YJpB); 
5. Insightful result of CoT traces (Reviewer HDKj); 
6. Open sourcing (Reviewer HDKj); 
7. Important benchmark with good quality (Reviewer HDKj, AM4v).

**During the rebuttal period, we provide detailed and rigorous responses to every posted comment. The Reviewer YJpB and AM4v raised the score from 5 to 6 and from 3 to 6, respectively, based on our supportive explanations. In those responses, we conduct a series of new experiments to solve the reviewers' concerns:**
1. A thorough comparison with numerous baselines on the photorealistic dataset HM3D.
2. A statistical comparison of our dataset DivScene with previous ones.
3. An analysis of the impact of scene complexity on navigation performance.
4. An analysis of the impact of varying numbers of scene and target object types on the navigation performance.

For the presentation of our work, we want to highlight that our primary objective is to benchmark the current progress of end-to-end LVLMs on the open-vocab object navigation task. Both the dataset and method sections are deliberately designed to serve this central goal, ensuring a cohesive and focused contribution.

Once again, we sincerely thank the reviewers for their thorough evaluation of our manuscript. Their valuable insights have significantly contributed to improving our work. Our comprehensive revisions and responses have successfully addressed the raised concerns and validated the significance of our research contribution.

Best Regards,

Authors from Submission 12096.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new benchmark for object navigation, with the goal of increasing diversity of scenes (4.6k) and types (81) leveraging prior GPT-based environmental description generation. The authors then benchmark various LLMs/VLMs on this task, which are trained to imitate the shortest path simulation-based ground truth. The reviewers appreciated the effort in generating the dataset, the development of a LLM/VLM-based approach, and the experimental results. However, the paper is a somewhat confused one, containing a new benchmark (without a clear comparison to prior benchmarks and why it is needed, HDKj) as well as a method (which is not compared to existing approaches to this problem, even if prior methods are closed-set, HDKj/YJpB but also open-vocabulary ones, AM4v). This results in a situation where it is difficult to understand where current methods fail and why the problem is hard (YJpB). Additional concerns included the lack of results in more real-world photorealistic settings, lack of details about the environment generation, etc. The authors provided a rebuttal addressing some of these aspects, comparing the benchmark statistics in a table (though only to THOR-based ones, which is a large limitation as HSSD, etc. are not compared to), comparing the method to other zero-shot methods on HM3D, and scale of performance with respect to object/scene diversity. Some of the reviewers mentioned most of their concerns were addressed.

  Overall, this paper is quite borderline; while three reviewers have positive scores, some still acknowledge the severe limitations of the paper. Some of this stems from the fact that the initial version was severely lacking given that it just proposed a new benchmark without a strong comparison/situation/justification compared to prior ones, and then proposed a method that was not compared to existing state-of-art. While the rebuttal did situate with respect to THOR-based benchmarks, and also did compare to state-of-art methods on HM3D, leading to raised scores, it is still a splintered paper that does not justify why this benchmark is better/harder and how existing methods fail at it (including through analysis), before presenting an approach that addresses those limitations. As a result, I do not recommend acceptance as significant effort is needed to making the paper both coherent and rigorous in terms of the contribution.