ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination

Authors: Xinxin Zhao, Wenzhe Cai, Likun Tang, Teng Wang
Primary area: applications to robotics, autonomy, planning
Keywords: robotics, visual navigation, vision-language model, scene imagination

Abstract:
Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability to accomplish long-horizon daily tasks. Many recent approaches use Large Language Models (LLMs) for commonsense inference to improve exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry layout only by texts. Both are important for making rational navigation decisions. In this work, we seek to unleash the spatial perception and planning ability of Vision-Language Models (VLMs), and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual navigation tasks in a mapless manner. We achieve this by developing the imagination-powered navigation framework ImagineNav, which imagines the future observation images at valuable robot views and translates the complex navigation planning process into a rather simple best-view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where2Imagine module, which is distilled to align with human navigation habits. Finally, to reach the VLM preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open-vocabulary object navigation benchmarks demonstrates the superiority of our proposed system.

Decision: accept (poster)
Reviewer scores: 3, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730710154049

thread_type:
official_review

<thread_object>
creator: reviewer_rhxu

<content>
summary:
This paper introduces ImagineNav, a novel framework that leverages VLMs for open-vocabulary object navigation without relying on explicit mapping or localization. The framework acts on a discrete action space spanning different views, then predicts candidate relative poses from the current observation, then uses NVS to generate images of those poses, and lastly uses a VLM for planning. The problem is posed as a best-view image selection problem for the VLM. Empirical experiments on challenging open-vocabulary object navigation benchmarks demonstrate that ImagineNav outperforms existing methods.

strengths:
The paper is mostly clear and effective in conveying its points. There is an effective set of baselines indicating thorough experimental evidence but the addition of error bars would be helpful to determine significance. There are also effective sections discussing the failed and successful trajectories and clear ablations of the imagination module and the high-level planning module. The originality of having NVS methods apply to scene-level settings is interesting.

weaknesses:
I think the main novelty within the method is the where2imagine module because prior work has used VLMs for high-level planning and NVS methods. I think the underlying claim is that waypoint as an intermediate representation (to predict poses and images with NVS) is a better reasoning intermediate than other intermediate representations and better than learning an end-to-end model. I think further investigating this claim could further add novelty to the paper as it would tackle a more fundamental question of are modular systems better than end-to-end models for object search and navigation. I would also try to rewrite the conclusion and portions of the introduction for clarity (paragraph 2). Lastly, the discussion of successful and failed trajectories are useful but I would like to see how to address those failed trajectories within the framework.

questions:
Why do you think that waypoints are the correct intermediate representation for navigation and object search?
What is the point of having images from NVS as it limits performance and rather uses the embeddings to train a navigation policy?
What do you think are the limitations of the VLM high-level planning?
How can you get the VLM to learn online from additional data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730749620707

thread_type:
official_review

<thread_object>
creator: reviewer_va7n

<content>
summary:
This work presents an innovative framework for mapless visual navigation that utilizes Vision-Language Models (VLMs) to streamline open-vocabulary navigation. Unlike conventional approaches that depend on mapping and text-based planning, this method relies solely on RGB/RGB-D inputs, redefining navigation as a task of selecting the best view image. Using the Where2Imagine module, it generates potential viewpoints based on human-like navigation tendencies, enabling the VLM to select the most suitable view to locate target objects. The NVS module then generates potential views of the next waypoint. This approach results in efficient paths without mapping and enhances success rates on standard benchmarks.

strengths:
1. This work introduces mapless navigation through the prediction of future 3D waypoints, generating possible image observations at these waypoints, and selecting the most promising next waypoint with VLM. The method is well-motivated and thoroughly described, making it feasible for the community to replicate the results.
2. The paper includes an extensive experimental evaluation of the HM3D and HSSD benchmarks, with the proposed method achieving notable improvements over baselines on the challenging HM3D and HSSD benchmarks.
3. A comprehensive ablation study is provided in Tables 2, 3, 4, and 5, highlighting the significance of different components within the proposed pipeline. Showing the effectiveness of using visual prompting and novel view thesis, as well as the waypoint prediction.
5. Figure 5 also offers some failure cases to help readers understand the method’s limitations.

weaknesses:
1. The paper claims that object detection, segmentation, and mapping modules increase computational demand for robots. However, the proposed method introduces a computationally intensive novel view synthesis module to generate imagined observations. A comparison of computational load would strengthen this claim.

2. The paper’s organization could be improved for clarity. The text references to Tables and Figures are sometimes distant from the actual tables or figures; repositioning these elements could improve the flow and clarity of the paper.

3. Although GPT4o-mini is a robust VLM model, comparisons with recent open-source VLMs featuring 3D spatial understanding would enhance this work, such as:

* Spatial VLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities

* SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

questions:
1. While the provided failure case intuitively demonstrates a failure mode of the method, it would be valuable to include an approximate distribution of failure modes, such as how many failures are due to inaccurate imagined novel views.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730820899999

thread_type:
official_review

<thread_object>
creator: reviewer_4kmy

<content>
summary:
This paper introduces a framework called ImagineNav, designed to enable effective visual navigation for robots in a mapless, open-vocabulary setting. ImagineNav leverages Vision-Language Models (VLMs) with on-board RGB/RGB-D camera inputs, without complex mapping and localization procedures. Instead of traditional planning approaches, it translates the navigation task into a series of "best-view" image selection problems, where the VLM selects optimal viewpoints based on imagined future observations. This approach views navigation as a simplified selection problem and showcases the potential of imagination-driven guidance for robotic systems.

strengths:
1. This paper proposes a relatively novel navigation method: using novel view synthesis as a form of "imagination" for indoor navigation.
2. Due to the inclusion of a VLM and diffusion model, this work achieves a higher success rate in open-ended tasks.
3. The writing in this paper is clear, the illustrative figures are accurate, and the explanations of the proposed framework are well-articulated.

weaknesses:
1. There are some concerns about the imagination module. The ImagineNav framework relies on the diffusion models for novel view synthesis. However, these could lead to some mistakes e.g. non-existing objects in the scene.  This could lead to incorrect navigation decisions by the VLM and reduce the overall success rate.
2. A small concern about this approach is its performance in multi-room or occluded scenarios. The use of human habits without interactive perception could cause the robot to become trapped in local optima, preventing it from locating the target.
3. The framework seems to only rely on immediate views for navigation decisions, without fully utilizing historical information from the navigation process. This lack of memory may limit the robot's ability to explore effectively and plan global paths over long distances or in intricate environments.

In summary, the method is novel (only to me, but if other reviewers show some related work, I will defer it). But the quality of the generated novel view (especially the emergence of non-existing objects) is concerned. If the author can explain more about this, I would increase the score.

questions:
No extra problems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730916420467

thread_type:
official_review

<thread_object>
creator: reviewer_j4ao

<content>
summary:
This paper proposes "ImagineNav", a mapless navigation framework for robots using vision-language models (VLMs) to perform object search in open environments. It replaces traditional perception + mapping and text-based LLM-based planning with a pipeline where future scene views are "imagined" using novel view synthesis and analyzed by a VLM to choose optimal next waypoint. The candidate waypoints are generated by the Where2Imagine module, which is learned from human-like navigation patterns. Tested on benchmarks like HM3D and HSSD, ImagineNav significantly outperformed baselines in terms of success rate and success rate weighted by path length.

strengths:
The system design significantly reduces complexity of an open-world object-goal navigation pipeline, and better integrates sensor observation with large models.

weaknesses:
There are 3 fundamental limitations of this approach
- First, there is no way to guarantee that the diffusion model used to synthesize the novel views understands object permanence. How often does it hallucinate non-existing object or leave out objects that should be there? How does the rest of the pipeline deal with this?
- There is no way to ensure the waypoints generated by Where2Imagine is reachable, especially in out-of-distribution scenarios. The method still needs sine kind of local map for low-level path planning. Some tests on real robot or new environments will erase this concern.
- The method does not deal with the ambiguity of object reference. For example, navigate to the chair (example used in Fig. 2) is very ambiguous as there might be many chairs in the environment.

questions:
- Some details are very unclear, especially on the novel view synthesis (i.e. future view imagination). What data is used to train the diffusion model? Is it in or out of distribution for indoor navigation? How about resolution and the speed? 
- More qualitative visualizations of the environments, imagined views, waypoint distribution would be nice.
- The structure of the VLM analysis output in Fig. 3 and Fig. 4 are inconsistent. Which one is used in evaluation?
- Why is the success rate lower with NVS model added in Table. 2 (row 3 vs 5)? More explanation is needed.
- How does this method compare with end-to-end approaches like SPOC [1]?

[1] Ehsani, Kiana, et al. "SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732343170398

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for all the reviewers.

comment:
Thank you to all the reviewers raising crucial questions and suggestions for this paper. During the rebuttal period, we follow your suggestions and try to deal with your concerns from the following aspects:

(1) We quantitively analyze the failure reasons and visualize the ratio of different reasons, including the non-existence of objects issue caused by imagination in response to the Reviewer **j4ao , 4kMy and Va7n.** According to our analysis, although the imagination module can generate non-existent objects (at the ratio of 1331/4926 with 100 episodes), the ratio of failed episodes caused by this reason is only 3/100.  **As the agent stop flag is determined by object detection module on the real observed images,** the imagination of the non-existent object will not significantly influence the success rate.

(2) We conduct an additional experiment about introducing the history memory into the VLM inference process in response to the Reviewer **4kMy.** This is achieved by concating downsampled frames along the entire history RGB information and feed as an additional input for VLMs. We find that **including history RGB as input can help improve the SPL in both HM3D and HSSD** datasets (HM3D SPL from 0.243 to 0.262, HSSD SPL from 0.241 to 0.281), and significantly improve the ImagineNav performance on HSSD (Success Rate from 47% to 55%).

(3) We conduct an additional experiment about replacing the gpt4o-mini with the spatial understanding VLMs (SpatialRGPT) in response to the Reviewer **Va7n,** and we find the open-sourced SpatialRGPT **shows poor generalization to the HM3D domain rendered image data** and get lower than 40% success rate in the HM3D benchmark but got almost 50% percent success rate in HSSD.

(4) We quantify the quality of the Where2Imagine module waypoint prediction in response to Reviewer **j4ao**. We report the success rate for the generated waypoint (> 50%), where success is defined as the waypoint coordinates that lie on the obstacles. The metrics show that it is not difficult to get with pre-trained representations or using depth as input.

(5) We make a comparison between our method and the end-to-end method SPOC in response to the Reviewer **j4ao** and **Rhxu. We find that the end-to-end SPOC model,** although trained in massive simulation data, **shows limited generalization to the out-of-domain HSSD and HM3D object navigation** and achieved a success rate lower than 30% in both HSSD and HM3D benchmarks.

(6) We analyze the computation burden and the inference time of the novel-view generation module in response to the Reviewer **j4ao** and Reviewer **Va7n.**

(7) We improve the writing of the paper, add more detailed descriptions, and adjust the positions of Tables and Figures in response to Reviewer **Va7n.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ImagineNav, a novel mapless visual navigation system for robots using Vision-Language Models (VLMs) to guide object search. The approach replaces traditional mapping and localization with an imagination-driven method that generates and evaluates future views for navigation. The system outperforms existing methods on navigation benchmarks. 

Strengths:

- The paper presents an novel method for navigation that reduces reliance on traditional mapping with a vision-language model-driven approach.

- Extensive experimental validation shows promising performance improvements.

- The reviewer appreciated the comprehensive ablation studies and failure analysis provided in the paper.

Weaknesses:

- The NVS module is computationally intensive, which might limit its use in real-time applications.

- There are concerns that the hallucination of non-existent objects in the imagined views could lead to incorrect navigation decisions.

- The lack of historical information in the navigation process limits long-distance planning, which was addressed by the authors in the rebuttal.

The paper presents a promising and innovative approach with promising experimental results. Despite concerns regarding scalability and hallucinated objects, most reviewers agree that the contributions are valuable. I recommend acceptance.