ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Authors: Xinxin Zhao
Wenzhe Cai
Likun Tang
Teng Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Robotics
Visual Navigation
Vision-Language Model
Scene Imagination
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a mapless visual navigation system by proposing a imagination-based visual prompting for pre-trained large vision-language models.

Abstract: Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability to accomplish long-horizon daily tasks. Many recent approaches use Large Language Models (LLMs) for commonsense inference to improve exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry layout only by texts. Both are important for making rational navigation decisions. In this work, we seek to unleash the spatial perception and planning ability of Vision-Language Models (VLMs), and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual navigation tasks in a mapless manner. We achieve this by developing the imagination-powered navigation framework ImagineNav, which imagines the future observation images at valuable robot views and translates the complex navigation planning process into a rather simple best-view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where2Imagine module, which is distilled to align with human navigation habits. Finally, to reach the VLM preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open-vocabulary object navigation benchmarks demonstrates the superiority of our proposed system.

Published as a conference paper at ICLR 2025
IMAGINENAV: PROMPTING VISION-LANGUAGE MOD- ELS AS EMBODIED NAVIGATOR THROUGH SCENE IMAGINATION
Xinxin Zhao", Wenzhe Cai', Likun Tang, Teng Wang School of Automation, Southeast University (xinxin.zhao, wz.cai, lktang, wangtengj@seu.edu.ccm
ABSTRACT
Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability accomplish long-horizon daily tasks. Many recent ap- proaches use Large Language Models (LLMs) for commonsense inference to im- prove exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry lay- out only by texts. Both are important for making rational navigation decisions. this work, we seek to unleash the spatial perception and planning ability of Vision- Language Models (VLMs),, and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual nav- igation tasks in a mapless manner. We achieve this by developing the imagination- powered navigation framework ImagineNav, which imagines the future observa- tion images at valuable robot views and translates the complex navigation plan- ning process into rather simple view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where21magine module, which distilled align with human navigation habits. Finally, to reach the VI.M preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open- vocabulary ob- ject navigation benchmarks demonstrates the superiority of our proposed system.
INTRODUCTION
A useful comeeassistant robot should be able to search for different kinds of objects without telling the exact 3D object coordinates for completing our human instructions. our human always buying and bringing new goods back home, the robot's object searching capability should not be limited a closed-set of categories. Researchers often refer this problem as open-vocabulary object navigation task. Recently, as the emergence of the foundation models, including capable vision
models (Radford et al.
2021: He et al. 2022 Zhou et al.) 2022 Cheng et al.] 2024 Kirillvv et al.
2023 eu al. [202aa]
Liu al. 2023) large language models (LLMs) and vision-language models
(VLMs) (Brown et al. 2020: C Chowdhery et al., 2023 Zhang et al. 2022
Touvron et al. 2023
Achiam et al. 2023] Team
2023] Liu al.) 2024 Chen et 2024 Dai al. 2023 Gao
et al. 2023), building an agent that can accomplish the voca-vocayulary object navigation becomes possible. A popular framework, as shown in Figure [ uses modular approach to deal with this problem, which often composes of four components: A real-time mapping and segmentation module to perceive robot surrounding environments. semantic map information into texts. A LLM- module to understand the textual information from the previous step and make a plan texts. Finally, a path-planning module which project the reasoning result from LLM back to the map and plan collision-free path navigating towards it.
template-bar -based translation module to compress the
Although such pipeline achieves great success in recent years (Zhou et al., 2023 Kuang et al. 2024 Wu et al. [2024b] Zhang et al.) 2024a) Shah et al. 2023 Yu et al. 2023a; Loo et al. (022)) these complex cascaded systems have several limitations. Firstly, both the depth camera and the robot localization module can suffer from perception error, especially for long-range depth esti- mation, and this can make the mapping process inaccurate. Secondly, online object detection and segmentation are required to augment spatial maps with semantic labels and prepare for LLM's
"Equal Contribution, 1 Corresponning Author: Teng Wang.

---

Published as a conference paper at ICLR 2025
reasoning input. This increases the computational burden for the robots. Thirdly, although the mantic information stored on the map can be easily expressed by text (e.g., list the categories of the observed objects), such pure text prompts have difficultly in explicitly describing the geometry information and object details in the map, making it difficult and ambiguous for LLMs to infer the best navigation plan.
LLM-Based Navigation Pipeline
Our ImagineNav Pipeline
Robot
Map
dedde the direction exaluat the pronded ckasigation Gosb <Panoramic imapeeo Choose the most
Sensor Pose Estimation
Textualize
A. <
VLM-Based Reasoning
Tocki Find <Torget
Options:
<itthhen.>>
<loilet..>>
<Corridor..>
29
Trajectory
LLM-Based Reasoning
Imagination
PointGool Policy
Figure I: The comparison between the conventional LLM-based navigation pipeline and our Imag- ineNav pipeline. The traditional LLM-based navigation framework, illustrated on the left, relies on intricate sensor data processing and pose estimation for map creation, followed by LLM-driven reasoning to decide the exploration direction. Instead, our ImagineNav directly translates the long- horizon object goal navigation task into sequence of best-view image selection tasks for VLM, which avoids the latency and compounding error in the traditional cascaded methods.
In this work, we try to explore whether is possible to circumvent the complicated and fragile mapping-> translation planning framework, but develop visual navigation approach with only raw RGB/RGB-D observations and pre-trained VLMs. Our proposed method ImagineNav seeks to maximize the capabilities of VLMs in multimodal understanding and reasoning, and make the VLMs become an efficient embodied navigation agent. As most VLMs cannot understand the continuous physical world, it is infeasible to directly ask VLMs to generate navigable 3D waypoints. Instead, translate the visual navigation problem into an imagination-powert and let the VLMs select. To generate appropriate options for VLMs choose from, we propose the Where2lmagine model which distills the human indoor navigation habits and generates future 3D navigation waypoints where human might navigate based on the current visual observation. Such 3D navigation waypoints indicate relative poses with respect to the current frame and can be easily translated into new observation images using novel view synthesis (NVS) models. Afterwards, the VLMs only need to select the best imagined observation that is mostly related to the target object and drive the robot follow the corresponding point-goal navigation trajectory. The above pose- aware aeggination-nndsseeeetee be decomposed into a sequence of point-goal sub-tasks, facilitating the creation of collision free navigation trajectories. Experimental results standard benchmarks demonstrate the superiority of our ImagineNav over previous methods in open- open-vocabulary object navigation. In summary, our contributions
best-view image selection task
capability allows the Object Goal Navigation (ObjectNav) task to
We propose a mapless navigation approach ImagineNav. leverages the imagination to generate image observations at potential future 3D waypoints as the VLMs' visual prompts, grounding the VLMs to become efficient navigation agents without any fine-tuning.
We design a task-triented model Where21magine understand human navigation habits. This model is crucial to bridge the task-agnostic high-level VLM planners and the low-level navigation policies.
Our ImagineNav increases success by large margin of 15.1% % and 10% respectively on HM3D (Ramakrishnan et al.) (021) and HSSD (Khanna et al. 2023). We also provide a detailed ablation analysis help understand the important conclusions in framework.
RELATED WORKS
2.1
LARGE MODELS FOR ROBOTIC PLANNING
Large-scale models pre-trainnt trained on extensive internet data have demonstrated formidable zero-shot reasoning capabilities in tasks such planning (Huang al. 2022), code generation (Liang al.)
2

---

Published as a conference paper at ICLR 2025
The in- context learning capability of LLMs allows them to be applied to robotic task planning. Some methods (Liang et al., 2023 Huang et al. 2023b Ahn et al.) 2022) leverage LLMs to decompose tasks into subtasks, enhancing execution efficiency. Cap (Liang et al. 2023) generates robotic policy code directly from example language commands, enabling autonomous control and task execution based on natural language instructions. Instruct2Act (Huang et al. 2023b) combines LLM with foundational models (e.g., SAM and CLIP), reducing error rates in complex task execution, while SayCan (Ahn et al., 2022) combines LLM task planning with the feasibility of physical skills us- ing pre-trained value functions, generating actionable plans for robots. However, one limitation of LLMs is their difficulty in embedding the robot's state directly into the planning process. To address this, many studies have turned to VLMs as alternatives. For instance, ViLA (Lin et al.) 2024) significantly improves performance on multimodal tasks without compromising text capa- bilities by systematically exploring VLM pretraining design choices. CoPa (Huang et al. 2024) incorporates commonsense knowledge from VLMs, proposing coarse-to to-eee task-oriented grasp- ing and task-aware motion planning approach. PIVOT (Nasiriany et al.) 2024) transforms tasks into iteratively optimized visual question-anssente problems via refinement process. Socratic model (Zeng et al., 2022) integrates multiple pretrained large models (e.g., VLMs, LLMs, and au- dio models) in modular fashion enable reasoning and task execution through language-based interaction. These methods employ set-of-examples (SOE) approach to guide VLM selection. We propose a new decision- decision-making paradigm based on imagined imagery, wherein decisions are made on imaginations, enabling more nuanced, context-aware interactions that better harness VLMs' spa- tial perception capabilities.
2023
Huang et al. 2023b), and solving science questions (Lewkowycz et al. 2022
2.2 OPEN-VOCABULARE OBJECT NAVIGATION
ObjectNav requires the robot to navigate toward specific target category in unseen environments. Although previous works (Yadav et al. 2022 Ramrakhya et al.] 2023 Chaplot et al., 2020 Ra-
makrishnan et al. 2022) can achieve high success rate widely accepted benchmarks (c.g.. habitat- challenge (Yadav et al. 2023)), most approaches are limited within a pre-defined object list, which is contradictory to the open-vocabulay
real world. Therefore, many researchers start to discuss the open-vocabulary object navigation problem. End-to-end methods try to make use of compact mul- timodal features space (e.g., CLIP (Radford et al. (22I]) for grounding text knowledge into visual
navigation problem but achieved limited performance (Khandelwal et al. 2022 Gadre et al. 2023
Majumdar et al. 2022). Instead, modular-based approaches (Huang al. 2023a Zhou al.
2023
Achiam et al. 2023) typically necessitate the use of sensors for localization and mapping, high-level planning, and low-level control. These methods rely heavily on precisprciiion sensors for accurate self-localization and real-time map construction. Our approach introduces an imagination-based, mapless navigation framework. This framework circumvents the need for extensive training by transforming the complex process of navigation planning into selection problem based solely on RGB inputs.
2.3 BAAGINATION-BAEED NAVIGATION
Recent methods (hai & Wang 2022 Ramakrishnan et al. 2022 Zhu et al. 2022) have employed
supervised learning to learn farget-related functions in order to address the subtask of 'Where to look?' in navigation, specifically focusing on predicting the localization of target. These meth- ods predict the absolute coordinates (Zhai & Wang 2022) of the target, the shortest distance to target (Ramakrishnan al. (002)) the nearest boundary (Zhu et al.) 2022) based on local maps. Instead, several studies (Ramakrishnan et al. 2020; Georgakis et al. | et al., 2021 Zhang al. 2024b) have proposed various approaches to enhance the prediction of unobserved regions. For instance. (Ramakrishnan et al. 2020) introduced occupancy anticipation, where the agent in- fers an occupancy map based on RGB-D inputs. The I.2M framework was introduced (Georgakis et al., (022). consisting of two-stage segmentation model that generates semantic map beyond the agent's field of view and selects long-term goals based on the uncertainty of predictions. SS- CNav algorithm (Liang et al.] (022) leverages semantic scene completion and confidence maps to infer the environment and guide navigation decisions. A self-supervised generative map (SGM) is proposed (Zhang et al. 2024b), which employs self-supervised approach to continually generate unobserved regions in the local map and predict the target's location. These methods primarily pre- dict unobserved regions in top-down maps derived from egocentric RGB-D projections. In contrast,
3

---

Published as a conference paper at ICLR 2025
Our method operates in the RGB space perform innieeddimainatiion.. We utilize compact model aligned with human navigation habits to generate new viewpoints, followed by the imagination pro- cess to produce corresponding visual representations.
METHODOLOGY
The open-vocabule nenn-vocbullary object navigation task requires the agent to locate untrained target object instance in an unknown environment. At the start of each episode, the agent is spawned at a random position in an unfamiliar environment without any prior knowledge of the layout. The task is to find target object 9ú. which can belong to any category in an open- veen-vocabulary setting. At each time step t, the agent receives egocentric panorama view 4, divided into 6 separate views, each represented by an RGB image I,, accompanied by its depth map Di,i- The discrete action space consists of the following commands: {Stop. MoveAhead, TurnLeft, TurnRight, LookUp, LookDown). The 'MoveAhead' action moves the agent forward by 0.25m, while the rotational actions "TurLeet' and 'TrnRighh' rotate the agent by 30 degrees. The task is considered successful if the agent reaches the target object with a geodesic distance smaller than a defined threshold (c.g., Im) and executes the 'Stop' command within fixed number of steps. Each episode has maximum limit of 500 steps.
<Navigation
Future- Imagination
Current Observations
ResNet-18
Relative Pose
6x=0.5, 0y-2.05, 8-223.6
Novel View Synthesis
Ax=1.29, 4y=2.14, 0=-29.1"
Train
Scratch
Ax--0.94, 4ym2.11, 8-22.3"
Scene Generation
Vision-Language Model
(Stwp.Forward. TurnLeft,Turn
<User>: Analyzing the composite image to
LookUp, LookDown)
Point Goal Navigation
ation
Reasoning
<VLM>:
This image shows hallway leading other roams where chair may be found.
Option:
(x=-3.50,yxx3.85)
O Free Arta
Candidala Wasgainis
The overall pipeline of our mapless, open-vocabulary navigation framework. At each it- eration, the agent captures panoramic view of surroundings. In the Imagination Module, the trained Wh Where2lmagine module couples with novel view synthesis model to generate novel scene views. Guided by prompt templates, the VLM engages in target-oriented inference. Subsequently, the system executes the PointNav policy to determine the next navigational action. The above imag- ination, reasoning and planning procedure iterates until the target reached.
Figure 2:
3.1 IMAGINENAV FRAMEWORK
This section provides an overview of the imagination-bused open-vocabulary object navigation framework (lmagineNav). As depicted in Figure 2 the agent initially employs the Where2lmagine module generate candidate locations for imagination based on the current observations. Sub- sequently, the visual observations at these locations are imagined by NVS model. Utilizing the generated images, which are annotated with option labels, the agent leverages multimodal large- scale model to assess both the spatial and semantic information of each scene, enabling the selection of a more efficient exploration direction. Specifically, the VLM is employed to reason over the imagined images from six different views using prompts, selecting the optimal waypoint. Finally, the agent executes low-level point navigation strategy to reach the corresponding sub-goal. This process iterates, where each new observation serves as input for further imagination, reasoning, and
4

---

Published as a conference paper at ICLR 2025
navigation, until the agent successfully identifies instance of the target object. Consequently, the ObjectNav task is reduced to a sequence of simpler point-to-poimt navigation subtasks. Since our ImagineNav does not require any training object-oriented data for reasoning and planning, it is open- open-vocabulary and can zero-shot generalize to novel semantic targets. See 3.2 introduces the imagination module, Sec 3.3 explains the use of the advanced planner VLM, and Sec describes our underlying point navigation strategy.
3.2 FUTURE- VIEW IMAGINATION
To better leverage the spatial perception and reasoning capabilities of VLMs for open-vocabulary object navigation in unknown environments, we propose future-view imagination model, which is composed by Whegien module and a NVS model. As shown in Figure 2 Where2lmagine predicts the relative pose (Ar, Ay, 0) of the potential next navigation waypoint based on the current RGB observation, where Aæ denotes lateral displacement, Ay represents longitudinal displacement, and refers to changes in the camera's viewing angle. Subsequently, a novel view image is generated based on the predicted relative pose. There are several advanced methods to achieve this, such as framing the task as a few-shot 3D rendering problem (Sargent et al. 2024 Wimbauer et al.] 2023 Cao & de Charette 2023) or utilizing generative models (e.g., diffusion models) for image synthesis (Yu et al. 2024 Tsegg et al. 2023; Yu et al. (023b). In this work, we employ a pretrained diffusion model Yu et al. 2024), which generates new view images by taking the current image and the predicted relative pose as inputs, enabling high-quality viewpoint transformation.
Specifically, we trained ResNet-18 from scratch for the relative waypoint prediction task, with training data sourced from the Habitat-Webb project (Ramrakhya et al.) 2022). Habitat-Web collects remote user demonstrations of virtual robot operations through virtual teleoperation system on the Amazon Mechanical Turk platform, including 80k ObjectNav and 12k Pick&Place demonstrations. Humans in these demonstrations typically opt for directions towards important semantic cues (e.g., doors) that facilitate exploration. These navigation preferences serve as a valuable basis for deter- mining potential waypoints, thereby enhancing the rationality and safety of the navigation strategy- We transformed the human demonstration trajectories into paired dataset (4,, P:+T), where L rep- resents the RGB image at frame t. and Pt-T denotes the relative pose (Az, Ay, 0) at frame + T with respect to frame f. To facilitate the learning of the network, we filtered out images with a depth threshold of less than 0.3, as these images generally lack rich semantic information (e.g.. a plain wall). Due to the limitations of the NVS model in generating high-quality images for large perspec- tive shifts (e.g.. 120", 188" (,100,200)) based solely on a single input image, we restricted the training data to instances where the angular deviation falls within +30%. Through the Where2lmagine module, our imagination model aligns with human navigation habits.
3.3 HIGH-LEVEL PLANNING
High-level planning module leverages the spatial awareness and common-sense reasoning capa- bilities of the VLM to select the direction most likely to locate the navigation target. We prefer GPT-4o-mini as the high-level planner because it offers balance between the reasoning capabilities and practical efficiency. Compared to larger models (c.g., GPT-4o), GPT-4o-mini is lightweight and cost-effective. Its smaller size ensures faster inference, allowing the system to make timely decisions in dynamic environments. To assist GPT-4ommnii in decision making, designed simple prompt template, requiring the VLM to summarize its choice in a JSON format containing { 'Reason', 'Choice' ). This format allows for clear understanding of the VLM's reasoning process. As illus- trated in Figure the VLM receives the synthesized observations at future navigation waypoints and the navigation goal as inputs. Based on the prompt. 'Your choice should first be based on discov- ering navigation targets, followed by the potential of unexplored areas..." the VLM analyzes each image's s semantic information, selects the optimal exploration direction, and returns the answer in the specified format. By providing the imagined observations as visual prompts to VLM, our Imag- ineNav offers significant advantages in spatial reasoning and decision-making processes. Firstly, the VLMs are more skilled at handling multiple-choice decision tasks compared to 3D geometry question answering (i.e., directly inferring the 3D coordinates of next waypoints). Moreover, the introduction of imagination enhances the decision- decision-making capability of the VLM by supplement- ing detailed information about distant or visually unclear objects. The advanced planning module proposes new sub-goals after the low-level controller completes navigation to the designated target.
5

---

Published as a conference paper at ICLR 2025
Target: Find a couch
Analyzing composite image for the navigation goal of finding couch:
Original scenarios
--Option 1: Shows a living area with a TV visible. This direction promising might contain the target object. --Option Shows living with couch visible.
--Option 3: Shows bedroom area, which is unlikely contain the couch. This direction not promising.
--Option This image shows wall with picture, making less relevant.
Future-view scenarios
-OOption 5: Shows a view of the kitchen with no couch visible.
Option
Optin
-OOption Displays dining area with chairs, no couch visible.
Option
Option
don
Based on the analysis, the most promising picture find couch ** *Option 2** where couch is clearly visible in the living area.
An example of the VLM analysis. By examining different future-view scenarios, the VLM pinpoints the direction most likely incorporate the target object couch.
Figure 3:
3.4 CONTROLLER
After the high-level planner provides the navigation points, the low-level controller executes Point Goal Navigation (PointNav) strategy to achieve these targets. Unlike ObjectNav, PointNav (turn to Az, Ay) does rely semantic information from the environment but is instead driven solely by spatial perception. Currently, there are many methods available for achieving PointNav (Yang et al.)
2023 Roth al. 2024 Wijmans al. 2022| Liang al. 2024). To determine the execution actions at each step of the PointNav process, we use Variable Experience Rollout (VER) (Wijmans et al., 2022) as our underlying goal navigation strategy. VER combines the advantages of synchronous and asynchronous reinforcement learning, improving training efficiency and sample utilization in PointNav tasks, thereby enabling the agent to demonstrate stronger adaptability and generalization capabilities in complex environments.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP AND METRICS
We evaluate the effectiveness and navigation efficiency of our proposed method using the Habitat v3.0 simulator (Pug al., 2023) on standard ObjectNav datasets: HM3D (Ramakrishaan al., 2021) and HSSD (Khanna et al. 2023). The HM3D dataset offers high-fidelity reconstructions of 20 entire buildings, including 80 training scenes and 20 validation scenes. The HSSD dataset provides 40 high-quality synthetic scenes, comprising 110 training scenes and 40 validation scenes. The experimental setup follows the ObjectNav-challeng--202. (Yadav et al. 2023). For the data collection of the Where21magine module, we leveraged human demonstration trajectories from the MP3D (Chang et al. 2017) dataset within the habitat- web project with the camera height 0.88m and horizontal field of view (HFOV) of 79". We report the performance in terms of Success Rate (SR), defined as the proportion of episodes where the agent's 's distance to the target object is less than Im after executing the STOP action, and SPL. (Anderson et al. 2018), Success weighted by path length,

---

Published as a conference paper at ICLR 2025
(1,) where S be a binary success indicator in episode i, pi is the SPL = #
agent path length and & is the GT path length.
4.2 BASELINES
conducted a comparative analysis of non-zero-shot and zero-shot ObjectNav methods to sub- stantiate our proposed approach. Yamauchi (Topiwala al., 20188) pioneered frontier-based ex- ploration strategy, emphasizing the boundaries between explored and unexplored regions. Sem- Exp (Chaplot et al. 2020) advanced this concept by implementing goal-directed semantic explo- ration through the construction of semantic maps. In addition, we examined non-mapping closed- set object navigation baselines, including those based on imitation learning (Ramrakhya al. and visual representation learning (Yadav et al. 2022).
2022)
For zero-shot object navigation, we consider several mapping-baser baselines (Zhou et 2023 Wu et al. 2024b; Yokoyama et al., (203)) that integrate commonsense knowledge and semantic information facilitate direct navigation toward target objects, leveraging semantic comprehension from pre-trained large models to aid in navigation. Furthermore, we explored RGB-bar beB-based non- mapping navigation baselines: ZSON (Majumdar et al., (022). which employs the CLIP (Radford et al. 2021) model to embed target images and object goals within a unified semantic space, thereby training a semantic noal-navigation agent, and PixNav (Cai al.) 2023), which utilizes pixel-level goal guidance, enabling pixel navigation through the use of VLMs and LLMs.
4.3 COMPARISON WITH PRIOR WORK
Table D presents a comparative analysis of the proposed ImagineNav against prior research efforts. On the HM3D dataset, ImagineNav achieves a success rate of 53.0% and SPL of 23.8%, sig- nificantly outperforming most of the methods especially at success rate. Moreover, ImagineNav achieves the highest success rate and SPL on the HSSD dataset. Particularly, in open-vocabulary navigation tasks, our mapless ImagineNav even outperforms the persoperforming map-based method VLFM (Yokoyama et al. 2023) by 0.5% at success rate. The above observations indicate that our ImagineNav demonstrates outstanding navigation performance across various settings, while main- taining low storage and computational complexities. Furthermore, since the pretrained NVS is di- rectly employed without finetunned on the HM3D and HSSD datasets, we see a disparity between the quality of images generated by the NVS model and real images, limiting the capability of model to some extent. To explore the upper limits of our framework, we instead use real panoramic images-speecifiallly the observation at the pose predicted by the Where21magine module-as vi- sual prompts for the VLM model. Notably, both the success rate and SPL exhibit obvious improve- ments, obtaining 62.0% and 59.0% success rate respectively H3MD and HSSD benchmarks, which further demonstrates the superiority of our imagination-bres based navigation framework.
Table I: ImagineNav: Comparison with previous work. The Where21magine model with T=11. utilizing ResNet-18 trained from scratch and GPT-4o-mini as the VLM, was evaluated over 200 epochs on the HM3D and HSSD datasets. ImagineNav uses NVS model generate novel view images, while ImagineNav-Oracle uses real images of the candidate points.
HM33
HISDD
Method
Vpee--ocabulary
Mapless
Success Rate
SPL.
Success Rate
SPL
FBE
Topiwala et al.
2018
x
X
33.7
15.3
36.0
17.7
SemFxp
Chaplot ot al
2020
x
37.9
18.8
Habitat- Web
Kamrakhya
2022
41.5
OVRL
X
62.0
26.8
ESC
Zhou et
2223
39.2
22.3
VoroNav Wu al
2024b
42.0
26.0
41.0
23.2
VLFM
Yokoyama
2023
52.5
30.4
7SON
Majundar
25.5
12.6
PixNav (Cn
37.9
20.5
ImagineNay
53.0
23.8
51.0
24.9
ImagineNav-Oracke
62.0
31.1
59.0
27.0
7

---

Published as a conference paper at ICLR 2025
4.4 ABLATION STUDY ON MAIN COMPONENTS
We conducted an ablation study on main components of the imagina- tion module to demonstrate their ef- fectiveness, with each variant eval- uated for 100 epoches. As shown Table 2] we increase success rate from 43.0 to 55.0 by utilizing future imaginations as visual prompt of VLM for deciding exploration direc- tion. Please note that at row #2 with- out Where2lmagine, we simply gen- erate future views at the six locations, which are meters away from rent observations along their respec- tive observation orientations. This improvement suggests the superior- ity of future imagination in facilitat- ing VLM's reasoning. Such improve- ments can be attributed the greater semantic disparity between different imaginations, as illustrated in Figure
Table 2: ImagineNav: ablation study on the imagination module. 'Imagination' refers to whether the future imagi- nations
used as visual prompts of the VLM. When it is removed, we feed current observations into VLM for decid- ing the best exploration direction, and set the next waypoint meters away from the current location along the direction. Here, the distance of meters is considered as it compara- ble to that generated by T=11. 'NVS' indicates whether the image captured from real environment or synthesized via the NVS model.
HM3D
Imagination
Where2Imaginee
NVS
Success Rate
SPL
Oracle
43.0
24.7
Orucle
55.0
27.6
Oracle
64.0
28.3
PolyOculus
49.0
23.3
PolyOculus 56.0
24.3
Further incorporating Where2lmage improve success rate from 55.0 64.0, and from 49.0 to
56.0 under settings 'NVS' and 'w/o NVS', respectively. As mentioned above, the capability of ImagineNav is limited by the performance of the off-the-shelf NVS model (Yu al. 2024) to some extent as evidenced by comparing rows #2 with #4 and rows #3 with #5. Nevertheless, the incorporation of Where2Imagine partially mitigates the adverse effects of the NVS model.
Current Observations
ption
Option
Option 3
You will be provided with the following elements: (1)<Naaigation Goal;; toilet (2)<Candidate Images> Evaluate these six images and choose the most suitable one for exploration.
Option &
Option
Answer=(RReason' 'The image shows clear view of hallway leading to another room, which likely contain the toilet., 'Choice':
Future Imaginations
Option
Option 2
Option 3
'The image shows hallway leading a bathroom. The other pictures do not indicate potential access to bathroom., 'Choice': 2)
Answer=('Reaoon'
Option 4
Option
opttinn 6
Figure 4: Visualization of the synthesized image observations at future navigation waypoints pre- dicted by the imagination module. It can be seen that there exists drastic semantic disparity between different imaginations. In contrast, the semantic information is relatively consistent across different current observations. The varying semantics across different future views highlight the advantages of the imagination module in enhancing the VLM's decision-making capabilities.
4.5 ANALYSIS OF SUCCESSFUL AND FAILED TRAJECTORIES
Figure iillustrates that method achieves efficient path planning and navigation across different targets. Especially, as shown in the top middle of Figure the agent needs to navigate through mul- tiple rooms. The complex environment increases the risk of losing direction, but ImagineNav successfully infers the optimal path and finds the target, demonstrating its ability to handle long- distance and multi-room scenarios. We also present some failure examples the bottom of Figure
8

---

Published as a conference paper at ICLR 2025
5 We identified three key factors contributing to these navigation failures. First, object in- stances are neglected for marking by the simulator, and therefore a successfully trajectory is wrongly considered failure (a.k.a. false failure) as shown in the bottom left of Figure Second, the syn- thesized image from the NVS does not align with the real observation, such as creating objects that present in the real scene as shown in the middle of Figure which causes the VLMM make incorrect inference. Moreover, the lack of historical information make the agent easily trapped in local optima, thereby limiting its performance in long-term navigationas shown in the bottom right of Figure 5
Chair, Success:True, SPL:0.60
Bed, Success:True, SPL:0..3
Couch, Success:True, SPL:0.90
Red Imaaee
TV, Strccess:False, SPL:0.0
Plant, Success: SPI:0,,PPL.OO. SPL:0.0
Bed, Success:False SPL.:000
Figure 5: Visualization of the navigation trajectory. The top and bottom rows respectively show the complete top-down trajectories of successful and unsuccessful examples.
4.6 ANALYSIS OF WHERE2IMAGINE MODULE
We explore the impact of the sampling step on the final navigation performance by varying from to 15. For each T, re-generate the labeled image data and re-train the ResNet-18 for relative pose prediction. Each variant was tested for 100 epochs under conditions where the agent had access to real panoramic observations. As shown Table 3 the best success rate and SPL are obtained when T is set to 11. Furthermore, we visualize several navigational trajectories under different values of T in Figure 0 facilitate explanation. As can be seen, when T is relatively small (i,e., 8), the agent is easily trapped as marked by red square, since it mainly resorts to local semantic information for inferring its exploration direction, making susceptible to converging on suboptimal local solutions.
T=8, Success:False, SPL:0.0 T=11, Success:True, SPL:0.35
T=15, Success: SPL:0.0
Figure 6: Comparison of trajectories at different sampling steps T. This image presents top-down view of the entire trajectory as the agent searches for the target (a chair). The red box highlights the situation where the agent encounters a local trap during navigation.

---

Published as a conference paper at ICLR 2025
Conversely, when T is excessively large, although the agent has access more distant information, is prone to miss some critical intermediate semantics which are closely related to target and are worth exploring, leading to erroneous long-ram range decisions, particularly in intricate environments. However, an optimally calibrated I' can strike delicate balance between exploration and perception, thereby facilitating to obtain impressive navigation performance.
We compared different backbones to evaluate their impacts on both the relative waypoint prediction and final navigation performances. Specifically, we modified the final output layers of the ResNet- 18 and VIT to fit our dataset, allowing parameter updates during training. In contrast, DINOv2 and MAE models were connected to a five-layer MLP, with only the MLP parameters trained while freezing the backbone. The experiments were conducted on real RGB observations without resort- ing to the NVS model. Each variant was tested for 100 epochs. The results in Table 4 show that the ResNet-18, when trained from scratch, achieves the best performances in both relative wayopint prediction and ObjectNav while featuring more lightweight architecture. Furthermore, the large performance gap between different backnones suggests the importance of the Where2Imagine mod- ule, indicating the usefulness of learning from human demonstrations, Please note that we observe slight inconsistency between waypoint prediction loss and navigation success rate under backnones of DINOv2 and MAE. This might be because the waypoint prediction is evaluated on the testing data from MP3D dataset while the navigation is performed HM3D. The differences between the datasets result certain degree of variability.
Table 3: Where2lmagines the influence of sampling step T on navigation perfor- mance.
Table 4: Where2lmagine: the impact of different backbones. Loss refers to the test loss of Where2lmagine. TFS: training from scratch, FT: fine-tuning.
HM3D
Backbone
Params
Flops
Loss
T
HM3D
Success Rate
SPL
Success Rate
SPL
ResNet-18 (TFS)
11.4M
1.8G
0.12
64.0
28.3
8
51.0
25.1
ResNet-11 (FT)
11.4M
1.8G
0.24
61.0
29.7
10
64.0
29.4
VIT (TFS)
86.0M
16.9G
0.22
61.0
29.7
11
64.0
28.3
ViT (FT)
86.0M
16.9G
0.23
58.0
31.0
12 59.0
30.0
DINOv2
22.6M
5.5G
0.22
58.0
27.9
15
59.0
26.8
MAE
87.1M
4.4G
0.20
57.0
26.5
4.7
ANALYSIS OF VLM PLANNER
We conducted comparative evaluation of the effects of differ- ent VLMs on navigation performance, as detailed in Table The experiments used real RGB without NVS model. The sults demonstrate that GPT-4o-mini and GPT-4-Turbo exhibit negligible differences in success rate and SPL metrics, while showing marked advantage over LLaVa, underscoring the significant role of advanced model reasoning capabilities fluencing experimental outcomes. Moreover, for models of the same architecture, it is possible to opt for more cost-effective variants without compromising navigation performance, thus enabling more resource-efficient and time-saving.
Table Effect of different VLM.
HM3D
VLM
Success Rate
SPL
LLaVa
44.0
21.6
GPT-4-Turbo
63.0
29.4
GPT-4o-mini
64.0
28.3
CONCLUSION
We propose the ImagineNav framework, a mapless, open-vocabulary object navigation approach leveraging VLM. By incorporating imagination mechanism, the system effectively predicts future waypoints, transforming traditional navigation planning into a visual selection task for the VLM. Ablation studies highlight ImagineNav's strong potential for long-horiag navigation. Moving for- ward, we aim to enhance the quality of viewpoint generation and optimize the use of historical memory to further improve navigation performance and robustness.
10

---

Published as a conference paper at ICLR 2025
6
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (Grant No. 62273093).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint StXiiv2303.08774, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not i say: Grounding language in robotic affordances, 2022. URL https: /laaxiv.org//aooo...aaoao..oo..a abs// 2204. 001.0.66.66. 01691
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents, 2018. URL
org/
abs/1807.0et 06757
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging zero-shot object navigation and foundation models through pixel-guided naviga- tion skill, 2023. URL
Anh-Quan Cao and Raoul de Charette. Scenerf: Self-superrvised monocull 3d scene reconstruction with radiance fields, 2023. URL http::///aaa.....rr///////222112
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference 3D Vision (3DV), 2017.
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247-4255. 2020.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24198. 2024.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time ooca-vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards -eeerol-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06:: 06500, 2023.

---

Published as a conference paper at ICLR 2025
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 23171-2318 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. dlama-adaptes v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.1 15010, 2023.
Georgios Georgakis, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, and Kostas Dani- ilidis. Learning to map for active semantic goal navigation, 2022. URL https://arxiv:
org/abs 2106. 15648
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation, 2023a. URL /tR.:/ttpee//aavt..rrraaa///////
Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models, 2024. URL https:
//arxxvv.rg//////...lea.. wrw.bbs 2403. 08248
Siyuan Huang. Zhengkai Jiang, Hao Dong. Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping modaltty instructions to robotic actions with large language model, 2023b. URL
www.a/wwwwwwwrsrr///rr// /ttp://wwww..r///////.//./. 11176
https:://rxiis
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero- shot planners: Extracting actionable knowledge for embodied agents, 2022. URL https://
btxiv.org/abss/22000 22t.bb//2221..70000
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but ef- fective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14829-1488 2022.
Mukul Khanna, Yongsen Mao, Hanxiao Jiang. Sanjay Haresh, Brenman Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat synthetic scenes dataset (00s An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation, 2023, URL http:: http://aaxvvvv.. org/ /b36
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Yuxuan Kuang, Hai Lin, and Meng Jiang. Openfmnav: Towards zho-sot object navigation via langunel foundation models. arXiv preprint arXiv:2402.10670 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022. URL TtLhttt:: /ttpiiiiivv.///.a.//..a//..aaava wwwoor/ggoo.... abs 2206. 14858
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL https::/arxixw> http:////aaaa..orrr//////20....
Jing Liang, Peng Gao, Xuesu Xiao, Adarsh Jagan Sathyamoorthy, Mohamed Elnoor, Ming C. Lin, and Dinesh Manocha. Mtg: Mapless trajectory generator with traversability coverage for outdoor navigation, 2024. /
08214
12

---

Published as a conference paper at ICLR 2025
Yiqing Liang, Boyuan Chen, and Shuran Song. Sscnav: Confidence-aware semantic scene com- pletion for visual semantic navigation. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13194-13200, 2021. doi: 10.1 00.10099CCRAA50002200...666
Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2024.
URL battps://azxiin org/ abs 2tts//331.0002 07533
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
Joel Loo, Zhanxin Wu, and David Hsu, Open scene graphs for open world object-goal navigation. arXiv preprint arXiv:2407.02477 2024.
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson:
Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 35:32340-32352, 2022.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang- Wei Edward Lee, Kuang- Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian lchter. Pivot: Iterative visual prompting elicits actionable knowledge for vims, 2024. https::::: waw.iiivvvvv./.vv www.oorggaaa......... 2 402 07872
Xavier Puig, Eric Undersander. Andrew Szot, Mikael Dallaire Cote. Tsung- Yen Yang, Ruslan Part- sey, Ruta Desai, Alexander William Clegg. Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Ak- shara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots, http/abb/2311...0.
2023. URL URLlhtpps: /tttps:://rxiiii
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Santhosh K. Ramakrishnan, Ziad Al-Halah, and Kristen Grauman, Occupancy anticiiatiin for effi- cient exploration and navigation, 2020.
C9285
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat- matttat-matterport dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kris- ten Grauman. Poni: Potential functions for objectgoal navigation with interaction-free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
18890-18908 2022.
Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale, 2022. URL https:
//axxiv.org/ab//222... 0
Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imita- tion and rl finetuning for objectnav. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17896-17908 2023.
13

---

Published as a conference paper at ICLR 2025
Pascal Roth, Julian Nubert, Fan Yang, Mayank Mittal, and Marco Hutter. Viplanner: Visual semantic imperative learning for local navigation, 2024. URL http::::/axxiiv.rr//bb.....
00982
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. Zeronvs: Zero-shot 360-degree view synthesis from a single image, 2024. URL https////xxxvvrrrg/e/ee
231 0. 17994
Dhruv Shah, Michael Robert Equi, Blažej Osiński, Fei Xia, Brian Ichter, and Sergey Levine. Naviga- tion with large language models: Semantic guesswork heuristic for planning. In Conference on Robot Learning, pp. 2683-2699. PMLR, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 211::2112211111000 2023.
Anirudh Topiwala, Pranav Inani, and Abhishek Kathpal Frontier based exploration for autonomous robot, 2018. Utthttss:
org awtpaaass./1/ 03581
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Con- sistent view synthesis with gose-g diffusion models, 2023. URL httthttss///xxiit org/
abs/2303. 17598
Erik Wijmans, Irfan Essa, and Dhruv Batra. Ver: Scaling on-policy rl leads to the emergence of navigation in embodied rearrangement, 2022. URL http/htts:://rxiv..oom http://wwwi..rrrb//////b// 05064
Felix Wimbauer, Nan Yang, Christian Rupprecht, and Daniel Cremers. Behind the scenes: fields for single view reconstruction, 2023. URL
Density http://ww..og/aa////001/.
Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3783-377337 3795,, 2024a.
Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronav: Voronoi-based zero-shot object navigation with large language model. arXiv preprint MxXiv:2401.02695 2024b.
Kanmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierr Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for em- bodied navigation, 2022. URL http:ttp://xiii.o/////////
absg///bs/20 13226
Karmesh Yadav, Jacob Krantz, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Jimmy Yang, Austin Wang, John Turner, Aaron Gokaslan, Vincent-Pierr Berges, Roozbeh Mootaghi, Olek- sandr Maksymets, Angel X Chang, Manolis Savva, Alexander Clegg, Devendra Singh Chaplot, and Dhruv Batra. hattps://aihbbiiie
www./coallloooooooooooooo......a eeeeeng/22200 2023/ 2023.
Fan Yang, Chen Wang, Cesar Cadena, and Marco Hutter. iplanner: Imperative path planning, 2023.
URL hanttts://axxiv.orrtt abs /2t///332... 11.02.11433 11434
Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision- language frontier maps for zero-shot semantic navigation, 2023. URL.htts::/aazivv.oo
abs/2312.5 03275
Bangguo Yu, Hamidreza Kasaci, and Ming Cao. L3mvn: Leveraging large language models for visual target navigation. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3554-3560, 2023a.
14

---

Published as a conference paper at ICLR 2025
Jason J. Yu, Fereshteh Forghani, Konstantinos G. Derpanis, and Marcus A. Brubaker. Long-term photometric consistent novel view synthesis with diffusion models, 2023b. URL https://
arxiv. Dexiv.org//aa abs 10700
Jason J. Yu, Tristan Aumentado-Armstron Fereshteh Forghani, Konstantinos G. Derpanis, and Marcus A. Brubaker. Polyoculus: Simultaneous multi-view image-based novel view synthesis,
2024. URtppsss://axxx wtw.vvo.r//aaao abs httpabbb./.222222
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Van- houcke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language, 2022. URL/ttps:://aaattt org. abs/ /2044. 00598
Albert J. Zhai and Shenlong Wang. Peanut: Predicting and navigating to unseen targets, 2022. URL abs
httpps:/aarse wttg 02497
Lingfeng Zhang. Qiang Zhang, Hao Wang, Erjia Xiao, Zixuan Jiang, Honglei Chen, and Ren- jing Xu. Trihelper: Zero-shot object navigation with dynamic assistance. arXiv preprint arXiv:2403.15223, 2024a.
Sixian Zhang, Xinyao Yu, Xinhang Song. Xiaohan Wang, and Shugiang Jiang. Imagine before go: Self-supervised generative map for object goal navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16414-16425, June 2024b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 1tt.::2205.0066. 2022.
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation, 2023. URL httpt:tta/xxii....tr
org /bs./300.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference Computer Vision, pp. 350-368. Springer, 2022.
Minzhao Zhu, Binglei Zhao, and Tao Kong. Navigating to objects in unseen environments by dis- org. http://..../.a..e////////a 2202.037333 tance prediction, 2022. URLhttps http://wwwww./.aaa.s.seseee/eeee
15


Reviews:
[{"review_id": "hCa9Jyc2Si", "reviewer": "Reviewer_j4ao", "summary": "This paper proposes \"ImagineNav\", a mapless navigation framework for robots using vision-language models (VLMs) to perform object search in open environments. It replaces traditional perception + mapping and text-based LLM-based planning with a pipeline where future scene views are \"imagined\" using novel view synthesis and analyzed by a VLM to choose optimal next waypoint. The candidate waypoints are generated by the Where2Imagine module, which is learned from human-like navigation patterns. Tested on benchmarks like HM3D and HSSD, ImagineNav significantly outperformed baselines in terms of success rate and success rate weighted by path length.", "questions": "- Some details are very unclear, especially on the novel view synthesis (i.e. future view imagination). What data is used to train the diffusion model? Is it in or out of distribution for indoor navigation? How about resolution and the speed? \n- More qualitative visualizations of the environments, imagined views, waypoint distribution would be nice.\n- The structure of the VLM analysis output in Fig. 3 and Fig. 4 are inconsistent. Which one is used in evaluation?\n- Why is the success rate lower with NVS model added in Table. 2 (row 3 vs 5)? More explanation is needed.\n- How does this method compare with end-to-end approaches like SPOC [1]?\n\n[1] Ehsani, Kiana, et al. \"SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The system design significantly reduces complexity of an open-world object-goal navigation pipeline, and better integrates sensor observation with large models.", "weaknesses": "There are 3 fundamental limitations of this approach\n- First, there is no way to guarantee that the diffusion model used to synthesize the novel views understands object permanence. How often does it hallucinate non-existing object or leave out objects that should be there? How does the rest of the pipeline deal with this?\n- There is no way to ensure the waypoints generated by Where2Imagine is reachable, especially in out-of-distribution scenarios. The method still needs sine kind of local map for low-level path planning. Some tests on real robot or new environments will erase this concern.\n- The method does not deal with the ambiguity of object reference. For example, navigate to the chair (example used in Fig. 2) is very ambiguous as there might be many chairs in the environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FnX9caf4DX", "reviewer": "Reviewer_4kMy", "summary": "This paper introduces a framework called ImagineNav, designed to enable effective visual navigation for robots in a mapless, open-vocabulary setting. ImagineNav leverages Vision-Language Models (VLMs) with on-board RGB/RGB-D camera inputs, without complex mapping and localization procedures. Instead of traditional planning approaches, it translates the navigation task into a series of \"best-view\" image selection problems, where the VLM selects optimal viewpoints based on imagined future observations. This approach views navigation as a simplified selection problem and showcases the potential of imagination-driven guidance for robotic systems.", "questions": "No extra problems.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a relatively novel navigation method: using novel view synthesis as a form of \"imagination\" for indoor navigation.\n2. Due to the inclusion of a VLM and diffusion model, this work achieves a higher success rate in open-ended tasks.\n3. The writing in this paper is clear, the illustrative figures are accurate, and the explanations of the proposed framework are well-articulated.", "weaknesses": "1. There are some concerns about the imagination module. The ImagineNav framework relies on the diffusion models for novel view synthesis. However, these could lead to some mistakes e.g. non-existing objects in the scene.  This could lead to incorrect navigation decisions by the VLM and reduce the overall success rate.\n2. A small concern about this approach is its performance in multi-room or occluded scenarios. The use of human habits without interactive perception could cause the robot to become trapped in local optima, preventing it from locating the target.\n3. The framework seems to only rely on immediate views for navigation decisions, without fully utilizing historical information from the navigation process. This lack of memory may limit the robot's ability to explore effectively and plan global paths over long distances or in intricate environments.\n\nIn summary, the method is novel (only to me, but if other reviewers show some related work, I will defer it). But the quality of the generated novel view (especially the emergence of non-existing objects) is concerned. If the author can explain more about this, I would increase the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mkL3sGsWY6", "reviewer": "Reviewer_Va7n", "summary": "This work presents an innovative framework for mapless visual navigation that utilizes Vision-Language Models (VLMs) to streamline open-vocabulary navigation. Unlike conventional approaches that depend on mapping and text-based planning, this method relies solely on RGB/RGB-D inputs, redefining navigation as a task of selecting the best view image. Using the Where2Imagine module, it generates potential viewpoints based on human-like navigation tendencies, enabling the VLM to select the most suitable view to locate target objects. The NVS module then generates potential views of the next waypoint. This approach results in efficient paths without mapping and enhances success rates on standard benchmarks.", "questions": "1. While the provided failure case intuitively demonstrates a failure mode of the method, it would be valuable to include an approximate distribution of failure modes, such as how many failures are due to inaccurate imagined novel views.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This work introduces mapless navigation through the prediction of future 3D waypoints, generating possible image observations at these waypoints, and selecting the most promising next waypoint with VLM. The method is well-motivated and thoroughly described, making it feasible for the community to replicate the results.\n2. The paper includes an extensive experimental evaluation of the HM3D and HSSD benchmarks, with the proposed method achieving notable improvements over baselines on the challenging HM3D and HSSD benchmarks.\n3. A comprehensive ablation study is provided in Tables 2, 3, 4, and 5, highlighting the significance of different components within the proposed pipeline. Showing the effectiveness of using visual prompting and novel view thesis, as well as the waypoint prediction.\n5. Figure 5 also offers some failure cases to help readers understand the method’s limitations.", "weaknesses": "1. The paper claims that object detection, segmentation, and mapping modules increase computational demand for robots. However, the proposed method introduces a computationally intensive novel view synthesis module to generate imagined observations. A comparison of computational load would strengthen this claim.\n\n2. The paper’s organization could be improved for clarity. The text references to Tables and Figures are sometimes distant from the actual tables or figures; repositioning these elements could improve the flow and clarity of the paper.\n\n3. Although GPT4o-mini is a robust VLM model, comparisons with recent open-source VLMs featuring 3D spatial understanding would enhance this work, such as:\n\n* Spatial VLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities\n\n* SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sCdmyL7VWS", "reviewer": "Reviewer_Rhxu", "summary": "This paper introduces ImagineNav, a novel framework that leverages VLMs for open-vocabulary object navigation without relying on explicit mapping or localization. The framework acts on a discrete action space spanning different views, then predicts candidate relative poses from the current observation, then uses NVS to generate images of those poses, and lastly uses a VLM for planning. The problem is posed as a best-view image selection problem for the VLM. Empirical experiments on challenging open-vocabulary object navigation benchmarks demonstrate that ImagineNav outperforms existing methods.", "questions": "Why do you think that waypoints are the correct intermediate representation for navigation and object search?\nWhat is the point of having images from NVS as it limits performance and rather uses the embeddings to train a navigation policy?\nWhat do you think are the limitations of the VLM high-level planning?\nHow can you get the VLM to learn online from additional data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is mostly clear and effective in conveying its points. There is an effective set of baselines indicating thorough experimental evidence but the addition of error bars would be helpful to determine significance. There are also effective sections discussing the failed and successful trajectories and clear ablations of the imagination module and the high-level planning module. The originality of having NVS methods apply to scene-level settings is interesting.", "weaknesses": "I think the main novelty within the method is the where2imagine module because prior work has used VLMs for high-level planning and NVS methods. I think the underlying claim is that waypoint as an intermediate representation (to predict poses and images with NVS) is a better reasoning intermediate than other intermediate representations and better than learning an end-to-end model. I think further investigating this claim could further add novelty to the paper as it would tackle a more fundamental question of are modular systems better than end-to-end models for object search and navigation. I would also try to rewrite the conclusion and portions of the introduction for clarity (paragraph 2). Lastly, the discussion of successful and failed trajectories are useful but I would like to see how to address those failed trajectories within the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)