Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models
Authors: Yue Zhang
Ziqiao Ma
Jialu Li
Yanyuan Qiao
Zun Wang
Joyce Chai
Qi Wu
Mohit Bansal
Parisa Kordjamshidi
Venue: Accepted by TMLR ()

Abstract: Vision-and-Language Navigation (VLN) has gained increasing attention over recent years and many approaches have emerged to advance their development. The remarkable achievements of foundation models have shaped the challenges and proposed methods for VLN research. In this survey, we provide a top-down review that adopts a principled framework for embodied planning and reasoning, and emphasizes the current methods and future opportunities leveraging foundation models to address VLN challenges.
We hope our in-depth discussions could provide valuable resources and insights: on one hand, to document the progress and explore opportunities and potential roles for foundation models in this field, and on the other, to organize different challenges and solutions in VLN to foundation model researchers.

Published in Transactions on Machine Learning Research (12/2024)
Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models
Yue Zhang
zhan 1624 @msu. edu marstin@umich edu
Ziqiao Ma
Jialu Li
jialuli@es cs. unc. edu
Yanyuan Qiao
yanyuan. mmmpuan.giao@uudllaide edu. ou zunwang Ocs. unc. edu chaijy@umich. edu qi. unu @adelaide. edu, ou mbonsal unc. edu
Zun Wang
Joyce Chai?
Qi Wunt
Mohit Bt
Parisa Kordjamshidi't
kordjams Gmsu. edu
Michiian State University
2 University of Michigan
UNC Chapel Hall
4 University of Adelaide
"Equal Contribution
Equal Supervision
Reviewed on OpenReview:
https: opwrreviee........ net/ forum? id- yigehazuuh
Abstract
Vision-and-Languaae Navigation (VLN) has gained increasing attention over recent years and many approaches have emerged to advance their development. The remarkable achieve- ments of foundation models have shaped the challenges and methods for VLN research. In this survey, we provide a top-down review that adopts a principled framework for embodied planning and reasoning, and emphasizes the current methods and future opportunities lever- aging foundation models to address VI.N challenges. We hope our in-depth discussions could provide valuable resources and insights: on the one hand, to document the progress and ex- plore opportunitics and potential roles for foundation models in this field, and on the other, to organize different challenges and solutions in VLN to foundation model researchers
1 Introduction
Developing embodied agents that are capable of interacting with humans and their surrounding environments
is one of the longstanding goals of Artificial Intelligence (AI) (Nguyen et al. 2021 Duan et al. 2022). These AI systems hold immense potential for real-world applications serve as multi-fucctiona assistants daily life, such as household robots (Szot et al. 2021), self-driving cars (Hu et al. 2023). and personal assistants (Chu et al. 2023 One formal problem setting to advance this research direction is Vision-and- Language Navigation (VLN) (nderson et al. 2018), multimodal and cooperative task that requires the agent to follow human instructions, explore 3D environments, and engage in situated communications under various forms of ambiguity. Over the years, VLN has been explored in both photorealistic simulators (Chang
et al. 2018 Savva et al. 2019 Xia et al. 2018 and real environments (Mirowski et al. 2018 Banerjee et al.
GGitHub repository:

---

Published in Transactions on Machine Learning Research (12/2024)
2021,, leading to number of benchmarks (Anderson et al. 2018 each presents slightly different problem formulations.
Ku et al.
2020
Krantz al.
2020 that
Recently, foundation models (Bommasani et al.,
2021j, ranging from early pre-trained models like
BERT (Kenton Toutanova 2019) contem-
Model
Grounding Reasoning
porary large language models (LLMs) and vision-
language models (VLMs) (Achiam et al.
2023 Rad-
Wannios
Malogue
Left Right y
ford et al. 2021, have exhibited exceptional abili- ties in multimodal comprehension, reasoning, and cross-domain generalization. These models are pre- trained on massive data, such as text, images, au- dio, and video, and could further be adapted for broad range of specific applications, including em- bodied AT lasks (Xu el. al. 2024b). Integrating these foundation models into the VLN task marks piv- otal recent advancement for embodied AI rescarch, demonstrated through significant performance im- provements (Chen et al. 2021b; Wang et al., 2023h Zhou et al. 2024a). Foundation models have also brought new opportunities to the VLN field, such expanding the research focus from multi-modal attention learning and strategy policy learning to pre-training generic vision and language representa-
The due
VLN Agent
Laft
Visual Perception
Navigation Exccution
Language
Language
Response
Instruction
The through
livting room area halhoog.
left front?
right, then right
Physical Environment
Human
Figure 1: Organizing challenges and solutions in VLN using LAW framework (Hu & Shu 2023
tions, hence enabling task planning, commonsense reasoning, as well as generalize to realistic environments,
Despite the recent impact of foundation models on VLN rescarch, the previous surveys on VLN (Gu et al. 2022 Park & Kim 2023; Wu el al. 2024p are from the pre-foundation--oode era and mainly locus on the VLN benchmarks and conventional approaches, i.e., they are missing a comprehensive overview of the existing methods and opportunities leveraging foundation models to address VLN challenges. Espccially with the emergence of LLMs, to the best of our knowledge, VL. tasks. Moreover, unlike previous efforts that discuss the VIN task objective of this survey is twofold: first, to milestone the progress and explore opportumities and potential roles for foundation models in this field; second, to organize different challenges and solutions VLN to foundation model researchers within a systematic framework. To build this connection, we adopt the LAW framework (Hu & Shu 2023), where foundation models serve as backbones of world model and agent model. This framework offers & general landscape of reasoning and planning in foundation models, and closely scoped with the core challenges in VLN.
review has yet discussed their applications in isolated downstream task, the
Specifically, at cach navigation step, the AI agents perceive the visual environment, receive language in- structions from humans, and reason upon their representation of the world and humans to plan actions and efficiently complete navigation tasks. As shown in Figure [ a world model is an abstraction that agents maintain to understand the external environment around them and how their actions change the world state (Ha & Schmidhuber 2018 Koh et al.) 2021]]. This model is part of broader agent model, which also incorporates human model that. interprets the instructions of its human partner, thereby informing the agent's is goals (Andreas) 2022 Ma et al. 2023). To review the growing body of work in VLN and to understand the milestones achieved, we adopt a top-down approach to survey the field, focusing on fundamental challenges from three perspectives:
Learning world model represent the visual environment and generalize unseen ones.
Learning a human model to effectively interpret human intentions from grounded instructions.
Learning VLN agent that leverages its world and human model ground language, communicate, reason, and plan, enabling it to navigate environments instructed.
2

---

Published in Transactions on Machine Learning Research (12/2024)
World
Human
VLN Agent
Dataset
Name
Domain
Environment
Turn
Format
Gran.
Type
Act. Sp.
Other
Text
Route
LANI/CHAI
20118
Indoors
CHALET
Single
Multi lustr
A
Dise
Mani
R2R
2015
Indoors
Matterport3D
Single Multi Instr
A
Robot
Graph
H
P
R4R 2013
Indoors
Matterport33
Single
Multi lnstr
A
Robot
Graph
H
P
RxR
A120
Indours
Matterport3D
Single
Multi
A
Robot
Graph
SOON
2021a
Indoors
Matterport3D
Single Multi Instr
G
Robot
Graph
H
P P
REVERIE
202ub
Indoors
Matterport:3
Single
Multi Instr
A. G
Robot
Graph
Detect
H
VNLA
2019
Indours
Matterport3D
Multi Multi Instr
Robot
Graph
HANNA
2019
Indoors
Matterport.3D
Multi Multi Instr
A, G
Robot
Graph
H
P H
CVDN
2020
Indoors
Matterport:3
Multi Restricted
A
Robot
Graph
H
VLN-CE
2
Indoors
Habitat, Mattcrport3D
Single
Multi Instr
A
Robot
P
Robo- VL.
2021
Indoors
Habitat, Matterport3D
Single Multi Instr
A
Robot
Cont
H
P
RobotSlang
202
Indoors
Real
Multi Freeform
A
Robot
Disc
H
P
ALFRED
2020
Indoors
A12-THOR
Singlo
Multi Instr.
Robot
Mani
P
TEACh
2022
Iudours
A12-THOR
Multi
Freeform
A. C
Robot
Disc
Mani
H
DialFRED
2122
Indoors
AI2-THOR
Multi
Restricted
A, G
Robot
Disc
Mani
H.
T
TouchDown
2UTV
Outdoors
Google Street View
Single
Multi Instr
A
Graph
Street Nav
ZUZ0
Outdours
Google Street View
Multi Multi Justr
A
Disc
T
Talk2Nav
2121
Outdoors
Google Street, View
Single Multi Instr
A, G
Disc
H
P
TtW
2018
Outdoors
Real
Multi Freeform
LCSD
2019
Outdoors
CARLA
Single
Multi lnstr
A
Driving
Disc
H
CDNLI
2120
Outdoors
CARLA
Multi Multi Tnstr
A, G
Driving Cont
H. T
SDN
2122
Outdoors
CARLA
Multi
Freeform
Driving Disc, Cont
AerialVLN
20233b
Outdoors
AirSim
Single
Multi Instr
C
Acrial Disc
ANDH
20223
Outdours
xView
Multi Freeform
A, G
Aerial Disc
H
Table 1: A summary of existing VLN benchmarks, taxonomized based on several key aspects: the world in which navigation occurs, the type of human interaction involved, the action space and tasks assigned to the VLN agent, and the methods of dataset collection. For the world, we consider their domain (either indoors or outdoors) and the environment. For the human, we consider their turns of interaction (either single or multiple turn), the format of communication (frecform dialogue, restricted dialogue, or multiple instructions), and the language granularity (action-directed and goal-directed). For the VLN agent, consider their agent types (e.g., household robot, autonomous driving vehicles, autonomous acrial vehicles), their action space (graph-based, discrete or continuous), and other additional tasks (manipulation and object detection). For dataset collection, we consider the text collection (by human or templated) and the route demonstrations (by human or planner).
We present a hierarchical and fine-grained taxonomy in Figure 2.2 to discuss challenges, solutions, and future directions based on foundation models for each model. To organize this survey, we start with a brief overview of the background and related research efforts as well as the available benchmarks in this field ($2) We structure the review around how the proposed methods have addressed the three key challenges described above: world model ($3), human model ($4), and VLN agent ($5). Finally, we discuss the current challenges and future research opportunities, particularly in light of the rise of foundation models ($6).
2
Background and Task Formulations
In this section, discuss the background, clarify the scope of this survey, define the VLN problem, and briefly overview the benchmarks.
2.1 Cognitive Underpinnings of VLN
Humans and other navigational animals demonstrate early understanding and strategies for navigating their environments (Rodrigo 2002 Brand et al., 2015 Lingwood et al.) 2018). For example, Gallistel (199)) describes two basic mechanisms: piloting, which involves environmental landmarks and computes distances and angles; and path inlegration, which calculates displacement and orientation changes through self-motion sensing. Central to understanding spatial navigation is the cognitive map hypothesis, suggesting that the brain forms a unified spatial representation to support memory and guide navigation (Epstein al. 2017 Bellmund et al. 2018). For instance, Tolman (1948) observed that rats could adopt the correct novel path when familiar paths are blocked and landmarks are absent. Neuroscientists also discovered hippocampal
3

---

Published in Transactions on Machine Learning Research (12/2024)
place cells, indicating spatial coordinate system that encodes landmarks and goals allocentrically (O O'Keefe & Dostrovsky 1971 O'keefe & Nadel 1978). Recent studies also propose non-Fuclidean representations, e.g. cognitive grophs, which illustrate the complexity of how we represent spatial knowledge of the world (Warren
2019 Ericson & Warren 2020). While visual and auditory perceptions are obviously integral to spatial repre- sentation (Klatzky et al. 2006); our linguistic skills and spatial cognition are also closely intertwined (Pruden ct al., 2011). For instance, rescarchers have shown that understanding different aspects of spatial language can help with space-related tasks (Pyers al., 2010), and that language influences how children interact with space by assisting them to recognize the importance of landmarks in identifying locations (Shusterman et al., 2011). Studying VIN not only enhances the development of embodied AT that follows human instructions in visual environments, but also deepens our understanding of how cognitive agents develop navigation skills, adapt to different environments, and how language use is connected to v visual perceptions and act ions.
2.2 Relevant Tasks and Scope of the Survey
Following natural language navigation instructions has traditionally been modeled using symbolic world reprosentations such as maps (Anderson ct al.] 1991 MacMahon et al.] 2006) Paz-Argaman & Tsarfaty 2019). However, our survey focuses on models that. employ visual environments and address the challenges of multimodal understanding and grounding. Likewise, we redirect readers to extensive surveys on visual
2022a Zhu et al. 2022) and mobile robot navigation (Gul navigation (Zhu et al.] [2221] Zhang et al.
ct al. 2019 Crespo et al. 2020 Möller et al. 2021 which concentrate on visual perception and physical embodiment. However, these studies provide minimal discussions on the role of language in navigation tasks. While we inevitably extend our discussions of VLN to encompass areas beyond navigation, such as mobile manipulation and dialogue, our primary focus remains on navigational lasks, for which we provide a detailed literature review. Besides, unlike previous VLN surveys (Gu al., 2022 Park & Kim 2023 Wu et al. 2024). which offer bottom-up summary focusing on benchmarks and modcling innovations, our survey adopts top-down approach, and uses the roles of foundation models to calegorize the research efforts into three fundamental challenges from the aspects of the world model, the human model, and the VLN agent. Note that this survey concentrates on frontier methods associated with the rise of foundation models. Thus, we point to the carlier generation of models (e.g., LSTM-based methods) very briefly at the beginning of cach section to motivate our discussions.
2.3
VLN Task Formulations and Benchmarks
VLN Task Definition. A typical VLN agent receives a (sequence of) language instruction(s) from human instructors at a designated position. The agent navigates through the environment using an egocentric visual perspective. By following the instructions, its task is to generate trajectory over a sequence of discrete views or lower-level actions and control (e.g., FORWARD 0.25 meter) to reach the destination, which is considered successful if the agent arrives within a specified distance (e-g., 3 meters) from the destination. Besides, the agent may exchange information with the instructor during navigation, either by requesting help or engaging in freeform language communication. Additionally, there has been an increasing expectation for VLN agents to integrate additional tasks such as manipulation (Shridhar et al. 2020) and object detection (Qi et al. 2020b), along with navigation.
Benchmarks. Unlike other multimodal tasks such as VQA, which have a relatively fixed task definition and format, VL. encompasses wide range of benchmarks and task formulations. These distinctions intro- duce unique challenges in addressing the broader VLN task and must be clearly understood as prerequisites for developing effective methods with appropriate foundation models. As is summarized in Table [ exist- ing VLN benchmarks can be taxonomized based on several key aspects in the LAW framework: (1) the world where navigation occurs, including the domain (indoors or outdoors) and the specifics of the envi- ronment. (2) the lype of human interaction involved, including the interaction turns (single or multiple), communication format (freeform dialogue, restricted dialogue, or multiple instructions). and language gran- ularity (action-directed or goal-directed).
(3) the VLN agent, including its types (e.g., houschold robots, autonomous driving vehicles, or autonomous acrial vehicles), action space (graph- based, discrete, or contin- uous), and additional tasks (manipulation and object detection). (4) the dataset collection, including text
4

---

Published in Transactions on Machine Learning Research (12/2024)
History and Memory
Hideey Zianciigg
The Role of Foundation Models
Grgb-lased
dest WET
Data and Knowledge
Decision Making
mmendo
World Model
Eeviectrnest Asgroensation
preprooess, angment, edit existing data: syythhoize new ditt: levurage aequined commencetean knonelallge
a dizlogre managee to week information:
a navigation planner;
La-gaugggg gee Nas Navigation Methods
Pretriized Viazd Reprosziallon
a gener ayjurppoo
Beneralize
decision maniiiiiiikkingggg agent.
Genernlization Ability
Piwerained Teet Reprosstalian
pre-training data.
throuugh Reiing
Human
Indractio Hyrihob
the ballway.. right, turm right enter
"G to the bathroom,
Representation
Task Learning
Model
Walk through the Hising Foom crea into the boilsse.
peneralizable text representations:
langusge grounding
Ambiguons Instraction
Perpytuud Cunteal
Mitur forward look ground Forward
Where is the hallway?
gonemnlaable visral regresetatione; learn, maintain, or process the history and memoryy information.
tasks whot
Learning emboodied
Vision -oooo--ndd-a waatp--nn---n-nnnn
Inforaaikus Betking
learatigg, coctest
learainng or traing.
Grounding and Reseaning
Exylick Groanding
Tiere right, ten furn right.
Challenges and Future Work
VLN Prettaining
c Firwwadd Farward.
Forward Right....
Forwward Forward Right.
Benchmarks
Limitations of Data and Task
Graph Saazner
Model
Adapting LLMs VLMS
Planning
Enter the pecill
Agent
LLM Parav
Which
Model
Instraction Dialogue
The moom the left the fruus?
Agyers
World Model
2D World 3D Worl
Agent Models
Leit
Agosts
Zight
Deployment
Simolation - Real Robots
Figure VLN challenges and solutions within the framework of world model, human model, and VLN agent. We discuss history and memory in the world model, ambiguous instructions in the human model, generalization ability in them both. For the VLN agent, we discuss methods for grounding and reasoning, planning, and adapting foundation models as agents. Depending the role served by the foundation models, we categorize these methods into four types. Additionally, we discuss the potential future of the foundation model for the VLN task.
collection method (human--eneraaae or templated) and route demonstrations (human-performed or planner- gencrated). Ropresentatively, Anddrson et al. (2018) create the Room-to-Room (R2R) datasct basod on the Matterport3D simulator /Chang et al. 2018), where an agent needs to follow fine-grained navigation instruc- tions to reach the goal. Room-across- Room (RxR) (Ku et al. 2020 is multilingual variation, including English, Hindi, and Telugu instructions. It offers a larger sample size and provides timo-aligned instructions for virtual poses, enriching the task's linguistic and spatial information. Matterport3D allows VLN agents to operate in a discrete enwironment. and rely on pre-defined connectivity graphs for navigation, where agents travel on the graph by teleportation between adjacent nodes, referred to as VLN-DE. To make the simplified setting more realistic, Krantz et al. (2020): Li et al. (2022c): Irshad et al. (2021) propose VLN in continu- ous encironments (VIN--C) by transferring discrete R2R paths to continuous spaces (Savva al. 2019). Robo-VIN [Irshad et al.) 2021 further narrows the sim-to-real gap by introducing VLN with continuous action spaces that are more realistic in robotics settings. Recent VLN benchmarks have undergone several
design changes and expectations, which we discuss in
Evaluation Metrics. Three main metrics have been employed to evaluate navigation wayfinding perfor- mance (Anderson et al., 2018): (1) Navigation Error (NE), the mean of the shortest path distance between the agent's ' final position and the goal destination; (2) Success Rate (SR), the percentage of the final position being close enough to the goal destination; and (3) Success Rate Weighted Path Length (SPT.) which normal- izes success rate by trajectory length to balance both the success rate in reaching the correct destination and the efficiency of the path. Some other metrics are used to measure the faithfulness of instruction following and the fidelity between the predicted and the ground-trute trajectory, for example: (4) Coverage Weighted by Length Score (CLS) (Jain et al.) 2019) measures how closcly an agent's 's trajectory follows the reference path. It balances two key aspects of the agent's performance: the exlent of coverage of the reference path and the efficiency of the agent's s navigation by considering the length score; (5) Normalized Dynamic Time Warping (nDTW) (Tharco et al., 2019), which penalizes deviations from the ground-truth trajectories; and (6) Normalized Dynamic Time Warping Weighted by Success Rate (sDTW) (Ilharco et al. 2019 which penalizes deviations from the ground-truth trajectories and also considers the success rate.
5

---

Published in Transactions on Machine Learning Research (12/2024)
2.4 Foundation Models
Foundation models are trained on large-scale datasets, which show strong generalization capability for a wide range of downstream applications. Text-only foundation models, such as pre-trained language models like BERT (Kenlon& & Toutanoval 2019h and GPT-3 (Brown et al. 2020), have revolutionized the field of NLP by setting new benchmarks for tasks like text generation, translation, and understanding. Building the success of these models, vision-language (VL) foundation models, like LXMERT (Tan & Bansal, 2019), CLIP (Radford et al.; 2021) and GPT-4 (Achiam et al.
2023), have expanded the paradigm to multimodal learning by integrating both visual and textual data, proving particularly impactful in various
VL applications (Li et al. 2019a Ramesh et al. 2021 Alayrac et al.) 2022 Hong et al.) 2021 Zhang et al.,
2025 Cheng et al. 2024 Kamali & KKrdjamshidi 2023). For a more comprehensive overview of foundation
models and their applications, we encourage readers to refer to existing survey papers such as Bommasani
ct al.
2021
Du et al. (2022) and Zhou et al. (2023).
3
World Model: Learning and Representing the Visual Environments
A world model helps the VLN agent to understand their surounding environments, predict how their actions would change the world state, and align their perception and actions with language instructions. Two challenges have boen highlighted in existing work about lcarning a world model: encoding the visual history of observations within the current episode as memory, and achieving generalization to unseen enviromments.
3.1 History and Memory
Different from other vision-language tasks like Visual Question Answering (VQA) (Antol et al. 2015). Visual Entailment (Xie et al. 2019). the VLN agent needs to incorporate the history information of past actions and observations into its current step's input to determine the action rather than solely considering the image and text in a single step. Prior to employing the foundation models in VLN, LSTM hidden states served implicit memory supporting agents' decision-making during navigation, and researchers further design
different attention mechanisms (Tan et al. 2019 Wang et al. 2019) auxiliary tasks (Ma et al. 2019 Zhu
et al. 2020 to improve the alignment between the encoded history and instructions.
History Encoding. Different techniques have been proposed to encode navigation history using foun- dation models. A multi-modal Transformer is built upon encoded instructions and navigation history for makinn, which is usually initialized from a model pre-trained on in-domain instruction-trraecton data like Prevalent (Hao et al. 2020). Some approaches encode the navigation history in recurrently updated state tokens. Hong et al. (2021) proposes to utilize a single [CLS] token from last step for encoding the history information, while Lin et al. (2022a) introduces a variable-length memory framework that stores multiple action activations from previous steps in a memory bank as the history encoding. Despite their effectiveness, these methods are limited by the need for step-by-step token updates, making it challenging to efficiently retrieve history encodings at arbitrary steps in the navigation trajectory, which can hinder scalability in pre-training. Another line of work directly encodes navigation history as a sequence with multi-modal Transformer. Among them, Pashevich et al. (2021) encodes single-view images for each step in a trajectory. [Chen et al. (2021b) further proposes a panorama encoder to encode the panoramic visual observation at each time step, followed by a history encoder to encode all the past observations. This hier- archical design separately processes the spatial relationship in a panoramic view and the temporal dynamics across panoramas in the navigation history. Besides, this method eliminates the dependency on recurrently updated state tokens for history encoding, facilitating efficient and large-scale pre-training on instruction- path pairs. Follow-up rescarch replaces the panorama encoder with mcan pooling of images (Kamath et al. 2023) or front-view image encoding (Qiao el. al., 2022), both maintaining ellective navigation performance. With the advent of LLM-based navigation agents, some works (Zhou et al.) 2024b) focus on converting the visual environment into textual descriptions, and explaining the world with text became the trend. The navigation history is then encoded as a sequence of these image descriptions, along with relative spatial information such heading, elevation, and distance. HELPER (Sarch et. al. 2023) designs external
6

---

Published in Transactions on Machine Learning Research (12/2024)
memory of language- program pairs that parses the free-form human-robot dialogue into action programs through retrieval-angmented LLM prompting.
Graph-based History. Another line of research enhances the navigation history modeling with graph information. For example, some of these techniques utilize structured Transformer encoder to capture
the geometric cues in the environment. (Chen el. al. 2022c
Deng el al. 2020 Wang el al.
2023b
Zhou &
2023 Su et al. 2023 Zheng et al.) 2024b Wang et al. 2021 Chen et al. Mu
2021a Zhu et al.
2021a
In addition to the topological graph used in encoding, many works propose to
include the top-down view
information (e.g., grid map (Wang et al. 2023g] Liu et al.) 2023a semantic map
Hong et al. 2023a
Huang
et al. 2023a Georgakis et al. 2022 Anderson et al. 2019 Chen et al.] (202aa] Irshad et al. 2022 local
metrics map (An et al.] 2023) ). and local neighborhood map (Gopinathan al.
2023) in modeling the
observation history during navigation. Recent advances in LLM-based navigation agents have introduced
innovative approaches to memory construction using maps. For instance, Chen et al.
(0024a proposes
novel map-gnided GPT-based agent that utilizes a linguistical-formed map to store and manage topological
graph information. MC-GPT (Zhan et al.) 2024b) introduces a topological map as the memory structure to record information about viewpoints, objects, and their spatial relationships.
3.2 Generalization across Environments
One main challenge in the VLN is learning from limited available environments and generalizing to new and unseen environments. Many works demonstrate that learning from semantic sogmentation features (Zhang et al. 2021a), dropout information in the environment during training (Tan et al. 2019), and maximizing the similarity between semantically--liguni
image pairs from different environments (Li et al. 2022a) improve agents' generalization performance to unseen environments. These observations suggest the need to learn from large-scale environment data to avoid overfitting to training environments. Next, we discuss how existing works collect new environment data, and utilize training.
Pre-trained Visual Representations. Most works obtain vision representations from ResNet pre- trained on ImageNct (Anderson et al.; 2018 Tan et al.) 2019). Shen et al. (2022) replace ResNet with the CLIP visual encoder (Radlord et al. 20211, which pre-trained with contrastive loss between image- text pairs and naturally better aligns the image with the instructions, boosting the VL. performance. Wang al. (2022b) further explores transferring vision representation learned from video data for VLN task, suggesting that temporal information learned from video is crucial for navigation.
Environment Augmentation. One main line of research focuses on augmenting the navigation environ- ment with auto-generated synthetic data. EnvEdit (Li et al., 2022b), EnvMix (Liu al. [202)), KED (Zhu et al. 2023). and FDA (He et al.) (0244]) generate synthetic data by changing the existing environments from Matterport3D. Specifically, they mix up rooms from different environments, change the appearance and style of the environments, and interpolate high-frequency features with the environments. Pathdreamer (Koh) et al. 2021)) and SE3DS (Koh et al. 2023) further synthesize the environments in future steps given current observations and explore utilizing the synthesis view as augmented data for VLN training.
The learning paradigm from the collected environments has changed with the advances in foundation models. Prior to the prevalence of pre-training in foundation models, most works directly augment the training environment with the auto-collected new environments and fine-tune LSTM-basee VLN agent (Li et al.
2022b Liu et al.] 2021 Koh et al. 2021 2023 Zhu et al. 2023 As pre-training has been demonstrated to be crucial for foundation models, it has also become a standard practice in VLN to learn from collected
2024 Kamath el al.) 2023 Chen el. al. 2022b Wang et al. 2023h Lin et al., 2023b Gulur et al. 2021a He et al. 2024a).. Large-scale pre- pre-training with augmented in-domain data has bocome crucial bridging the gap between agents' and humans' performance. The in- domain pre-trained multi-modal transformer has been proven be more effective than the multi-modal Transformer initialized from VLMs, like Oscar (Li et al. 2020 and LXMERT.
environments during the pre-training stage (Li & Bansa]

---

Published in Transactions on Machine Learning Research (12/2024)
4 Human Model: Interpreting and Communicating with Humans
In addition to learning and modcling the world, VLN agents need human model that comprehends human- provided natural language instructions per situation to complete navigation tasks. There are two main challenges: resolving ambiguity and generalization of grounded instructions in different visual environments.
4.1 Ambiguous Instructions
Ambiguous instructions mainly arise in single-turn navigation scenarios, where the agent follows an initial instruction without further human interaction for clarification. These instructions lack the flexibilitty to train the agent to adapt its language understanding and visual perception to the dynamic environments. For instance, instructions may contain landmarks invisible at the current view or indistinguishable landmarks visible from multiple views (Zhang & Kordjamshidi 2023). The issue of ambiguous instructions is barely addressed before the application of foundational models to VLN. Although LEO (Xia et al. 2020) attempts to aggregate multiple instructions to describe the same trajectory from different perspectives, it still relies on human-annolated instructions. However, comprehensive percept-ual context and commonsense knowledge from foundational models enable the agent to interpret ambiguous instructions using external knowledge, as well as seek assistance from other human models.
Perceptual Context and Commonsense Knowledge. Large-scale cross-modal pre-trained models like CLIP are capable of matching visual semantics with text. This enables the VLN agent to utilize information from the visual objects and their states in the current perception to resolve ambiguity, especially in single- turn navigation scenarios. For example, VLN--rrans (Zhang & Kordjamshidi 2023) constructs easy-to- follow sub-instructions with visible and distinctive objects obtained from CLIP to pre-train a Translator that converts original ambiguuus instructions into easily understandable sub-instruction representations. LANA+ (Wang et al. 2023f) leverages CLIP to query a text list of landmark semantic tags with the visual panoramic obscrvations, and selects the top-ranked retrieved textual cues as representations of the salient landmarks to follow. KERM (Li et al., 2023a] proposes a knowledge-onhaneed reasoning model to retrieve facts where knowledge is described by language descriptions for the navigation views. NayHint (Zhang et al. 2024b) constructs hint dataset, providing detailed visual descriptions to help the VLN agent build a comprchensive understanding of the visual environment rather than focusing solely on the objects mentioned in the instructions. On the other hand, the commonsense reasoning ability of LLMs can be used to clarify or correct ambiguons landmarks in the instructions, and break instructions into actionable items. For example, Lin et al. (2024b) use LLMs to provide commonsense about open-world landmark co-occurrences and conduct CLIP-driven landmark discovery accordingly. SayCan (Ahn et al. 2002)) breaks an instruction into a ranked list of pre-defined admissible actions and combines them with an affordance function that assigns higher weights to the objects appearing in the current scene.
Information Seeking. While ambiguous instructions can be resolved based on visual perception and situational context, another more direct approach is to seek help from the communication partner, i.c.,
the human speakers who generate the instructions (Nguyen & Daumé III, 2019: Paul et al. 2022) There are three key challenges in this line of work: (1) deciding when to ask for help (Chi et al. 2020 (2) generating information-seekiee
questions, e.g., next action, objects, and directions (Roman et al. 2020
Singh et al. 2022); (3) developing an oracle that provides the queried information, which could be either real humans (Singh et al. 2022), rules and templates (Gao et al.) 2022), or neural models (Nguyen & Daumé III, 2019). LLMs and VLMs could potentially fit two roles in this framework, either information-seeking models, or proxies for human helpers information-providiin models. Preliminary research has explored model, addressing determining both when and what to ask. This
the use of LLMs as the information is achieved with the help of techniques including conformal prediction (CP) (Ren et al. 2023) or in-context learning (ICL) (Chen et al. 2023cp. For the latter, foundation models play the role of a helper who has access to oracle information, such as the location of the destination and a map of the environment, which is not available to the task performer. Very recently, VINNCopilot (Qiao et al. 2024) enables agents to actively seek assistance when encountering confusion, with the LLM serving as copilot to facilitate navigation. Fan et. al. (2023b) demonstrate that GPT-3 can decompose ground-truth responses in the training data
8

---

Published in Transactions on Machine Learning Research (12/2024)
step-by-step, which helps in training oracle model using a pre-trained SwinBert (Lin al 2022b video- language model. They also demonstrate large vision-language models like mPI.UG-Owl (Ye et al. 2023) can serve as strong zero-shot oracles off the shelf. In addition, self-motivated communication agents have been developed (Zhu et al., 2021c) by learning the confidence of the oracle to produce a positive answer, which enables a sel-QQ&A manner where the oracle can be removed at inference time.
4.2 Generalization of Grounded Instructions
The limited scale and diversity of navigation data is another significant issue affecting the VLN agent's ability to comprehend various linguistic expressions and follow instructions effectively, particularly in unseen navigation environments. Although the language style itself has good generalization capability across seen and unseen environments (Zhang et al. 2021a), how to ground the instructions with the unseen environments is potentially hard task given the limited scale of training instructions. Foundation models help address these issues through both pre-trained representations and instruction generation for data augmentation.
Pre-trained Text Reprosentations. Before the foundation models, many works roly on text encoders, such as LSTM, to represent text instructions (Anderson et al. 2018 Tan et al.) 2019). The foundation models significantly enhance the VLN agent's language generalization ability through pre-trained represen- tations. For example, PRESS (Li et al. 2019b) fine-tunes the pre-trained language model BERT (Kenton & Toutanova 2019 to obtain text representations that generalize better to previously unseen instructions. The multi-modal Transformers (Tan & Bansal 2019 Lu et. al. 2019) boost methods, such as VLN-BERT (Majum- dar et al., 2020) and PREVALENT (Hao et al. 2020). to obtain more generic vision-lingee linguistic representations by pro-training on largo-scale text-image pairs collected from the web. Airbert (Guhur et al. 2021b) trains ViLBERT-like architecture to learn text representations from image-caption pairs collected Irom the Inter- net. CLEAR (Li et al., 2022a) learns cross-lingual language representations that capture the visual concepts behind the instruction. ProbES (Liang et al.) 2022) self-explores environments by sampling trajectories and automatically constructs the corresponding instruction by filling the instruction templates with movements and object phrases detected by CLIP. Additionally, it leverages prompt-based learning to facilitate fast adaptation of language embeddings. NayGPT-22 (Zhou et al. 2025 ) explores leveraging aitto---ndlannguaee
representations from pre-trained VLMs (InstructBLIP (Dai el al. 2024) with Flan-T5 (Chung et al. 2024)
or Vicuna (Zheng et al. 2023 )) to enhance policy learning for navigation and navigational reasoning.
Instruction Synthesis. Another method to improve the agent's generalization ability is to synthesize more instructions. Early works employ the Speaker- framework (Fried et al. 2018 Tan et al.) 2019 Kurita & Cho 2020 Guhur et al. 2021a) to train an offline speaker (instruction generator) using human- annotated instruction-trajector pairs. It then generates new instructions based on sequences of panoramas along a given trajectory. However, ZZhao et al. (2021) observe that these generated instructions are low- quality and show a poor performance in human wayfinding evaluation. Marky (Wang et al.] 2022a, Kamath et al.] 2023) addresses this limitation using a multi-modal modal extension of the multilingual T5 model (Xue et al. 2020 with text-aligned visual landmark correspondences, achieving near-human quality R2R-style paths in unseen environments. PASTS (Wang et al., 2023c) introduces a progress-aware spatial-temporal Transformer speaker to better leverage the sequenced multiple vision and action features. SAS (Gopinathan et al. 2024) generates instructions with rich spatial information using semantic and structural cues from the environment. SRDF [Wang et al.) 2024ch builds a strong instruction generator with iterative self-training, Additionally, instead of training an ollline instruction generator, some recent research (Liang el al. 2022 Lin et al.] [023bb Zhang & Kordjamshidi] [023 Wang al., 2023e Magassouba et al. (021) generates instructions while navigating. For instance, LANA (Wang al. 2023e) introduces a language-capable navigation agent that not only executes navigation instructions but also provides route descriptions.
5 VLN Agent: Learning an Embodied Agent for Reasoning and Planning
While the world and human models empower visual and language understanding abilitics, VLN agents need to develop embodied reasoning and planning capabilities to support their decision-making. From this
9

---

Published in Transactions on Machine Learning Research (12/2024)
perspective, we discuss two challenges: grounding and reasoning, and planning. We also explore the method of directly applying foundation models as the VLN agent backbone.
5.1 Grounding and Reasoning
Different from other VL tasks, such as VQA and Image Captioning, which primarily focus on static alignment between images and corresponding textual descriptions, the VLN agent needs to reason about spatial and temporal dynamics in the instructions and the environment based on its actions. Specifically, the agent should consider previous actions, identify the part of the sub-instruction to execute, and ground the text to the visual environment to execute the action accordingly. Previous methods primarily rcly on explicit semantic modeling or auxiliary task design to obtain such abilities. However, pre-training with specially designed tasks has become the dominant approach with the advent of foundation models.
Explicit Semantic Grounding. The previous efforts enhance the agent's grounding ability through ex- plicit semantic modeling in both vision and language modalitics, including modcling motions and land- marks (Hong et al. 2020b He et al., 2021 Hong et al. 2020a; Zhang et al. 2021b Qi et al., 2020ab, utilizing syntactic information in the instruction (Li et al.) 2021). as well as spatial relations (Zhang & Kordjamshidi 2022b; An et al.) 2021). Very few works (Lin et al.) 2023a Zhan et al.
2024a Wang et al. 2023b) ex- (023a proposes actional plore explicit grounding in the VLN agent with the foundation models. Lin ct al.
alomic-concept learning and map visual observations to faciliate multi-modal alignments.
Pre-training VLN Foundation Models. Except for explicit semantic modeling, the previous research
also enhances the agent's grounding ability through auxiliary reasoning tasks (Ma et al. 2019 Wu et al.
2021 Zhu et al. 2020 Raychaudhuri et al. 2021 Dou & Peng, 2022 Kim et al. 2021]. Such methods are less explored in VLN agents with foundation models, as their pre-training already provides a general understanding of spatial and temporal semantics prior to navigation. Various pre-training methods with specially designed tasks have been proposed to improve the agent's grounding ability. Lin et al. (2021) intro- duce pre-training tasks specifically designed for scene and object grounding. LOVIS (Zhang & Kordjamshidi 2022ag) formulates two specialized pre-training tasks to enhance orientation and visual information separately, HOP QQia et al. 2022 2023a) introduces history-and-orde
aware pre-training paradigm that emphasizes historical information and trajectory orders. Li & Bansal 2023) suggests that enhancing the agent with the ability to predict future view semantics helps the agent in longer path navigation performance. Dou el al. (2023) design a masked path modeling objective to reconstruct the original path given a randomly masked sub-path. Cui et al. (0223) propose entity-aware pre-training by predicting grounded entities and aligning them text.
5.2 Planning
Dynamic planning enables VLN agents to adapt to environmental changes and improve navigation strategies on the fly. Alongside the graph-based planners that utilize global graph information to enhance local action spaces, the rise of foundational models, particularly LLMs, has brought based planners into the VLN field. These planners use L.Ms'' vast commonsense knowledge and advanced reasoning to create dynamic plans that improve decision-making
Graph-based Planner. Recent vdvanemennts VLNN emphasize enhancing navigational agents' planning capabilities through global graph information. Among them, WWan et al. (2021): ): Chen al. (2022c); Deng
et al. (2020); Zheng et al. (2224b) enhance the local navigation action spaces with global action steps from graph frontiers of visited nodes for better global planning. Gao et al. 20223 further enhances navigation decision-making with high-level planning for zonc selection and low-level planning for nodc selection. More- over, Liu et. al. (2023a) enriches the graph-froniier-based global and local action spaces with grid-level actions for more accurate action prediction. In continuous environments, Krantz et al. (2021); Hong et al. (2022): Anderson et al. (2021) adopt a hicrarchical planning approach utilizing high-level action spaces instead of low-level ones by selecting a local waypoint from a predicted local navigability graph. CM2 (Georgakis et al. 2022 facilitates trajectory planning by grounding instructions within local map. Expanding on this
10

---

Published in Transactions on Machine Learning Research (12/2024)
Wang et al. (003gg) Chang et al. (024); Wang et al. (202cc construct global topological graph or grid maps to facilitate map- based global planning. Additionally, Wang et al.] (2023a] 2024a) predict multiple future waypoints using either a video prediction model or neural radiance reprosentation model to plan the best action based on the long-term effects of predicted candidate waypoints.
strategy, An et al. (2024 2023
LLM-based Planner. In parallel, some studies leverage common-sense knowledge from LLMs to generate lext-based plans (Huang et al., 2022: 2023b). LLM-Planner (Song et al. 2023) creates detailed plans com- posed of sub-goals, dynamically adjusting these plans in real-time by integrating detected objects according to predefined program patterns. Similarly, Mic (Qiao ct al. 2023b) and Aa (Chen et al. 2023b) special- ize in breaking down navigalion lasks into detailed textual instructions, with Mic generating step-by-step plans from both static and dynamic perspectives, while Na uses GPT-3 to parse instructions into ac- tionable sub-tasks. ThinkBot (Lu et al. 2023) employs thought chain reasoning to generate missing actions with interactive objects. VL-Map (Huang et al. 2023a) decomposes navigation instructions into sequential, goal-related functions in code format using code-written LLMs (following the Code-as-Policy [Liang et al. 2023) framework) and utilizes a dynamically built, queryable map to guide the execution of these goals. Additionally, SayNav (Rajvanshi et al.] 2024) builds a 3D scene graph of the explored environment as input to LLMs for generating feasible and contextually appropriate high-level plans for the navigator.
5.3 Foundation Models as VLN Agents
The architecture of VLN agents has undergone significant transformations with the advent of foundation models. Initially conceptualized by Anderssn et al.] (2018), VLN agents were formulated within a Seq2Seq framework, employing an LSTM and an attention mechanism to model the interaction between vision and language modalities. With the advent of foundation models, the agent backend has transitioned from LSTM to Transformer and, more recently, these large-scale pre-trained systems.
VLMs as Agents. The mainstream methodology leverages single-strearn VLMs as the core structure of VLN agents (Hong et al., 2021, Qi et al.) 2021 Moudgil et al., 2021; Zhao et al., 2022). These models process inputs from language, vision, and historical tokens simultaneously each time step. It performs self-attention over these modal tokens to capture the textual-visual correspondence, which is then used to infer the action probability. In the zero-shot VIN, CLIP-NAV (Dorbala et al., 2022) utilizes CLIP to obtain natural language referring expressions that describe the target object and make sequential navigational decisions. VLN-CE agents (Krantz et al., 2020) differentiate themselves from the VLN-DE (Anderson et al., 2018) agents by their action space, executing low-level controls in the continuous environment instead of graph- based high-level actions of view selection. Despite early works (Krantz et al.) 2020 Raychaudhuri et al. 2021) utilizing LSTM to infer low-level actions, the introduction of waypoint predictors has allowed
to transfer methods from DE to CE (Krantz et al. 2021: Krantz & Lee 2022 Hong ct al. 2022 Anderson
et al. 2021 An et al. 2022 Zhang& & Kordjanshidi 2024). All these methods use waypoinl predictor to
obtain a local navigability graph, allowing foundation models in DE to adapt to the continuous environment.
In particular, the waypoint detection process primarily involves using visual observations (e.g., panoramic RGBD images) to predict navigable candidate adjacent waypoints from the agent's current position as possible targets. Given the predicted waypoints, the agent selects one as the current destination.
LLMs as Agents. Since LLMs have powerful reasoning ability and semantic abstraction of the world, and also show strong generalization ability in unknown large-scale environments, recent research VLN has started to directly employ LLMs as agents complete navigation. Typically, visual observations are converted into textual descriptions and fed into the LLM along with instructions, which then perform ac- tion predictions. Innovations such NavGPT (Zhou et al., 2024a) and MapGPT (Chen et al.) 2024aa demonstrate the fcasibility of zero-shot navigation, with NavGPT autonomously generating actions using GPT-4 and MapGPT converting topological maps inlo global exploration hints. DiscussNav {Long el al., 2024b) extends this approach by deploying multiple domain-specific VLN experts to automate and reduce human involvement in navigation tasks. includes Instruction Analysis Experts, Vision Perception Experts, Completion Estimation Experts, and Decision Testing Experts. The use of multiple donain-specific VLN experts distributes tasks among specialized agents, reducing the burden on single model and allowing
11

---

Published in Transactions on Machine Learning Research (12/2024)
for optimized, task-specific processing. This multi-expert approach enhances robustness, transparency, and overall performance by leveraging the collective strengths of multiple large models. MC-GPT (Zhan et al. 2024b) employs memory topology maps and human navigation examples to diversify strategies, while In- structNav [Lnng et al., 2024a breaks navigation into sub-tasks with multi-sourced value maps for effective execution. In contrast to zero-shot usage, some works (Zheng et al. 2024a Zhang et al. 2024a Pan et al. 2024) fine-tune L.Mss to address the embodied navigation tasks effectively. Some studies have incorporated the Chain-of-Thoughh (CoT) (Wei et al., 2022) reasoning mechanism to improve the reasoning process. Nav- CoT (Lin et al., 2024ag transforms LLMs into world model and navigational reasoning agent, streamlining decisions by simulating future environments. This demonstrates the flexibility and practical potential of fine-tuned language models in both simulation and real-world scenarios, marking a significant advancement over traditional applications.
6 Challenges and Future Directions
While foundation models have enabled novel solutions to VLN, several limitations remain under-explored, and new challenges arise. In this section, we outline the challenges and future direction of VLN from the perspectives of benchmarks, the world model, the human model, the agent model, and real robot deployment.
Benchmarks: Limitations of Data and Task. The current VLN datasets have limitations regarding quality, diversity, bias, and scalability. For example, in the R2R datasct, the iistruction-trajcctorry pairs are biased to the shortest path, which may not accurately represent real-world navigation scenarios. We discuss the trends and recommendations on how VLN benchmarks can be improved.
Unified and Realistic Tasks and Platforms. Establishing robust benchmarks and ensuring reproducibil- ity are crucial for evaluating VLN in real-world scttings. Real-world variability necessitates compre- hensive benchmarks reflecting navigation challenges. A universal sim-to-real evaluation platform, like OVMM (Yenamandra et al.] (202),, , is needed for standardized testing across simulated and real-world settings. In addition, the tasks and activities should be realistic and designed originated from human needs. For instance, BEHAVIOR-IK (Li et al., 2024a) presents a benchmark of everyday household activities in virtual, interactive, and ecological environment to address the demands for diversity and realism.
Dynamic Environment. Real-world environments are inherently complex and dynamic, with moving objects, people, and variations like lighting and weather presenting unexpected situations (Ma et al., 2022). These factors disrupt the visual perception of navigation systems and make maintaining reliable performance difficult. Recent efforts like HAZARD (Zhou et al., 2024c). Habitat 3.0 (Puig et al. 2024). and HA-VLN (Li et al. 2024b) consider dynamic environments and provide a good starting point.
Indoors to Outdoors. VLN agents navigating in outdoor environments, e.g., autonomous driving and aerial vehicles, also start to gel more allention (Vasudevan et al., 2021 Li et al. 2024c), with various language-guidee datasets (Sriram et al.) 2019 Ma et al.) 2022) developed. Early studies have attempted to involve LLMs in these tasks, either with prompt engineering (Shah et al. 2023 Sha et al. 2023 Wen
let al.
2023 or by fine-tuning LLMs to predict the next action or plan future trajectories (Chen et al.
2024b Mao et al.] 2023). To adapt off- offethh-sheef VLMs to these outdoor navigation domains, world
driving videos (Xu et al. 2024a) Yuan et al.) 2024b. simulated driving data (Wang et al.] 2023d Shao ct al.] (024) and them both (Sima et al. [023 Huang al., (024b) have boen utilized for instruction
luning so that these foundation models learn to predict future throtile and steering angles. Additional
reasoning and planning modules have also been integrated into foundation model driving agents (Huang
ct al.
2024b Tian et al.] 2024). We refer the readers to surveys and position papers for dctailed
review
Li et al. 2023b
Cui et al. 2024 Gao et al. 2024 Yan et al. 2024
World Model: From 2D to 3D. Building effective world representations is a central research theme in embodied perception, reasoning, and planning. VIN is fundamentally a 3D task, where the agent perceives the real-world environment in 3D. Although the current rescarch represents the world with strong and generic 2D representations, they fall short of spatial language understanding in the 3D world (Zhang et al. 2024d
12

---

Published in Transactions on Machine Learning Research (12/2024)
Many explicit 3D representations are developed in prior work, including various semantic SLAMs and vol-
umetric representation (Chaplot et al. 2020 Min et al.) 2021 Saha et al. 2022 Blukis et al. 2022 Zhang
eet al. |022b] Liu et al., 2024], depth information (An et al. 2023). Bird's Barrd's-Ey-- Eye- View representations like grid
map (Wang ct al. 2023g Liu et al.) 2023a and local metrics map (An et al. 2023
These representations
are limited because they reduce the object set to a closed set, making them inadequate for open-vocabulary settings with natural language. Several studies develop queryable map/scene representations by integrat- ing multi-view image features captured from CLIP into 3D voxel grids (Jatavalaabula et al.] 2023 Chang et al. 2023) top-down feature maps (Huang et al., 2023a Chen et al., 2023a as well utilizing scene graphs (Rana et al.. 2023 Gu et al.) 2024) to represent spatial relationships. However, adapting 3D repre- sentations learned from large-scale data for VLN agents to better perceive the 3D environment is still under exploration. The recent. rise of 3D loundation models (Hong el al. 2023b; Huang et. al.) 2024a Chen el. al.
2024d e), including 3D reconstruction models (Hong et al. 2024] and 3D multimodal representations
Yang
ct al. 2024 Zhang ct al. 2024c cb. can be crucial for VLN.
Human Model: From Instruction to Dialogue. Previous efforts predominantly adopt either a speaker-
listener paradigm or restricted QA dialogue (Thomason ct al. 2020 Gao et al.) 2022) that only allows the agent to ask for help. Recently, there has been a surge in new benchmarks leaturing open-ended dialogue
instructions (De Vries et al. 2018 Banerjee et al.) 2021 Padmakumar et al. 2022 Ma et al. 2022 Fan et al.,
2023a), supporting fully free- form communication where agents can ask, propose, explain, suggest, clarify, and negotiate even in ambignous or confusing scenarios. Still, current approaches rely on cule-bases dialogue
templates to tackle these complexities (Zhang et al., 2023: Parekh et al. 2023 Gu et al. 2023), though
they might feature a foundation model component. Huang et al. (2024b) perform conversational tuning on a video-language model using human-human dialogue data paired with simulated navigation videos, showcasing enhanced dialogue generation capabilities while navigation. Moving forward, it is imperative for future rescarch to integrate foundation models for situated task-oriented dialogue management (Uleer ct al. 2024), or explore existing foundation models for task-oriented dialogue (He et al.) 2022
Adapting Foundation Models for VLN. While foundation models show strong gen- eralizability, incorporating them into navigation tasks remains challenging. LLMs fundamentally lack the capability to visually perceive the actual enviromment and are prone to hallucinations. We als discuss capa- bilitics of LLMs in planning and reasoning.
Agent Model:
Lack of Embodied Experience. This limitation can lead to scenarios where LLMs rely solely on pre- established commonsense for task planning and reasoning, which might not meet specific real-world needs (Xiang et al., 2024). Some pipelines tackle this issue by captioning the visual observations to textual descriptions as prompts for LLMs (Zheng et al.) 2022). with a potential loss of essential visual semantics. Compared with LLMs, VLM agents demonstrate the potential to perceive the visual world and plan (Zhang et al. 20244). Still, these models are primarily developed from internet data, which lack em- bodied experiences (Mu et al.] 2024) and need finetuning for robust agentic decision-making (Zhai et al. 2024). Further research is needed to transfer the commonsense knowledge in foundation model agents to generalize to embodied situations. Recently proposed embodied foundation models (such as Embod- ieGPT (Mu et al. (0044), PaLM-E (Driesss al. (203)) and Octopus (Yang et al.] (025)) offer a promising solution for enabling agents to operate more effectively in interactive environments. They fine-tune foun- dation models across multiple embodied tasks to bridge the gap between an agent's understanding of vision, language, and embodied actions, enhancing the foundation model's ability to comprehend and execute based on multimodal input.
Hallucination Issue LLMs and VLMs might generate non-cxistent objects, leading to misinformation (Li et al. 2023c Chen et al.) 2024ch. For example, when LLM performs task planning, it may generate instructions such as "go forward and turn left at the sofa" even if there is sofa in the room. This inaccuracy may cause them to execute incorrect or impossible actions.
LLMs in Planning and Reasoning. There are some literatures evaluating the zero-shot reasoning and planning capabilitios of LLMs, particularly relation PlanBench (Valmockam et al. 2022) and Co- gEval (Momennejad et al.) 2023), which highlight LLMs' limitations in more complex planning tasks. These works assess LLMs in a variety of challenging settings, such as plan generation, optimality, robust-
13

---

Published in Transactions on Machine Learning Research (12/2024)
ness, and reasoning, and identify that LLMs sometimes struggle with hallucinations fail to grasp the relational structures underlying complex planning problems. In the context of VI.,, the action space and the planning requirements are relatively constrained due to the fixed indoor environments and the limited set of navigational actions. This bounded setting makes it more feasible for LLMs to provide step-by sep-by-stop instructions for coarse-grained directions, which has been demonstrated to be effective in previous works. In VLN tasks, the LLM's role is not to take over the entire planning process, but rather to assist by offering structured breakdown of instructions. The agent's 's actual decision-making remains primarily reliant other components, such perception and motion control. Therefore, in VLN tasks, the LLM's planning serves more as supplementary guide rather than the sole decision-making factor.
Deployment: From Simulation to Real Robots. Simulated settings often lack the complexity and variability of real-world environments, and lower-quality rendered images exacerbate this issue. First, the perception gap results in decreased performance and accuracy, highlighting the need for more robust pereep- tion systems. Wang et al. (2024b) have started to explore the use of semantic maps and 3D feature fields to provide monocular robots with panoramic perception shows improved performance. The embodiment gap and the data scarcity are also bottlenccks. The rise of robot teleportation (He et al. (024b) also provides an alternative to scale up VLN data for foundation models in real human-robot communications.
7 Broader Impact
Foundation models hold great promise for advancing vision-language navigation. However, it is essential to address their broader ethical, legal, and socictal implications. Given that they pro-trained vast, web-scale datasets, these models can carry inherent biases, which may result in fairness concerns, e.g., to multilingual users. Some approaches involve continual model training, it is critical to acknowledge and mitigate any potential risks to user privacy, especially when deployed in real-world applications such as home robotics.
8 Acknowledgement
This work is supported in part by the ARO Award W911NF2110220, NSF grant IIS-1949634, and ONR grant N00014-23-112411 N00014-23-1-2356 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. anXiv preprint arXin:230s.00 08774, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not i say: Grounding language in robotic affordances. ui preprial or arXiv:2204.014 016691,, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Conference Newral Information Processing Systems, volume 35, pp. 23716-23703 2022.
Dong An, Yuankai Qi, Yan Huang, Qi Wu, Liang Wang, and Tieniu Tan. Neighbor-view enhanced model for vision and language navigation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 5101-5109, 2021.
Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, and Jing Shao. 1st place solutions for rxr-habitat vision-and-languugee navigation competition (cvpr 2022). 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Mul- timodal map pre-training for language-guider navigation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 2737 2748, 2023.
Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-languasn navigation in continuous environments. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2024.
Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. The here map task corpus. Language and speech,
366, 1991.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languuae navigation: Interpreting disually-grounde nav- igation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 3674-3683, 2018.
Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, and Stefan Lee. Chasing ghosts: Instruction following as bayesian state tracking. In Conference on Neural Information Processing Systems, volume 32, 2019.
Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim-to-real transfer for vision-and-language navigation. In Conference on Robot Learning, pp. 681. PMLR, 2021.
Jacob Andreas. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769-5779, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Shurjo Banerjee, Jesse Thomason, and Jason Corso. The robotslang benchmark: Dialog-guided robot local- ization and navigation. In Conference Robot Learning, pp. 1384-1393. PMLR, 2021.
Jacob LS Bellmund, Peter Gärdenfors, Edvard Moser, and Christian F Doeller. Navigating cognition: Spatial codes for human thinking. Science, 362(6415)
2018.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction exccution. In Conference on Robot Learning, pp. 706 717. PMLR, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXim:2108 07258, 2021.
Rebecca J Brand, Kelly Escobar, Adrien Baranes, and Amanda Albu. Crawling predicts infants' under- standing of agents' navigation of obstacles. Infancy, 20(4):405-415 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Maleusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsoll, M.F. Balcan, and H. Lin (ods.), Advances in Newral Information Processing Systems, volume 33, pp. 1877 1901, 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Macicj Halber, Matthias Nicbner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 7th IEFE International Conference on 3D Vision, 3DV 2017, pp. 667-676, 2018.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Haonan Chang, Kowndinya Boyalakuntla, Shiyang Lu, Siwei Cai, Eric Pu Jing, Shreesh Keskar, Shijie Geng, Adeeb Abbas, Lifeng Zhou, Kostas Bekris, et al. Context-aware entity grounding with open-vocabulary 3d scene graphs. In Proceedings of the 2023 Conference on Robot Learning (CORL). JMLR, 2023.
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, et al. Goat: Go to any thing. In Robotics: Science and Systems 2024, 2024.
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinv. Object goal navigation using goal-oriented semantic exploration. In Conference on Neural Informalion Processing Systems, volume 33, pp. 4247-4255 2020.
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Danicl Kappler. Open-vocabulary queryable scene representations for real world planning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11509-11522, IEEE, 2023a.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural lan- guage navigation and spatial reasoning in visual street environments. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12538-12547, 2019.
Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong. Mapgpt: Map- guided prompting for unified sisin--nd-languuae navigation. orXiv preprint arXiv:2401007114 2024a.
Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for vision-anddllanuuage navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogmition, pp.
2021a.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamic Shotton. Driving with Ilms: Fusing object-level vector modality for explainable au- tonomous driving. In 2024 IEEE International Conference Robotics and Aulomation (ICRA), pp. 14093-14100. IEEE, 2024b.
Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas H. Li, Mingkui Tan, and Chuang Gan. Weakly-supervieed multi-granularity map learning for dssion-nd-languagg navigation. In Advances in Newral Information Processing Systems, 2022a.
Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. A nav: Action-aware zero-shot robot navigation by exploiting vision-and rision-and-languaee ability of foundation models. 2023b.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-languuae navigation. Tn Advances in Neural Information Processing Systems, pp. 1884458477 2021b.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Learning from unlabeled 3d environments for vision-and- -siionanddlannguaee navigation. In European Conference on Computer Vision, pp.
Springer, 2022b.
Shizhe Chen, Liers Gulur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laplev. Think global, act local: Dual-scale graph transformer for aadon-and--anguuee navigation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16537-16547, 2022c.
Xiaoyu Chen, Shenao Zhang, Pushi Zhang, Li Zhao, and Jianyu Chen. Asking before acting: Gather information in embodied decision making with language models. arXiv preprint wrXin:2305.16655, 2023c.
Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhcy, and Joyce Chai. Multi-object hallucination vision language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Zhimin Chen, Longlong Jing, Yingwei Li, and Bing Li. Bridging the domain gap: Self-supervised 3d scene understanding with foundation models. Advances in Newral Information Processing Systems, 36, 2024d.
Zhimin Chen, Liang Yang, Yingwei Li, Longlong Jing, and Bing Li. Sam-guided masked token prediction for 3d scene understanding. arXiv preprint arXiv:2410.12158, 2024e.
Zixu Cheng, Yujiang Pu, Shaogang Gong, Parisa Kordjamshidi, and Yu Kong. Shine: Saliency-aware hier- archical negalive ranking for compositional temporal grounding. arXiv preprint or arXi:22407.05116, 2024.
Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani- Turk. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pp. 2459 2466, 2020.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXic preprint orXiv:2312.1888 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-inetunee language models. Journal of Machine Learning Research, 25(70):1-55) 11:) 2024.
Jonathan Crespo, Jose Carlos Castillo, Oscar Martinez Mozos, and Ramon Barber. Semantic information for robot navigation: A survey. Applied Sciences, 10(2):497, 2020.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuci-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVE Winter Conference on Applications of Compuler Vision, pp. 958 979, 2024.
Yibo Cui, Liang Xie, Yakun Zhang, Meishan Zhang, Ye Yan, and Erwei Yin. Grounded entity-landmark adaptive pre-training for lision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12043 12053, 2023.
Wenliang Dai, Junnan Li, DONGXU LI, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Conference on Neural Information Processing Systems, volume 36, 2024.
Harm De Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. In Visual Learning and Embodied Agents in Simulation Environments (VLEASE) Workshop at ECCV, 2018.
Zhiwei Deng, Karthik Narasimhan, and Olga Rassakovsky. Evolving graphical planner: Contextual global planning for Assionaadd-lnguua navigation. In Conference Neural Information Processing Systems, volume 33, pp. 20660 20672, 2020.
Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav S Sukhatme. Clip-nav: Using clip for zero-shot vision-anddlanguuage navigation. In Workshop on Language and Robotics at CoRL 2022, 2022.
Zi-Yi Dou and Nanyun Peng. Foam: A follover-aware speaker model for vision-anddlangguage navigation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational 2022.
Linguistics: Human Language Technologies, pp.
Zi-Yi Dou, Feng Gao, and Nanyun Peng. Masked path modeling for lision-and-languge navigation. In Findings of the Association for Computational Linguistics: FMNLP, pp. 15255-15269) 2023.
Danny Dricss, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference Machine Learning, pp. (469-8888,88 2023.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint or Xiv:2200.. 10936, 2022.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied al: From simulators to rescarch tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2): 230 244, 2022.
Russell A Epstein, Eva Zita Patai, Joshua Julian, and Hugo J Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20(11):1504-1513, 2017.
Jonathan D Ericson and William H Warren. Probing the invariant structure of spatial knowlodge: Support for the cognitive graph hypothesis. Cognition, 200:104276, 2020.
Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, and Xin Wang. Aerial vision-and-dialog navigation. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 3043-3061, 2023a.
Yue Fan, Jing Gu, Kaizhi Zheng, and Xin Wang. R2h: Building multimodal navigation helpers that respond to help requests. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 14803 14819, 2023b.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrice Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and- language navigation. In Conference on Neurol Information Processing Systems, volume 31, 2018.
Charles R Gallistel. The organization of learning. The MIT Press, 1990.
Chen Gao, Xingyu Peng, Mi Yan, He Wang, Lirong Yang, Haibing Ren, Hongsheng Li, and Si Liu. Adap- tive zone-aware hierarchical planner for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14911-14920, 2023.
Haoxiang Gao, Yaqian Li, Kaiwen Long, Ming Yang, and Yiqing Shen. A survey for loundation models in autonomous driving. arXiv preprint wrXiv:2402.01103 2024.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gauray S Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters (RA- L), 2022.
Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Danilidiss. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 15439-15449, 2022.
Muraleekrishna Gopinathan, Jumana Abu-Khalaf, David Suter, Sidike Paheding, and Nathir A Rawashdeh. What is near?: Room locality learning for enhanced robot wisoon-aaguugg--nviitttio in indoor living environments. arXiv preprint arXiv:2309.06: 050006,000 2023.
Muraleckrishna Gopinathan, Martin Masck, Jumana Abu-Khalaf, and David Suter. Spatially- speaker for aibbo--and-laguuaee navigation instruction generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13601-13614, 2024.
Jing Gu, Eliana Stefani, Qi Wu, Josse Thomason, and Xin Wang. Vision-and-language navigation: A survey tasks, methods, and future directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 7606-7623, 2022.
Jing Gu, Kaizhi Zheng, Kaiwen Zhou Yue Fan, Xuchai He Jialu Wang Zonglin Di, and Xin Eric Wang. Slugjarvis: Multimodal commonsense knowledge-basee based embodied for simbot challenge. In Aleza Prize SimBol Challenge Proceedings, 2023.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
Pierro-oouis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In- domain pretraining for vision-aundlanguaage navigation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 1634-16444 1644,,63, 2021a.
Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: Tn- domain pretraining for vision-and-langnage navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1634 1643, 2021b.
Faiza Gul, Wan Rahiman, and Syed Sahal Nazli Alhady. A comprehensive study for robot navigation techniques. Cogent Engineering, 6(1):1632046, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Newral Information Processing Systems 31, pp. 2451-2463. 2018
Weituo Hao, Chunyuan Li, Xiujum Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and-languuag navigation via pre-training. In Proceedings of the IEEE/CV conference Computer Vision and Pattern Recognition, pp. 13134 13143, 2020.
Keji He, Yan Huang, Qi Wu, Jianhua Yang, Dong An, Shuanglin Sima, and Liang Wang. Landmark-rxr: Solving viston-and-languuge navigation with fine-grained alignment supervision. In Conference on Neural Information Processing Systems, volume 34, pp. 652 663, 2021.
Keji He, Chenyang Si, Zhihe Lu, Yan Huang, Liang Wang, and Xinchao Wang. Frequency-enhanced data augmentation for vision-ande and- and-language navigation. In Conference on Neural Information Processing Sys- tems, volume 36, 2024a.
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning numan-to-humanoii rcal-time whole-body telcoperation. In 2024 IEEE/RS) /RSJ International Conference on Intelligent Robots and Systems (IROS), 2024b.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, et al. Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 10749-10757, 2022.
Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, and Raia Hadsell. Learning to follow directions in street view. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11773-11781, 2020.
Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity rela- tionship graph for agent navigation. In Conference on Neural Information Processing Systems, volume 33, pp. 7685-7696, 2020a.
Yicong Hong, Cristian Rodriguez, Qi Wu, and Stephen Gould. Sub-instruction aware vision-and-language navigation. In Proceedings of the 2020 Conference Empirical Melhods in Natural Language Processing (EMNLP), pp. 3360-3376 2020b.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo, and Stephen Gould. A recurrent vision-and- language BERT for navigation. In Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, 2021.
Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for lanig-ag navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18439-15444 2022.
19

---

Published in Transactions on Machine Learning Research (12/2024)
Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernoncourt, Trung Bui, Stephen Gould, and Hao Tan. Learning navigational visual representations with semantic map supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3055 3067, 2023a.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3d. In The Twelfth International Conference on Learning Representations, 2024.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. NeurIPS, 2023b.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning- oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, 2023.
Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The law for machine reasoning and planning. NewrIPS 2023 Tutorial, 2023.
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In IEEE International Conference on Robotics and Automation, pp. 10608-10615. IEEE, 2023a.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the Internaiional Conference on Machine Learning (ICML), 2024a.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 0118-91477 PMLR, 2022.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pp. 1769-1782, 2023b.
Yidong Huang, Jacob Sansom, Ziqiao Ma, Felix Gervits, and Joyce Chai Drivime: Enhancing Ilm-based autonomous driving agents with embodied and social experiences. In 2024 IEFE/RS, /RSJ International Con- ference on Intelligent Robots and Systems (IROS), 2024b.
Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene le, and Jason Baldridge. General evaluation for instruc- tion conditioned navigation using dynamic time warping. In Advances in newral information processing systems, 2019.
Muhammad Zubair Irshad, Chih- Cho Ma, and Zsolt Kira. Hierarchical cross cross-modal agent for robotics vision- and-language navigation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 13238 13246. IEEE, 2021.
Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Sama- rasekera, and Rakesh Kumar. Semantically-aware -eatoolt reasoning agent for dision-and-languaee navigation in continuous cnvironments. In International Conference on Pottern Recognition, pp. 4065- 4071. IEEE, 2022.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in rision-and-languus navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1862 1872, 2019.
Krishna Murthy Jatavallabhula, Alihuscin Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. In 1C2A Workshop Pretraining for Robotics (PT4R), 2023.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Danial Kamali and Parisa Kordjamshidi. Syntax-guided transformers: Elevating compositional general- ination and grounding in multimodal environments. In Proceedings of the 1st GenBench Workshop (Benchmarking) Generulisation in NLP, pp. 130- 142, 2023.
Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. A new path: Scaling rision-ndd-langaaan navigation with synthetic instructions and imitation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10813 10823, 2023.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. pp. 4171-4186, 2019.
Hyounghun Kim, Jialu Li, and Mohit Bansal. Ndh-full: Learning and evaluating navigational agents on full-length dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Roberta L Klatzky, James R Marston, Nicholas A Giudice, Reginald G Golledge, and Jack M Loomis. Cognitive load of navigating without vision when guided by virtual sound versus spatial language. Journal of experimental psychology: Applied, 12(4):223, 2006.
Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14738 14748, 2021.
Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Simple and effective synthesis of indoor 3d scenes. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 1169-1178, 2023.
Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision-and-languaage navigation in continuous environ- ments. In European Conference Computer Vision, pp. 588 603. Springer, 2022.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision- and-language navigation in contimouus environments. In Europeon Conference on Computer Vision, pp. 104. Springer, 2020.
Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 15161- 15171, 2021.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Te, and Jason Baldridge. Room-across-roomm Multi- lingual vision-and-languuae navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4392-4412, 2020.
Shuhei Kurita and Kyunghyun Cho. Generative languago-grounde policy in vision-and- viston-and-langungo navigation with bayes' rule. In International Conference on Learning Represenlations, 2020.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martin, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, et al. Behavior- 1k: A numan-centered, embodied ai benchmark with 1, 000 everyday activitics and rcalistic simulation. CoRR, 2024a.
Heng Li, Minghan Li, Zhi-Qi Cheng, Yifei Dong, Yuxuan Zhou, Jun-Yan Hc, Qi Dai, Teruko Mitamura, and Alexander G Hauptmann. Hunan-aware vision-and-languuag navigation: Bridging simulation to reality with dynamic human interactions. In The Thirty-eight Conference on Neural Information Processing Systems Datosets and Benchmarks Track, 2024b.
Jialu Li and Mohit Bansal. Improving saiion-and-lnguuaa navigation by generating futuro-view image semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2023.
21

---

Published in Transactions on Machine Learning Research (12/2024)
Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and- language navigation. In Conference on Neural Informalion Processing Systems, volume 36, 2024.
Jialu Li, Hao Tan, and Mohit Bansal, Improving cross-modal alignment in vision language navigation via syntactic information. In Proceedings of the 2021 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1041-1050, 2021.
Jialu Li, Hao Tan, and Mohit Bansal. Clear: Improving vision-language navigation with cross- lingual, auvironment-agnoste representations. In Findings of the Association for Computational Linguistios: NAACL 2022, pp. 633 649, 2022a.
Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-languagge navigation. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15407-15417 2022b.
Jialu Li, Aishwarya Padmakumar, Gaurav Sukhatme, and Mohit Bansal. VIn-video: Utilizing driving videos for outdoor vision-and-langueee navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18517-18526, 2024c.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arxiv: 1908. 03557, 2019a.
Xiangyang Li, Zihan Wang, Jiahao Yang, Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge enhanced reasoning for vision-and-language navigation. In Proceedings of the TEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3583-2592 2023a.
Xin Li, Yeqi Bai, Pinlong Cai, Licheng Wen, Daocheng Fu, Bo Zhang, Xuemeng Yang, Xinyu Cai, Tao Ma, Jianfei Guo, et al. Towards knowledge-driven autonomous driving. arXiv preprint orXiv:2312.04816, 2023b.
Xinghang Li, Di Guo, Huaping Liu, and Fuchun Sun. Reve-ce: Remote embodied visual referring expression in contimuous environment. IEEE Robotics and Automation Letters, 7(2):1494-155) 2022c.
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Çelikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 1494-1499, 2019b.
Xinjun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Com- puter
2020: 16th European Conference, Glasgou, UK, August 28-28, 2020, Proceedings, Part XXX 16, pp. 121-137, Springer, 2020.
Yilan LI, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large dision-languse models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023c.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1993-5500. IEEE, 2023.
Xiwen Liang, Fengda Zhu, Lingling Li, Hang Xu, and Xiaodan Liang. Visual- Visual-language navigation pretrain- ing via prompt-based environmental self-exploration. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4837 4851, 2022.
Bingqian Lin, Yi Zhu, Xiaodan Liang, Liang Lin, and Jianzhuang Liu. Actional atomic-concept learning for demystifying vasion-language navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1568-155 1176, 2023a.
22

---

Published in Transactions on Machine Learning Research (12/2024)
Binggian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting Ilm-based vision-and-language navigation via learning disentangled reasoning. arXie preprint arXiv:2403.0733 07376, 2024a.
Bingqian Lin, Yunshuang Nie, Ziming Wei, Yi Zhu, Hang Xu, Shikui Ma, Jianzhuang Liu, and Xiaodan Liang, Correctable landmark discovery via large models for fision-languan navigation. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2024b.
Chuang Lin, Yi Jiang, Jianfei Cai, Lizhen Qu, Cholamreza Haffari, and Zehuan Yuan. Multimodal trans- former with variable-length memory for vision-and-lan languge navigation. In European Conference on Com- puter Vision, volume 13696, pp. 380-397. Springer, 2022a.
Kevin Lin, Linjic Li, Chung- Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse altention for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognåtion, pp. 17949-179999 17958, 2022b.
Kunyang Lin, Peihao Chen, Diwei Huang, Thomas H Li, Mingkui Tan, and Chuang Gan. Learning vision- and-language navigation from youtube videos. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 8317-8326, 2023b.
Xiangru Lin, Guanbin Li, and Yizhou Yu. Scene-intuitive agent for remote embodied visual grounding. In Proceedings of the IEEE/CVI conference on computer vision and pattern recognition, pp. 7036-7045, 2021.
Jamie Lingwood, Mark Blades, Emily K Farran, Yannick Courbois, and Danielle Matthews. Using virtual environments to investigate wayfinding in 8-to 12-year-olds and adults. Journal of experimental child psychology, 166: 178-189, 2018.
Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. Vision-language navigation with random environmental mixup. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 1644-1654, 2021.
Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird's-eye-view scene graph for vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10968 10980, 2023a.
Rui Liu, Wenguan Wang, and Yi Yang. Volumetric environment represcntation for vision-language navi- gation. In Proceedings of the IEEE/CV /VF Conference on Compuler Vision and Pallern Recognilion, pp. 16317-16328, 2024.
Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang, Yanning Zhang, and Qi Wu. AerialvIn: Vision-and- language navigation for uavs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15384 15394, 2023b.
Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong, Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In 8th Annual Conference on Robot Learning, 2024a.
Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions. In 2024 IEEE International Conference on Robotics and Aulomation (ICRA), pp. 17380- 17387. TEFE, 2024b.
Guanxing Lu, Ziwei Wang, Changliu Liu, Jiwen Lu, and Yansong Tang. Thinkbot: Embodied instruction following with thought chain reasoning. arXiv preprint arXiv:2312.07062, 2023.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic tasks. In Conference on Neural Information Processing Systems,
representations for volume 32, 2019.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Chih-Yaa Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In The Seventh International Confer- ence on Learning Representations, 2019.
Ziqiao Ma, Benjamin VanDerPloeg, Cristian-Paul Bara, Yidong Huang, Eui-In Kim, Felix Gervits, Matthew Marge, and Joyce Chai. Dorothie: Spoken dialogue for handling unexpocted situations in interactive autonomous driving agents. In Findings of the Associalion for Compulational Linguisties: EMNLP 2022.
pp. 4800 1800-4822, 4822, 2022.
Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai Towards holistic landscape of situated theory of mind in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1011-1031, 2023.
Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowl- edge, and action in route instructions. Def, 2(6):4, 2006.
Aly Magassouba, Komei Sugiura, and Hisashi Kawai. Crossmap transformer: A crossmodal masked path transformer using double trakstranslttion for vision-and-llngguaae navigation. IEEE Robotics and Au- tomation Letters, 6:6258 (662586666,, 2021. URL https: stw.aaiiiissmmmttcccccccocoaaaaa http://ariiiiciiiaaoooooo///raaa org/CorpusID: 232075933.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-languuaee navigation with image-text pairs from the web. In European Conference on Computer Vision, pp. 259 274, 2020.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. In NewrIPS 2023 Foundation Models for Decision Making Workshop, 2023.
So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhut- dinov. Film: Following instructions in language with modular methods. In International Conference Learning Representations, 2021.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. In Conference on Neural Information Processing Systems, volume 31, 2018.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Map- ping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2018 2018.
Conference Empirical Methods in Natural Language Processing, pp.
Ronja Möller, Antonino Furnari, Sebastiano Battiato, Aki Härmä, and Giovanni Maria Farinella. A survey on human-aware robot navigation. Robotícs and Autonomous Systems, 145: 103830,,,, 2021.
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. In Advances in Neural Information Processing Syslems, volume 36, 2023.
Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal, Stefan Lee, and Dhruv Batra. SOAT: A scene- and object-aware transformer for aisio---nd-aanguaa navigation. In Advances in neural information processing systems, pp. 7357-7367, 2021.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Eunbodiedgpt: Vision-language pre-training via embodied chain of thought. In Conference on Neural Information Processing Systems, volume 36, 2024.
Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective -unoosity- imitation learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNI.P), pp- 684-695, Hong Kong, China, November 2019.
24

---

Published in Transactions on Machine Learning Research (12/2024)
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language- based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CV Con- ference on Computer Vision and Pattern Recognition, pp. 12527 12537, 2019.
Phuong DH Nguyen, Yasmin Kim Georgie, Ezgi Kayhan, Manfred Eppe, Verena Vanessa Hafner, and Ste- fan Wermter. Sensorimotor representation learning for an "active self" in robots: model survey, KI- Kümslliche Intelligenz, 35:9 35, 2021.
John O'Keefe and Jonathan Dostrovsky. The hippocampus spatial map: preliminary evidence from unit aclivity in the freely-moving rat. Bruin research, 1971.
John O'keefe and Lynn Nadcl. The hippocampus as cognitive map. Oxford university press, 1978.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Span- dana Gclla, Robinson Piramuthu, Gokhan Tur, and Dilck Hakkani-Tun Turr Teach: Task-driven embodied agents that chat. In AAAI, 2022.
Bowen Pan, Rameswar Panda, Sou YouYo Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. Langnav: Language as perceptual representation for navigation. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 950 950-974, 2024.
Amit. Parekh, Malvina Nikandrou, Georgios Pantazopoulos, Bhathiya Hemanthage, Arash Eshghi, Toannis Konstas, Oliver Lemon, and Alessandro Suglia. Emma: A foundation model for embodied, interactive, multimodal task completion in 3d environments. In Aleza Prize SimBot Challenge Proceedings, 2023.
Sang-Min Park and Young-Gat Gab Kim. Visual language navigation: A survey and open challenges. Artificial Intelligence Review, 56(1):365-4277 2023.
Alexander Pashovich, Cordelia Schmid, and Chon Sun. Episodic transformer for vision-anddlanguagg navi- gation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15922 15932. IEEE, 2021.
Sudipta Paul, Amit Roy-Chower ChyChhwd and Anoop Cherian. Avlen: Audio visiio-viiaallangaaee embodied nav- igation in 3d environments. In Conference on Neural Information Processing Systems, volume 35, pp. 6236 6249, 2022.
Tzuf Prgaman and Reut Tsarfaty. RUN through the streets: new dataset and baseline models for realistic urban navigation. In Proceedings of the 2019 Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- TJCNLP); pp. 6449 6455, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 00.1663/v11/D11-1653 1681. URL httpp:///aclaataooogyoor/////
Shannon M Pruden, Susan C Levine, and Janellen Huttenlocher. Children's spatial thinking: Does talk about the spatial world matter? Developmental science, 14(6):1417-1430) 2011.
Xavier Puig, Eric Undersander, Andrew Szot,, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clogg, Michal Hlavac, So Yeon Min, et Habitat 3.0: co-habitat for humans, avatars, and robots. In The Twelfth Indernational Conference on Learning Representalions, 2024.
Jennie E Pyers, Anna Shusterman, Ann Senghas, Elizabeth S Spelke, and Karen Emmorey. Evidence from an emerging sign language reveals that language supports spatial cognition. Proceedings of the National Academy of Sciences, 107(27):12116-22))
2010.
Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. Object-and-action aware model for visual language navigation. In European Conference on Computer Vision, pp. 303 317. Springer, 2020a.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IFFF/CVF Conference on Computer Vision and Patlern Recognition, pp. 2020b.
25

---

Published in Transactions on Machine Learning Research (12/2024)
Yuankai Qi, Zizheng Pan, Yicong Hong, Ming-Hsuan Yang, Anton van den Hengel, and Qi Wu. The road to know-where: An object-and-room informed sequential bert for indoor vision-laguuage navigation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1655 1664, 2021.
Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. HOP: history-and-order aware
pro-training for vision-and-langnuage navigation. In Proceedings of the IEEE/CVF conference Computer 2022.
Vision and Pattern Recognition, pp.
Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-languuge navigation. IEFE Transactions on Pottern Analysis and Machine Intelligence, 2023a.
Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu. March in chat: Interactive prompting for remote embodied referring expression. In Proceedings of the IEEE/CU /CF International Conference on Computer Vision, pp. 15738-15767, 2023b.
Yanyuan Qiao, Qianyi Liu, Jiajun Liu, Jing Liu, and Qi Wu. Lim as copilot for coarse-grained vision-and- language navigation. In Europeon Conference on Computer Vision, pp. 459-476, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. pp. 8748 8763, 2021.
Abhinav Rajvanshi, Karan Sikka, Xiao Lin, Bhoram Lee, Han-Pang Chiu, and Alvaro Velasquez. Saynav: Grounding large language models for dynamic planning to navigation in new environments. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, pp. 464-474, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πlya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821 8831. Pmlr, 2021.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In 7th Annual Conference on Robot Learning, 2023.
Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel X. Chang. Language-aligned waypoint (LAW) supervision for vision-and-language navigation in continuous environments. In EMNLP, pp. 4018 4028, 2021.
Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. Proceedings of Machine Learning Research, 229, 2023.
T Rodrigo. Navigational strategies and models. Psicológico, 23(1), 2002.
Junha Roh, Chris Paxton, Andrzej Pronobis, Ali Farhadi, and Dieter Fox. Conditional driving from natural language instructions. In Proceedings of the Conference on Robot Learning, pp. 540-551, 2020.
Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. Rmm A recur- sive mental model for dialogue navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1732-1745, 1745, 2020.
Homagni Saha, Fateme Fotouhi, Qisai Liu, and Soumik Sarkar. A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment. Frontiers in Robotics and AI, 9, 2022.
Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. Open-ended instructable embodied agents with menory-augmenter large language models. In Findings of the Association for Computational Lin- guistics: FMNLP 2023, pp. 3468 05000, 2023.
26

---

Published in Transactions on Machine Learning Research (12/2024)
Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun. Habitat: A platform for embodied Al research. pp. 9338 9346, 2019.
Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempe: Large language models as decision makers for autonomous driving. arXiv preprint arXic:231000000 2023.
Dhruv Shah, Blažej Osiáski, Sergey Levine, et al. Lan-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pp. 492-504 PMLR, 2023.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lindriv: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15120-15130, 2024.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit rision-and-languag tasks? In International Conference on Learning Represendations, 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVI conference on compuler vision and pallern recognilion, pp. 10740 10749, 2020.
Anna Shusterman, Sang Ah Lee, and Elizabeth S Spelke. Cognitive effects of language on human navigation. Cogmition, 120(2):186 201, 2011.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In First Vision and Language for Autonomous Driving and Robotics Workshop, 2023.
Kunal Pratap Singh, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mot- taghi. Ask4help: Learning to leverage an expert for embodied tasks. In Conference on Neural Information Processing Systems, volume 35, pp. 16221-16232, 2022.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei Chao, and Yu Su. Lim- planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1998-3009, 2023.
NN Sriram, Tirth Maniar, Jayaganesh Kalyanasundaram, Vineet Gandhi, Brojeshwar Bhowmick, and K Madhava Krishna. Talk to the vehicle: Language conditioned autonomous navigation self driv- ing cars. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5284 5290. IEEE, 2019.
Yifei Su, Dong An, Yuan Xu, Kehan Chen, and Yan Huang. Target-grounded graph-aware transformer for aerial vision-and-dialog navigation. arXiv preprint Xix::22308. 11561, 2023.
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John M. Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel X. Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. Tn Advances in Neural Information Processing Systems, pp. 251-266, 2021.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint. Conference on Natural Language Processing (FMNI.-IJCNI.P), pp. 5100-5111 2019.
27

---

Published in Transactions on Machine Learning Research (12/2024)
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2610-26200 2021, 2019.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning, pp. 394 406. PMLR, 2020.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevim: The convergence of autonomous driving and large vision-language models. In Conference on Robol Learning (CoRL), 2024.
Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4))189, 1948.
Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang, Bootstrapping Ilm-based task-oriented dialogue agents via self-talk. arXiv preprint arXi::2401.5003. 2024.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models planning and reasoning about change. In Advances in Newral Information Processing Systems, 2022.
Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Talk2nav: Long-range vision-and-language navigation with dual attention and spatial memory. International Journal of Computer Vision, 129(:): 246-266, 2021.
Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, and Jianbing Shen. Structured scene memory for vision-language navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 8455 8464, 2021.
Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for contimu- ous vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 10873-10883, 2023a.
Liuyi Wang, Zongtao He, Jiagui Tang, Ronghao Dang, Naijia Wang, Chengju Liu, and Qijun Chen. A dual semantic-aware recurrent global-adaptive network for rision-and-languase navigation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 1479-1487, 2023b.
Liuyi Wang, Chengju Liu, Zongtao He, Shu Li, Qingqing Yan, Huiyi Chen, and Qi Chen. Pasts: Progress- aware spatio-temporal transformer speaker for vision- aaiion-andllanguage navigation. Eng. Appl. Artif. Intell, 128:107487, 2023c. URL https: haaaaamiiiccollar..rrrrrrrppppps Cttg:/CrppsII::::: 258833536
Su Wang, Coslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Peter Anderson. Less is more: Generating grounded naviga- tion instructions from landmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2022a.
Pattern Recognition, pp.
Wenhai Wang, Jiangwei Xie, Chuan Yang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXie preprint ar arXiv:2312.000 09245,, 2023d.
Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. Lana: A language-capable navigator for instruction following and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19048 19058, 2023e.
Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. Learning to follow and generate instructions for language-capable navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023f.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision- language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 6629 6638, 2019.
28

---

Published in Transactions on Machine Learning Research (12/2024)
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint Xrx::22212 03191, 2022b.
Zehao Wang, Mingxiao Li, Minye Wu, Maric-Francine Moens, and Tinne Tuytelaars. Find a way forward: a language-guided semantic map navigator. arXiv preprinl ar Xiw: 20ne220300333,, 2022c.
Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shugiang Jiang. Gridmm: Grid memory map for vision-and-languaaee navigation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 15625 15636, 2023g.
Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13753 13762, 2024a.
Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Sim-to-real transfer via 3d feature fields for vision-and-language navigation. arXiv preprint or XrXin:2406. 000.0....9..... 2024b.
Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pp. 12009- 12020,, 2023h.
Zun Wang, Jialu Li, Yicong Hong, Songze L.,, Kunchang Li, Shoubin Yu, Yi Wang, Yu Qiao, Yali Wang, Mohit Bansal, et al. Bootstrapping language-guided navigation learning with self-refining data flywheel. arxiv preprinl XrXie:242..0688 0846677, 2024c.
William H Warren. Non-euelidean navigation. Journal of Esperimental Biology, 222((uppl 1):jeb187971, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning large language models. In Advances in Newral Information Processing Systems, 2022.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A dnowledge-driven approach autonomous driving with large language models. In The Twelfth International Conference on Learning Representations, 2023.
Wansen Wu, Tao Chang, Xinmeng Li, Quanjun Yin, and Yue Hu. Vision-language navigation: a survey and taxonomy, Neural Computing and Applications, 36(7):3291 3316, 2024.
Zongkai Wu, Zihan Liu, and Donglin Wang. Multi-grounding navigator for self-supervised vision-and- language navigation. In 2021 International Joint. Conference on Neural Nelworks (IJCNN), pp. 1-8. IEEE, 2021.
Fci Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savaresc. Gibson cnv: Real- world perception for embodied agents. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 9068 9079, 2018
Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, and Noah A Smith. Multi-view learning for vision-and-language navigation. arXiv preprint arXiv:2003.00857 2020.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. In Conference on Neurul Information Processing Systems, volume 36, 2024.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: novel task for fine-grained image understanding. ar Xie preprint arxi 1ett1091100666 2019.
29

---

Published in Transactions on Machine Learning Research (12/2024)
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. TEEE Robotics and Automation Letters, 2024a.
Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang. A survey on robotics with foundation models: toward embodied ai. arXiv preprint arXi::242. 2024b.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 11934, 2020.
Xu Yan, Haiming Zhang, Yingjic Cai, Jingming Guo, Weichao Qiu, Bin Gao, Kaiqiang Zhou, Yuc Zhao, Huan Jin, Jiantao Gao, et al. Forging vision foundation models for autonomous driving: Challenges, methodologies, and opportunities. orvi preprint or arXiv:2401.0004 2024.
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Foubey, and Joyce Chai. Lim-grounder: Open-vocabulaty 3d visual grounding with large language model as agent. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024.
Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from envirommen- tal leedback. In European Conference on Compuler Vision, pp. 20 38. Springer, 2025.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multi- modality. arXiv preprint Xiw:2304.14178, 2023.
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, et al. Homerobot: Open- vocabulary mobile manipulation. In Conference on Robot Learning, pp. 1975-2011, 2023.
Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag- driver: Generalisable driving explanations with eetrieval-angmentt in-context learning in multi-modal large language model. aixiv preprint 0r0 10828, 2024.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vasion-language models as decision-making agents via reinforcement learning. In The Tiihty- Annual Conference on Newral Information Processing Systems, 2024.
Zhaohuan Zhan, Jinghui Qin, Wei Zhuo, and Guang Tan. Enhancing vision and language navigation with prompt-based scene knowledge. IEEE Transactions on Circuits and Systems for Video Technology, 2024a.
Zhaohua Zhan, Lisha Yu, Sijic Yu, and Guang Tan. Me-gpt: Empowering viston-and-langunge navigation with memory map and reasoning chains. arXiv preprint arXiv:2405. 10620, 2024b.
Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomong Fang. Qi Wu, Zhizheng Zhang, and Wang He. Navid: Video-based vlm plans the next step for vision-and-languagee navigation. In Robotics: Science and Systems (RSS), 2024a.
Tianyao Zhang, Xiaoguang Hu, Jin Xiao, and Guofeng Zhang. survey of visual navigation: From geometry to embodied ai. Engineering Applications of Artificial Intelligence, 114:105036, 2022a.
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, and Joyce Chai. Danli: Deliberative agent for following natural language instructions. In Proceedings of the Conference on Empirical Melhods in Natural Language Processing, 2022b.
Yichi Zhang, Jianing Yang, Keunwoo Yu, Yinpei Dai, Shane Storks, Yuwei Bao, Jiayi Pan, Nikhil Devraj, Ziqiao Ma, and Joyce Chai. Seagull: An embodied agent for instruction following through situated dialog. Tn Aleza Prize SimBot Challenge Proceedings, 2023.
30

---

Published in Transactions on Machine Learning Research (12/2024)
Yubo Zhang, Hao Tan, and Mohit Bansal. Diagnosing the environment bias in vision-and-languue naviga- tion. In Proceedings of the Twenby-Ninn Ninlh International Conference on International Joind Conferences on Artificial Intelligence, pp. 890 897, 2021a.
Yue Zhang and Parisa Kordjamshidi. Lovis: Learning orientation and visual signals for vision and language navigation. Proceedings of the 29th International Conference on Computational Linguistics, pp. 5745 5754, 2022a.
Yue Zhang and Parisa Kordjamshidi. Explicit object relation alignment for vision and language navigation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 322-331, 2022b.
Yue Zhang and Parisa Kordjamshidi. VLN-Trans Translator for the vision and language navigation agent. In The G1 Ansuaa Meeting Of The Association For Computational Länguisties, 2023.
Yue Zhang and Parisa Kordjamshidi. Narrowing the gap between vision and action in navigation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 856-865, 2024.
Yue Zhang, Quan Guo, and Parisa Kordjamshidi. Towards navigation by reasoning over spatial configu- ralions. In Proceedings of Second International Combined Workshop on Spatial Language Underslanding and Grounded Communication for Rohotics, pp- 42-52, 2021b.
Yue Zhang, Quan Guo, and Parisa Kordjamshidi. NavHint: Vision and language navigation agent with a hint generator. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 92 103, 2024b.
Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. Spartun3d: Situated spatial understanding of 3d world in large language models. arXiv preprint arXix: 10.00887, 2024c.
Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deepfake detection. In European Conference on Computer Vision, pp. 399 415. Springer, 2025.
Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do rision-languag models represent space and how? evaluating spatial frame of reference under ambiguities. In Pluralistic Alignment Workshop at NewrIPS 2024, 2024d.
Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang. Tamm: Triadapter multi-modal learning for 3d shape understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21413 21423, 2024e.
Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexander Ku, Jason Baldridge, and Eugene Te. On the evaluation of vision-and-langunes navigation instructions. In Proceedings of the 16th Conference of the European Chapter of the Association for Compulational Linguistics: Main Volume, pp. 1302 1316, 2021.
Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and Si Liu. Target-driven structured transformer planner for vision-language navigation. pp. 4194-4203, 2022.
Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13624-13634, 2024a.
Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Di, Xuehai He, and Xin Eric Wang. Jarvis: A neuro- seuro-symbolic commonsense reasoning framework for conversational embodied agents. arXiv preprint ar Xrx::2200.11 13266, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhangha Wu, Yonghao Zhuang, Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as- a-judge with mt-bench and chatbot arena, 2023.
31

---

Published in Transactions on Machine Learning Research (12/2024)
Qi Zheng, Daqing Liu, Chaoyue Wang, Jing Zhang. Dadong Wang, and Dacheng Tao. Esceme: Vision-and- language navigation with episodic scene memory. International Journal of Computer Vision, pp. 1-21, 2024b.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. arXiv preprinl Xrx::2232. 094 19, 2023.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-languuaee navigation with large language models. pp- 7641-7649. AAAI Press, 2024a. doi: 10.1609 V3177.22887 URL
htttp:://db..org1100.o
v3817. 28817.28597
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-ande and-languagee navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7641 7649, 2024b.
Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision langun- models. In European Conference on Computer Vision, pp. (60-278. Springer, 2025.
Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B Tenenbaum, and Chuang Gan. Hazard challenge: Embodied decision making in dynamically changing environments. In The Twelfth International Conference on Learning Representations, 2024c.
Xinzhe Zhou and Yadong Mu. Tree-structured trajectory encoding for vision-and-languaee navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3814-3824, 2023.
Chen Zhu, Michael Meurer, and Christoph Günther. Integrity of visual navigation navigationn developments, challenges, and prospects. NAVIGATION: Journal of the Instätute of Navigalion, 69(2), 2022.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. Tn Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 10012-10022, 2020.
Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang, Soon: object navigation with graph- based exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion, pp. 12689 12699, 2021a.
Scenario oriented
Fengda Zhu, Yi Zhu, Vincent Lee, Xiaodan Liang, and Xiaojun Chang. Deep learning for embodied vision navigation: A survey. arxiv preprint arxiv:2108, 04097, 2021b.
Fengda Zhu, Vincent CS Lee, Xiaojun Chang, and Xiaodan Liang. Vision language navigation with knowledge-driven environmental dreamer. In International Joint Conference on Artificial Intelligence 2023, pp. 1840 1848. Association for the Advancement of Artificial Intelligence (AAAI), 2023.
Yi Zhu, Yue Weng, Fengda Zhu, Xiaodan Liang, Qixiang Ye, Yutong Lu, and Jianbin Jiao. Self-motivatod communication agent for real-world vision-dialog navigation. In Proceedings of the IEEE/CVE Interna- tional Conference Computer Vision, pp. 1604-1603, 2021c
32


Reviews:
[{"review_id": "cpHGQgR8w1", "reviewer": "Reviewer_s3ya", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n1. The paper is well-organized. The authors take a top-down approach and divide the paper into primarily three sections (world model, human model, VLN agent), which is intuitive. Further, within each section, they identify the key role(s) of each component (e.g., the world model needs to encode the history of the actions and observations made by the agent) and describe how previous works have approached them. Wherever possible, they also discuss how foundation models can help improve performance in this respect. It neatly organizes the VLN literature, which is crucial for a survey.\n2. The paper is well-written, with most parts being easy to follow.\n3. The paper identifies important challenges and future directions.\n\n\n=== Weaknesses ===\n1. Individual subsections are a bit dense and it is hard to understand the core ideas from the papers from the survey alone (especially for people with limited background in VLN/ foundation models). I do understand that it might be hard for the authors to explain the ideas in more detail given the space limit.\n2. The reader might benefit from a brief introduction to relevant foundation models, i.e., the ones that have been applied to VLN, such as CLIP, LLMs, etc. Perhaps the paper could point to some surveys on these models or provide a brief overview.\n3. I didn’t see much emphasis on multimodal LLMs (like GPT-4o). Have some works tried to use such models in the VLN setting? Some examples could be [4], [5]. I would urge the authors to discuss more works using multimodal LLMs if applicable.\n\n\n=== References ===\n\n[1] Sarch, Gabriel, et al. \"Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.\n\n[2] Li, Chengshu, et al. \"Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.\" Conference on Robot Learning. PMLR, 2023.\n\n[3] Liang, Jacky, et al. \"Code as policies: Language model programs for embodied control.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[4] Yang, Jingkang, et al. \"Octopus: Embodied vision-language programmer from environmental feedback.\" arXiv preprint arXiv:2310.08588 (2023).\n\n[5] Driess, Danny, et al. \"PaLM-E: An Embodied Multimodal Language Model.\" International Conference on Machine Learning. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper surveys the field of vision-language navigation (VLN) with an emphasis on foundation models.\n2. It provides a brief summary of the VLN setup, the benchmarks in the field, and their taxonomy.\n3. It leverages the LAW framework to outline the three components of a VLN agent (world model, human model, VLN agent). Subsequently, it summarises the literature addressing each of these components, including both the traditional approaches as well as those involving foundation models.\n4. The paper identifies challenges and directions for future work in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These changes could strengthen the work:\n\n1. Addressing the weaknesses.\n2. The evaluation metrics could be explained better. For example, CLS has not been explained at all. Does SPL aim to provide higher scores for longer trajectories?\n3. Section 3 (world model): LLMs have strong commonsense and world knowledge. Have existing works leveraged this to build better world models? I’m curious to hear the authors’ thoughts on this.\n4. Section 4.1 (information seeking): Most of the papers discussed here involve using LLMs as an oracle. Are there works that leverage LLMs for deciding when to ask for help and for asking informative questions?\n5. Section 4.2 (Pre-trained representations): Most of the works in this section are from 2020 or before. Are there more recent works in this direction?\n6. Section 3.2 (Environmental Augmentation): In the last paragraph, the authors say that the paradigm has shifted from using augmented environments in the fine-tuning stage to the pre-training stage. While in general I understand the different between the two stages, it’s not clear to me what it means in this context? Is it that the scale of data being used now (in the pre-training stage) is much larger than before (in the fine-tuning stage)? or is there more to it. Some clarification here would be helpful.\n7. Section 3.1 (History Encoding): The references involving panorama encoders could be re-written to explain how panorama encoders work and are useful for VLN.\n8. Section 4.1 (Information Seeking): the references for using foundation models as oracle models (Fan et al., Lin et al., Ye et al., Zhu et al.) could be re-written to enhance clarity.\n9. Section 5.3 (VLMs as agents): The content regarding waypoint detection was not clear. Perhaps a brief explanation of waypoint detection would be helpful for the reader.\n10. Section 5.3 (LLMs as agents): DiscussNav – why is a multi-agent approach useful?\n11. Some other references that could be useful: [1], [2] (benchmarks), [3]\n12. [typo] Section 3.1: “Prior to employing the foundation models in VLN, LSTM hidden states serve…”, grammatically, the sentence reads a bit off (using present tense following the word ‘prior’). It’s possible that the grammar is technically okay, just wanted to flag it in case this was unintentional.\n13. [typo] Section 4.1: “Zhang & Kordjamshidi (2023)..” (two periods)\n\n\n=== References ===\n\n[1] Sarch, Gabriel, et al. \"Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.\n\n[2] Li, Chengshu, et al. \"Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.\" Conference on Robot Learning. PMLR, 2023.\n\n[3] Liang, Jacky, et al. \"Code as policies: Language model programs for embodied control.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[4] Yang, Jingkang, et al. \"Octopus: Embodied vision-language programmer from environmental feedback.\" arXiv preprint arXiv:2310.08588 (2023).\n\n[5] Driess, Danny, et al. \"PaLM-E: An Embodied Multimodal Language Model.\" International Conference on Machine Learning. PMLR, 2023.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VJr1ftProo", "reviewer": "Reviewer_Eo4S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this review to be very comprehensive, and appreciated the clear organizational framework. I think it will serve as a very useful survey of this field and the ways in which it is being influenced by pretrained foundation models. I have very little to say in terms of weaknesses, but do have just a couple of comments below that may be helpful for further improving the paper.\n\nComments:\n- My main comment is that it is stated several times in the paper that LLMs provide powerful reasoning and planning capabilities that may aid VLN agents. I can see how this may be a reasonable characterization relative to the previous state-of-the-art in this field (i.e. prior to the adoption of pretrained foundation models), but there is not much discussion of the extensive literature and controversy surrounding the zero-shot reasoning and planning capabilities of these types of systems. Of direct relevance to the VLN domain is recent work showing that LLMs have poor zero-shot planning abilities [1,2]. It would be helpful to discuss the implications of these findings for the use of such systems as VLN agents.\n- One very minor comment is that I found it odd to use the word 'milestone' as a verb, as this isn't a usage that I have seen before. It might be more straightforward to say something like 'document the progress' rather than 'milestone the progress', but again this is a minor comment.\n\n[1] Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S. (2024). Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36.\n\n[2] Momennejad, I., Hasanbeig, H., Vieira Frujeri, F., Sharma, H., Jojic, N., Palangi, H., ... & Larson, J. (2024). Evaluating cognitive maps and planning in large language models with CogEval. Advances in Neural Information Processing Systems, 36.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive summary of vision and language navigation research, with a special focus on the role of pretrained foundation models. The paper presents a useful taxonomy for organizing the literature, dividing the problem into three subproblems involving the formation of world models, human models, and the implementation of the navigation agent itself. A useful taxonomy is also presented for characterizing the range of datasets and tasks that have been considered in this literature. Finally, the paper identifies several limitations of current work and avenues for future work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be helpful to discuss the implications of findings indicating poor zero-planning in LLMs for their potential use as VLN agents.", "broader_impact_concerns": "I did not identify any ethical implications of this work that would necessitate a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z6iN2yhw26", "reviewer": "Reviewer_HbEf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Paper mostly well structured (see also weaknesses) and written.\n- Covers many different approaches.\n- Given the rise of foundation models and there increased importance and success, a survey paper is very valuable.\n- The section about challenges and future directions is interesting and can inspire future research.\n\n**Weaknesses**\n- The paper often refers to general foundation models, without specifying which kind of. For example last paragraph of Section 3. Giving a more strucutured overview and/or being more specific of which foundation models, covering which modalities is used for which task would be helpful, e.g., as done in section 5.3, or even providing a tabular overview if possible. However, this structure, focusing on the foundation model is not consistent througout the paper. E.g., in 3.1. it is less clear. There it seems to be more general about transformer based architecture and not specifically foundation models. This separation should be clearer.\n- The paragraph and table 1 about benchmarks is a bit disconnected from the rest of the story line. That part is less connected to foundation models. The same applies to the part about evaluation metrics. Also there the connection to foundation model and the purpose of the paragraph for the whole paper is unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper represents a survey of the role of foundation models in vision-and-language navigation (VLN). It analyses the topic based on the 3 areas of VLN (world model, human model, and VLN agent). It also includes an overview of existing benchmarks. Besides discussing multiple existing methods and how they differ in terms of the usage of foundation models it also includes some perspective of open challenges and potential for future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The paper would benefit from partially improved representation of the content, especially when currently talking only generally about foundation models. Given that this is the main topic, it should be clear what kind of foundation model (e.g., which modalities) is used. Especially Section 3 is lacking in this regard. \n- Instead of giving a tabular overview of the datasets, something like this focusing on the main content of the paper, i.e., section 3-5, and representing the content in a structured way would be more helpful for the reader.", "broader_impact_concerns": "Does not have a broader impact statement. It could a general one be added as foundation models are very powerful, but can also be biased.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is fine as is, but for the camera-ready, remember to remove the blue-colored text which identifies the changes (useful for the revision, but should not be camera-ready).
Decision: Accept (poster)